From f0b0aeaf649d8d8b9ba626b4615f7a4e869c163d Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 18 Aug 2026 23:47:15 +0000 Subject: [PATCH 001/535] docs(protected-content): add mint-sell-buy-play journey implementation plan Vision-aligned end-to-end plan (design + UI/UX) to take the canonical protected-content extraction from trust-core foundation to the full mint->sell->buy->play journey on the ESP substrate. Co-authored-by: Anders --- docs/PROTECTED_CONTENT_JOURNEY_PLAN.md | 350 +++++++++++++++++++++++++ 1 file changed, 350 insertions(+) create mode 100644 docs/PROTECTED_CONTENT_JOURNEY_PLAN.md diff --git a/docs/PROTECTED_CONTENT_JOURNEY_PLAN.md b/docs/PROTECTED_CONTENT_JOURNEY_PLAN.md new file mode 100644 index 000000000..d70bbaef7 --- /dev/null +++ b/docs/PROTECTED_CONTENT_JOURNEY_PLAN.md @@ -0,0 +1,350 @@ +# Protected-content journey implementation plan (mint → sell → buy → play) + +Status: proposal. This plan describes how to take the canonical protected-content +extraction (`feat/protected-content-*`) from a trust-core foundation to the full +end-to-end user journey — **mint → sell → buy → play** — with first-class design and +UI/UX, coherent with [PRINCIPLES.md](../PRINCIPLES.md) and the canonical architecture in +[PROTECTED_CONTENT.md](PROTECTED_CONTENT.md). + +It is written to complement, not replace, the "PR #15 disposition" already recorded in +[PROTECTED_CONTENT.md](PROTECTED_CONTENT.md): PR #15 (`feat/dkms-esp-port`) is research and +behavior evidence; this plan is how the canonical line reaches the same user-visible +capability on the ESP substrate without inheriting PR #15's rejected patterns. + +--- + +## 1. Goal and definition of done + +One sentence: **a signed-in person can seal an asset in Create, list it in the Store, a +second person can buy it with an explicit spend confirmation, and open and play it from +their Library — and at no point does a capsule, the marketplace, or the viewer hold raw +key material, wallet authority, or chain authority.** + +"Done" for the 0.7 window is a real vertical slice, not the whole catalog of features: + +- one content asset type (media), one custody topology (the source 2-of-3 pool, one-node + provider process acceptable for the dev lane), one viewer (`elastos.viewer/media@1`); +- mint, list, buy, acquire, open, and play all reachable from Home by an ordinary user; +- a green end-to-end acceptance oracle proving the whole rail; +- every unauthorized step fails closed with an explanation. + +Everything past that (apps/games as sellable assets, resale/trade, multi-segment media, +DKG/rotation/revocation, PQ-hybrid custody) is explicitly deferred and named in §9. + +--- + +## 2. Vision alignment (this is the coherence contract) + +Every stage below is designed against [PRINCIPLES.md](../PRINCIPLES.md). The mapping is the +acceptance criterion for "aligned with the ElastOS vision", not decoration: + +| Principle | How this journey honors it | +|---|---| +| 1 Local first | The bought asset is a normal object pinned into the buyer's `localhost://` Library; discovery and playback are explained as Library/Store objects, not host URLs. | +| 3 No ambient authority | Capsules request typed Runtime resources only. Marketplace, Create, and viewers never hold a signer, CEK, IV, RPC, or provider route. | +| 4 Carrier is not the capsule contract | Custody nodes are reached over Carrier *behind* Runtime resources; capsules never name node endpoints. | +| 5 Small trusted core | Authority, orchestration, audit live in Runtime; crypto lives in the canonical crates + providers; UI lives in capsules. No app reimplements decryption or license policy (15). | +| 10 One canonical path / 11 Fail closed | One `open` gate, one `buy` verb, one `mint` verb. Missing rights → 403 → Home offers buy; missing custody → explicit error, never a weaker fallback. | +| 13/14 Objects, capsules, spaces distinct; human names | Product surfaces read **Create**, **Store**, **Library**, **Wallet** — not dKMS/dDRM/keygen. Internal terms stay in developer docs. | +| 15 Trust travels with signed content | Viewer selection comes from the sealed object's `viewer.required_interface`; access policy and CEK commitment are bound into signed metadata. | +| 16 UI surfaces are not authority | Every buy/open/mint requires the capability for that surface. Home binds browser-frame messages to the launched frame + capsule-scoped capability before acting; a launch token in a URL fragment is not authority by itself. | +| 17 Design tokens are product contracts | All surfaces use `capsules/_shared` tokens; money verbs get one consistent, unmistakable visual treatment across the whole journey. | +| 7 Humans and agents share one authority model | The buy/open/mint operations an agent would call are the same capability-scoped operations the UI triggers; no UI-only side door. | + +If a design choice cannot be expressed as one of the rows above, it is out of scope or +wrong. + +--- + +## 3. Architecture spine (reused, not reinvented) + +The canonical path from [PROTECTED_CONTENT.md](PROTECTED_CONTENT.md) is the backbone for +all four stages: + +``` +capsule -> Runtime coordinator -> rights-provider -> custody providers -> decrypt-provider +``` + +- **Runtime** derives authority from the authenticated Profile, exact Wallet approval, + session, object, and action; selects providers; owns durable orchestration and audit. +- **rights-provider** evaluates the exact approved policy through typed Chain evidence; it + does not release keys. +- **custody providers** each independently verify the exact Runtime operation and rights + evidence and return only a recipient-encrypted contribution + authenticated receipt. +- **decrypt-provider** is the only boundary that reconstructs and briefly holds a live CEK, + returns scoped output, and zeroizes. +- **Carrier** transports Runtime-selected endpoint traffic; it is not rights/custody/key + authority. + +Canonical crates already in the extraction (reuse as-is; extend at their boundaries): +`elastos-protected-content-contracts`, `elastos-protected-content-custody`, +`elastos-protected-content-provider-contracts`, and the `capsules/custody-provider` +process. + +--- + +## 4. The journey, stage by stage + +Each stage lists the **user story**, the **UI surface**, the **Runtime seam**, the +**providers**, the **authority/security**, and the **object model**. Reuse/re-derive/reject +decisions vs PR #15 are consolidated in §8. + +### 4.1 Mint — "Create and seal an asset" + +- **User story:** a creator picks a file in **Create**, sets access terms (free / buy-once + / buy-and-resell) and price, and publishes. One action seals + escrows + lists (sale + terms are baked into the mint, so primary listing == minting). +- **UI surface:** `capsules/creator` (adopt from PR #15, harden). Single "Create asset" + flow: choose object from Library → choose viewer interface (auto from media type) → set + terms → review → **mint (money verb)**. Poster/cover is chosen here; no secure-preview + egress ever (asset detail later shows poster only). +- **Runtime seam:** `POST /api/create/mint` (`mint_authority`) — a money verb (§6). +- **Providers/crypto:** CEK minted inside the encrypt/custody boundary; CENC-encrypt; + content-address the ciphertext; escrow CEK shares to the custody pool at publish time via + the canonical `custody-provisioning` + `elastos-protected-content-custody` provisioning. + Public metadata carries only bounded identities, threshold/epoch/pool facts, CEK + commitment, and signatures — never shares. +- **Object model:** the sealed object binds the exact pool identity, epoch identity, and + committee-authorization identity it was minted against; `viewer.required_interface` is + written into the object so Runtime can later pick the viewer without guessing. + +### 4.2 Sell — "List it in the Store" + +- **User story:** the minted asset appears in the unified **Store** for others to discover; + the creator can see/manage their listings. +- **UI surface:** one canonical **Store** (consolidate `capsules/marketplace` and + `capsules/marketplace-content` into a single asset-type-aware surface; fold the existing + app-store in as the "Apps" fulfillment path later). The Store **mints nothing, plays + nothing, holds no signer/CEK/RPC** — it only decodes listings and produces (a) an unsigned + buy intent and (b) a pinned encrypted file on fulfillment. +- **Runtime seam:** read side of `/api/market/*` (listing/index decode over + `chain-provider` getLogs cache). Marketplace uses the `content/*` plane, never + `ipfs-provider` directly. +- **Authority/security:** listing reads are non-money; no signer. Asset detail shows + poster/cover only. + +### 4.3 Buy — "Acquire the access right" + +- **User story:** a buyer opens asset detail, taps **Buy**, sees exactly what they are + about to sign and spend, confirms with a passkey, and the encrypted file lands in their + Library. +- **UI surface:** Store asset detail → **Buy** → **Home spend-confirmation dialog** + (`capsules/home/browser/home-spend-prompt.js`, adopt + harden): *what is displayed is what + is signed*. Rendered in Home chrome, not inside the app frame (16 UI-is-not-authority). +- **Runtime seam:** `POST /api/market/buy` (`buy_authority`) — money verb. Re-verifies the + listing live and aborts on drift; produces an **unsigned** `buyAccess` for the wallet to + sign and broadcast. Fulfillment: `object-provider Acquire` pins the encrypted CID into the + buyer's Library. +- **Authority/security:** money verb (§6) — passkey step-up + Wallet approval; the CEK is + never touched at buy time (buying is an on-chain access right, not a key handover). + +### 4.4 Play — "Open and play from the Library" + +- **User story:** the buyer opens the asset from their Library; it plays; the key is never + exposed; when they close it, the session and any live key material are gone. +- **UI surface:** Runtime-selected viewer capsule — `elacity-player` for + `elastos.viewer/media@1`, `ddrm-viewer` for `elastos.viewer/document@1` (adopt + harden). + The Store and other apps never choose a viewer. +- **Runtime seam:** `POST /api/viewers/open { uri }` is the single open gate: + 1. resolve the object inside the principal's own root only; + 2. resolve the on-chain subject (the parked `RequiredHomeLaunchToken` rewire — §6); + 3. ask `rights-provider → chain-provider has_access_by_content_id`, get a signed rights + receipt; + 4. launch authority bound to `content_id` + rights-receipt hash (welded into the decrypt + transcript AAD); + 5. return `{ viewer, session, title, play_url, rights_binding }`. + A rights-denied open returns 403; Home offers the buy and retries (one canonical retry + path). A non-owned and a non-existent asset return **byte-identical 404** — the refusal is + not an existence oracle. +- **Session lifecycle:** open → session-bound scoped reads → close → sweeper reap. The + decrypt boundary reconstructs the CEK only inside the scoped session and zeroizes on close + or expiry. The viewer receives scoped output only (rendered / stream / working-copy), + never key material. + +--- + +## 5. UI/UX design (primary focus) + +### 5.1 Design system and consistency + +- All journey surfaces consume `capsules/_shared/{elastos-theme.js, elastos-ui.css, + elastos-accent-picker.js, fonts}`. No one-off color literals (17). Colors are named by + role; Home owns the wallpaper and brand layer. +- Each functional surface may use a scoped accent palette (Store, Create, Viewer), but the + same role token means the same thing everywhere (a "destructive"/"spend" token is one + color across Home, Store, and the spend prompt). +- Windows use the canonical `home-gui` chrome modes + (`capsules/home-gui/browser/{home-gui.js, shell-core.js, style.css}`). Create, Store, and + viewers are ordinary App windows; the spend prompt is Home chrome, never app chrome. + +### 5.2 Information architecture (human nouns, principle 14) + +- **Create** — seal + list (one app; primary listing == minting). +- **Store** — one canonical discover/buy/manage surface (consolidated). Asset-type aware; + content today, apps/games later via the existing install path. +- **Library** — where bought objects live and are opened; protected assets are normal + objects with a lock affordance and a Store provenance chip. +- **Wallet** — signer and balances; surfaced only through Runtime-mediated approval, never + embedded in the Store. +- **Viewer** — Runtime-selected; full-bleed playback with a minimal chrome and a visible + session/expiry indicator. +- Money verbs are **never** silent: mint and buy always route through the Home + spend-confirmation dialog. + +### 5.3 Key flows (wireframe-level) + +1. **Discover → detail:** Store grid → asset detail (poster/cover only, terms, price, + creator identity chip). No secure preview, no decrypt at detail. +2. **Buy:** detail **Buy** → Home spend-prompt (asset, price, chain, destination, the exact + `buyAccess` being signed) → passkey step-up → progress → "Added to your Library" with a + deep link to open. +3. **Open/play:** Library item **Open** → (if owned) viewer launches and plays; (if not + owned) 403 → inline "You don't own this yet — Buy?" → buy flow → auto-retry open. +4. **Close:** explicit close or expiry → session ends, key zeroized, viewer shows a clean + "session ended" state; reopening re-runs the open gate. + +### 5.4 States and copy (fail closed, then explain — principle 11) + +Every surface defines: loading, empty, success, rights-denied (403 → offer buy), +custody-unavailable (explicit, no fallback), session-expired, chain/wallet-declined, +and offline. Copy explains *what is missing and what the correct path is*; no spinner that +hides a fail-closed state. Error tone is calm and specific, consistent with Home/System. + +### 5.5 Accessibility and trust cues + +Money verbs and the spend prompt meet the highest contrast/focus/confirmation bar in the +product: keyboard-first, explicit confirm, no defaulted destructive focus, and a visible +"what is signed" region that is literally the signed payload. The lock/ownership state on a +Library object is conveyed by more than color. + +--- + +## 6. Authority and security design (the hard core) + +- **Money verbs** (`/api/create/mint`, `/api/market/buy`) require passkey step-up that is + intent-, launch-, and time-bound and single-use, brokered under Home's cookie authority, + and rendered by the Home spend-confirmation dialog (what is displayed is what is signed). + Launch tokens travel in URL fragments, never query strings. +- **Subject-resolution rewire (the PR #15 parked blocker):** thread + `RequiredHomeLaunchToken` through `viewer_open` and the `gateway_marketplace` read sites + so `resolve_subject_address` can mint a `RuntimeWalletAuthority` and enumerate the + principal's linked EVM account. Until this lands, chain-gated open/buy is dev-lane only + and must say so (11). **This is scheduled in Phase 2, not deferred**, because it is the + actual gate between the dev lane and a real buy. +- **Key custody invariants (already canonical):** CEK never whole outside a sandbox; one + node-sealed share per node; recipient-sealed contributions (RFC 9180 X25519 + HKDF-SHA256 + + AES-256-GCM); exact-threshold reconstruction (required-plus-one rejected); CEK + commitment checked after reconstruction; zeroization; audit. +- **Refusal is not an oracle:** non-owned and non-existent objects are byte-identical 404s. +- **No second authority path:** the provisional `drm/rights/key/decrypt` providers and + `elastos_common::protected_content` DTOs are removed atomically at cutover (§7 Phase 5), + never kept as a parallel decoder. + +--- + +## 7. Workstreams and sequencing + +Difficulty is described by scope and risk, not calendar time. Phases are ordered by +dependency; the 0.7 target is Phases 0–3 plus the Phase 4 minimum. + +**Phase 0 — decisions and scaffolding (unblocks everything).** +- Stand up a single 0.7 integration branch that composes collaboration + this journey. +- Adopt PR #15's `dkms_rail` e2e as a *behavioral oracle* on the canonical line (start red; + it defines "done"). +- Converge naming: product nouns Create/Store/Library/Wallet/Viewer; internal terms + protected-content/custody/rights/decrypt; retire dKMS/dDRM from user-facing copy. +- Fix the `custody-provider` process-test clock coupling (inject a clock or widen windows) + so the one runnable piece is CI-stable. + +**Phase 1 — dev-lane vertical slice (proves the contracts compose).** +- Wire Runtime orchestration end to end for one media asset with `ELASTOS_DDRM_SUBJECT` dev + subject: mint → list → buy(dev) → acquire → open → reconstruct (one-node custody) → play. +- Minimal UI: Create (mint), Store (list/detail/buy), Library (open), viewer (play). +- The `dkms_rail` oracle goes green in the dev lane. + +**Phase 2 — real chain-gated buy/open.** +- Land the subject-resolution rewire (`RequiredHomeLaunchToken` threading). +- Real Wallet + Chain integration for `buy`/`mint`; Home spend-confirmation dialog + + passkey step-up (money-verb authority). +- rights-provider evaluates live chain evidence; open gate uses real subject resolution. + +**Phase 3 — viewer session hardening.** +- Full open → session → close → sweeper lifecycle; expiry; zeroization proofs; media and + document viewers; scoped-output-only enforcement; refusal-not-oracle test. + +**Phase 4 — UI/UX polish and consolidation.** +- Consolidate to one canonical Store; Create app polish; Library lock/provenance chips; + full design-token pass; all states/copy per §5.4; accessibility bar for money verbs. + +**Phase 5 — atomic cutover and release gates.** +- Remove the provisional providers + DTOs atomically; align docs/state/tests (12); + entropy/WCI/verify gates; manual browser walkthrough of the whole journey. + +**Deferred past 0.7 (named, fail-closed):** resale/trade verbs, apps/games as sellable +assets, multi-segment media, DKG/rotation/re-share/revocation, PQ-hybrid custody, +multi-node production custody topology. + +--- + +## 8. Disposition vs PR #15 (reuse / re-derive / reject) + +Consistent with the "PR #15 disposition" in [PROTECTED_CONTENT.md](PROTECTED_CONTENT.md): + +- **Adopt and harden (UI/flows, re-anchored on canonical authority):** `creator`, + consolidated `marketplace`/`marketplace-content` → Store, `elacity-player`, + `ddrm-viewer`, `home-spend-prompt.js`, the `/api/market/*` and `/api/viewers/*` seams, and + the `dkms_rail` e2e as an oracle. +- **Re-derive at the canonical boundary (do not port wholesale):** custody durable shard + storage, rights evaluation, decrypt boundary, provider roles, Runtime open scenarios — + built on the canonical crates, not PR #15's internals. +- **Reject from the product path:** public aggregated `shares[]` metadata, capsule-owned + authority, raw-CEK operations, `rail_shim`/reference fallbacks, old `drm-provider` + orchestration, direct TCP/IP topology in capsules/contracts, and the standalone harness as + a product route. PR #15's `escrow.json` fixture is not canonical metadata. + +--- + +## 9. Testing and acceptance + +- **Behavioral oracle:** `dkms_rail`-style e2e proving `mint → Runtime authority/selection/ + audit → per-node custody release → reconstruction/decryption → play`, in the dev lane + first, then chain-gated. +- **Per-stage:** mint seal/commitment, listing decode, buy money-verb authority + drift + abort, acquire pin, open gate (owned/denied/nonexistent byte-identical 404), session + lifecycle + zeroization. +- **Negative/fail-closed:** tamper/expiry/wrong-binding rejection, required-plus-one + rejection, custody-unavailable explicit failure, refusal-not-oracle. +- **UI:** manual walkthrough of discover → buy (spend prompt + passkey) → play → close, plus + each defined state. +- **Stability:** clock-injected custody-provider process tests green on the CI runner class; + no wall-clock-coupled flakes. + +--- + +## 10. Risks and open decisions + +- **Scope risk:** the journey's product surface (commerce + viewer + Runtime orchestration) + is large relative to the extracted foundation; Phase 1's dev-lane slice is the de-risking + gate — if it is not green, do not add custody depth. +- **Blocker risk:** the subject-resolution rewire is cross-module; if it slips, 0.7 must + ship dev-lane buy explicitly labeled, not a silent half-path (11). +- **Consolidation decision:** one canonical Store (fold app-store in as fulfillment) must be + committed to early to avoid two storefronts (10). +- **Naming decision:** lock the product vocabulary in Phase 0 so docs, code, tests, and UI + agree (12/14). + +--- + +## 11. Coherence checklist (merge gate) + +- [ ] Every capsule in the journey requests only typed Runtime resources (3, 18). +- [ ] No capsule holds a signer, CEK, IV, RPC, or provider route (3, 15). +- [ ] One open gate, one buy verb, one mint verb; no fallbacks (10, 11). +- [ ] Money verbs always render the Home spend-confirmation dialog; what is displayed is + what is signed (16). +- [ ] Product copy uses human nouns; dKMS/dDRM stay in developer docs (14). +- [ ] All surfaces use `_shared` design tokens and canonical window chrome (17). +- [ ] Refusal is not an existence oracle; unauthorized steps fail closed with explanation + (11). +- [ ] Provisional providers removed atomically; docs/code/tests/state agree (10, 12). +- [ ] The `dkms_rail` behavioral oracle is green. From 610af84cfdca666349fb2e21f57032ea74e05480 Mon Sep 17 00:00:00 2001 From: Irzhy Ranaivoarivony Date: Tue, 1 Sep 2026 11:48:18 +0300 Subject: [PATCH 002/535] docs(agents): adopt Cursor Cloud environment notes from PR #21, refreshed for the 0.7 tree Carries the Cursor Cloud specific instructions section from PR #21 (cb80aab5, Anders Alm) with its stale 0.6-era claims corrected: toolchain pin now 1.91.0, the capsule test lane's shared target-capsules dir described, and the public-install gap paragraph made version-agnostic with state.md as the published-version truth. --- AGENTS.md | 44 +++++++++++++++++++ ...6-08-18-protected-content-journey-plan.md} | 12 ++--- 2 files changed, 50 insertions(+), 6 deletions(-) rename docs/{PROTECTED_CONTENT_JOURNEY_PLAN.md => audits/2026-08-18-protected-content-journey-plan.md} (97%) diff --git a/AGENTS.md b/AGENTS.md index 96a63d6ae..fda560473 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -346,3 +346,47 @@ with Browser engine isolation or product media proof. Cosmopolitan Libc may be researched for small C/C++ helper binaries, but it is not a drop-in answer for Rust workspace packaging, Chromium, WebView, GPU/audio, microVM isolation, or `.dmg` distribution. + +## Cursor Cloud specific instructions + +This section is durable guidance for Cloud Agents. The startup update script +already installs `just` (`cargo install just`) and the `wasm32-unknown-unknown` +Rust target; do not re-document dependency installation here. + +- Toolchain: Rust is pinned by [rust-toolchain.toml](rust-toolchain.toml) + (1.91.0 with `rustfmt`/`clippy`) and `rustup` auto-syncs it on first cargo + use. `wasm32-unknown-unknown` is the capsule Component target; the release + scripts also add it on demand via `ensure_rust_target_installed`. +- Core gate has no JS/Python package deps: the `.mjs` checks in `just verify` + (`home-entropy-check.mjs`, `browser-entropy-check.mjs`, + `browser-window-close-handshake.test.mjs`, etc.) and the Python smokes use + only stdlib/`node --test`. `playwright` appears only as a scanned string in + entropy checks and as an import in Browser/GUI headless smokes that are NOT + part of the core gate, so no `npm install` is needed for build/lint/test. +- Build/lint/test run from the repo root (the recipes `cd elastos` + themselves): `just build`, `just lint`, `just test`. A cold `just build` is + slow (runtime crate ~6 min). Each provider under `capsules/` is its own cargo + project; `build.sh --all` and `just home-frontdoor-smoke` build per-capsule + `target/` dirs, while `just test-capsules` shares one `target-capsules/` dir + so common dependencies compile once across the capsule suites. +- Running the app from source: stage the built `localhost-provider` into + `$XDG_DATA_HOME/elastos/bin/` and write a `components.json` beside the data + dir whose `external.localhost-provider.platforms..checksum` matches + that binary's `sha256:`, then `elastos serve --addr 127.0.0.1:`. The + runtime verifies every staged component against that checksum (no dev bypass). + It then comes up healthy (`/api/health` → `{"status":"ok",...}`), Carrier P2P + online; authenticate via `POST /api/auth/attach` using `attach_secret` from + `$XDG_DATA_HOME/elastos/runtime-coords.json`. Self-contained commands + (`elastos identity show`, `elastos identity nickname set`, `elastos init`) + work directly against an isolated `HOME`/`XDG_DATA_HOME`. +- Known release-gated gap (not an environment bug): `just home-frontdoor-smoke` + and the `public-install-*` smokes run `scripts/install.sh`, which downloads + the PUBLISHED release binary from `https://elastos.elacitylabs.com`. Whenever + the published binary trails this dev tree's `install.sh` contract (newer + subcommands, setup profiles), the installed-path proofs fail; + [state.md](state.md) ("Public Install Truth") records the currently published + version and its compatibility. Use the source `elastos serve` path above to + run the runtime locally instead. +- No KVM: crosvm/microVM paths (Browser VM, full-screen chat microVM) warn and + fail closed here, so product Browser proof and microVM chat are not runnable + in this environment. diff --git a/docs/PROTECTED_CONTENT_JOURNEY_PLAN.md b/docs/audits/2026-08-18-protected-content-journey-plan.md similarity index 97% rename from docs/PROTECTED_CONTENT_JOURNEY_PLAN.md rename to docs/audits/2026-08-18-protected-content-journey-plan.md index d70bbaef7..e6d80b6f3 100644 --- a/docs/PROTECTED_CONTENT_JOURNEY_PLAN.md +++ b/docs/audits/2026-08-18-protected-content-journey-plan.md @@ -3,11 +3,11 @@ Status: proposal. This plan describes how to take the canonical protected-content extraction (`feat/protected-content-*`) from a trust-core foundation to the full end-to-end user journey — **mint → sell → buy → play** — with first-class design and -UI/UX, coherent with [PRINCIPLES.md](../PRINCIPLES.md) and the canonical architecture in -[PROTECTED_CONTENT.md](PROTECTED_CONTENT.md). +UI/UX, coherent with [PRINCIPLES.md](../../PRINCIPLES.md) and the canonical architecture in +[PROTECTED_CONTENT.md](../PROTECTED_CONTENT.md). It is written to complement, not replace, the "PR #15 disposition" already recorded in -[PROTECTED_CONTENT.md](PROTECTED_CONTENT.md): PR #15 (`feat/dkms-esp-port`) is research and +[PROTECTED_CONTENT.md](../PROTECTED_CONTENT.md): PR #15 (`feat/dkms-esp-port`) is research and behavior evidence; this plan is how the canonical line reaches the same user-visible capability on the ESP substrate without inheriting PR #15's rejected patterns. @@ -35,7 +35,7 @@ DKG/rotation/revocation, PQ-hybrid custody) is explicitly deferred and named in ## 2. Vision alignment (this is the coherence contract) -Every stage below is designed against [PRINCIPLES.md](../PRINCIPLES.md). The mapping is the +Every stage below is designed against [PRINCIPLES.md](../../PRINCIPLES.md). The mapping is the acceptance criterion for "aligned with the ElastOS vision", not decoration: | Principle | How this journey honors it | @@ -58,7 +58,7 @@ wrong. ## 3. Architecture spine (reused, not reinvented) -The canonical path from [PROTECTED_CONTENT.md](PROTECTED_CONTENT.md) is the backbone for +The canonical path from [PROTECTED_CONTENT.md](../PROTECTED_CONTENT.md) is the backbone for all four stages: ``` @@ -288,7 +288,7 @@ multi-node production custody topology. ## 8. Disposition vs PR #15 (reuse / re-derive / reject) -Consistent with the "PR #15 disposition" in [PROTECTED_CONTENT.md](PROTECTED_CONTENT.md): +Consistent with the "PR #15 disposition" in [PROTECTED_CONTENT.md](../PROTECTED_CONTENT.md): - **Adopt and harden (UI/flows, re-anchored on canonical authority):** `creator`, consolidated `marketplace`/`marketplace-content` → Store, `elacity-player`, From c511b1339ace5d5295f80cff6c25f1f524149736 Mon Sep 17 00:00:00 2001 From: Irzhy Ranaivoarivony Date: Wed, 2 Sep 2026 12:57:39 +0300 Subject: [PATCH 003/535] feat(build): share intermediate build artifacts across all workspaces Add a root .cargo/config.toml with build.build-dir = "target-build" (stable since cargo 1.91, which rust-toolchain.toml pins): dependency artifacts, fingerprints, and incremental state from every workspace in the repo (elastos/, capsules/*, elastos/tools/*) now compile once into one shared dir, while final artifacts stay in each workspace's own target dir so every existing target/{debug,release} path keeps working. - CI caches the shared dir in all five jobs (rust-cache cache-directories for the elastos jobs, the actions/cache path list for test-capsules) under bumped key namespaces so pre-split caches are not restored as dead weight - the CI-replica tar recipes exclude target-build for cold accuracy - clean.sh reclaims target-build and target-capsules - .gitignore covers /target-build/ --- .cargo/config.toml | 10 ++++++++++ .github/workflows/ci.yml | 35 ++++++++++++++++++++++------------- .gitignore | 2 ++ justfile | 23 +++++++++++++---------- scripts/build/clean.sh | 10 ++++++---- 5 files changed, 53 insertions(+), 27 deletions(-) create mode 100644 .cargo/config.toml diff --git a/.cargo/config.toml b/.cargo/config.toml new file mode 100644 index 000000000..e28990c43 --- /dev/null +++ b/.cargo/config.toml @@ -0,0 +1,10 @@ +[build] +# Shared intermediate build dir for every workspace in this repo (elastos/, +# capsules/*, elastos/tools/*): dependency artifacts, fingerprints, and +# incremental state compile once and land in one CI-cacheable path. +# Final artifacts (binaries, test executables' uplifted outputs) stay in each +# workspace's own target dir, so every existing target/{debug,release} path +# keeps working. Config-relative, so this resolves to /target-build +# from any subdirectory. Requires cargo >= 1.91 (build-dir stabilization), +# which rust-toolchain.toml pins. +build-dir = "target-build" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7064aaaec..32c5cb986 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -68,7 +68,9 @@ jobs: - uses: Swatinem/rust-cache@v2 with: workspaces: elastos - prefix-key: ${{ runner.os }}-elastos + prefix-key: ${{ runner.os }}-elastos-bd + # Shared intermediate build dir (root .cargo/config.toml build-dir) + cache-directories: target-build # clippy runs the full type-check pass, so a separate `cargo check` # would be redundant. - name: cargo clippy @@ -88,7 +90,9 @@ jobs: - uses: Swatinem/rust-cache@v2 with: workspaces: elastos - prefix-key: ${{ runner.os }}-elastos-test + prefix-key: ${{ runner.os }}-elastos-test-bd + # Shared intermediate build dir (root .cargo/config.toml build-dir) + cache-directories: target-build # The three provider workspaces are release-built from cold on every # run otherwise; with the cache restored, cargo only re-fingerprints # the workspace members and reuses every dependency artifact. @@ -99,9 +103,9 @@ jobs: capsules/protected-content-protect-provider/target capsules/protected-content-decrypt-provider/target capsules/custody-provider/target - key: providers-target-${{ runner.os }}-${{ hashFiles('capsules/protected-content-protect-provider/Cargo.lock', 'capsules/protected-content-decrypt-provider/Cargo.lock', 'capsules/custody-provider/Cargo.lock', 'rust-toolchain.toml') }} + key: providers-bd-target-${{ runner.os }}-${{ hashFiles('capsules/protected-content-protect-provider/Cargo.lock', 'capsules/protected-content-decrypt-provider/Cargo.lock', 'capsules/custody-provider/Cargo.lock', 'rust-toolchain.toml') }} restore-keys: | - providers-target-${{ runner.os }}- + providers-bd-target-${{ runner.os }}- - name: build provider process binaries for tests run: | cargo build --release --manifest-path capsules/protected-content-protect-provider/Cargo.toml --target-dir capsules/protected-content-protect-provider/target @@ -133,9 +137,9 @@ jobs: with: # Registry/index cache only: this job never builds the elastos # workspace, and the capsule build artifacts live in the shared - # target-capsules dir cached below. + # intermediate build dir cached below. workspaces: elastos - prefix-key: ${{ runner.os }}-capsules + prefix-key: ${{ runner.os }}-capsules-bd cache-targets: false # The capsule workspaces overlap almost entirely in dependencies, so # `just test-capsules` builds them into one shared target dir; caching @@ -145,20 +149,21 @@ jobs: - name: cache capsule build artifacts uses: actions/cache@v4 with: - path: target-capsules - key: capsules-target-${{ runner.os }}-${{ hashFiles('capsules/*/Cargo.lock', 'rust-toolchain.toml') }} + # The shared intermediates (root .cargo/config.toml build-dir); + # final artifacts stay in each capsule's own small target dir. + path: target-build + key: capsules-bd-target-${{ runner.os }}-${{ hashFiles('capsules/*/Cargo.lock', 'rust-toolchain.toml') }} restore-keys: | - capsules-target-${{ runner.os }}- + capsules-bd-target-${{ runner.os }}- # The elastos workspace suite never builds or tests own-workspace # capsules; this mirrors `just test-capsules`. - name: test own-workspace capsules run: | set -euo pipefail - root="$PWD" for lock in capsules/*/Cargo.lock; do capsule="$(dirname "$lock")" echo "== testing $capsule ==" - (cd "$capsule" && cargo test --target-dir "${root}/target-capsules") + (cd "$capsule" && cargo test) done # The source-home jobs run only once the test phase is green, in parallel @@ -187,7 +192,9 @@ jobs: - uses: Swatinem/rust-cache@v2 with: workspaces: elastos - prefix-key: ${{ runner.os }}-${{ runner.arch }}-elastos + prefix-key: ${{ runner.os }}-${{ runner.arch }}-elastos-bd + # Shared intermediate build dir (root .cargo/config.toml build-dir) + cache-directories: target-build - name: require 16 GiB free disk run: | set -euo pipefail @@ -257,7 +264,9 @@ jobs: - uses: Swatinem/rust-cache@v2 with: workspaces: elastos - prefix-key: ${{ runner.os }}-elastos + prefix-key: ${{ runner.os }}-elastos-bd + # Shared intermediate build dir (root .cargo/config.toml build-dir) + cache-directories: target-build # dist.ipfs.tech is intermittently slow or unreachable from GitHub's # macOS runners, and setup's downloader makes a single 600s attempt. # Cache the verified release tarball and pre-seed bin/kubo so setup diff --git a/.gitignore b/.gitignore index a89fb7b0c..c7886b866 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,8 @@ # Rust **/target/ /target-capsules/ +# Shared intermediate build dir (root .cargo/config.toml build-dir) +/target-build/ **/*.rs.bk # Main Cargo.lock committed for reproducible builds; negate the ignore !elastos/Cargo.lock diff --git a/justfile b/justfile index 74fa5a6b5..af8c87702 100644 --- a/justfile +++ b/justfile @@ -51,15 +51,18 @@ test-elastos *args: prepare-providers test-capsules: #!/usr/bin/env bash set -euo pipefail - # One shared target dir: the capsule workspaces overlap almost entirely in - # dependencies, so sharing compiles each dep once instead of ~25 times and - # gives CI a single cacheable path. Cargo fingerprints keep per-workspace - # correctness; the loop is sequential, so there is no lock contention. - root="$PWD" + # Dependency artifacts are shared across all workspaces via the root + # .cargo/config.toml build-dir (/target-build), so each dep compiles + # once instead of ~25 times. Deliberately NO --target-dir pin: with a + # shared build-dir, cargo reuses compiled test executables across + # invocations without re-baking their CARGO_BIN_EXE_* paths, so every + # invocation style (just, plain cargo test, rust-analyzer, CI) must agree + # on each workspace's default target dir or process tests spawn stale + # binary paths. for lock in capsules/*/Cargo.lock; do capsule="$(dirname "$lock")" echo "== testing $capsule ==" - (cd "$capsule" && cargo test --target-dir "${root}/target-capsules") + (cd "$capsule" && cargo test) done # Run the workspace suite and every own-workspace capsule suite @@ -70,7 +73,7 @@ test: test-elastos test-capsules # nodejs matches the ubuntu-latest runner, where node is preinstalled and # elastos-server integration tests spawn it. ci-test-elastos: - tar --no-xattrs --no-mac-metadata --no-fflags --exclude='.git' --exclude='target' --exclude='target-capsules' --exclude='*/target' --exclude='capsules/*/target' -cf - . | \ + tar --no-xattrs --no-mac-metadata --no-fflags --exclude='.git' --exclude='target' --exclude='target-capsules' --exclude='target-build' --exclude='*/target' --exclude='capsules/*/target' -cf - . | \ docker run --rm -i -e RUSTFLAGS="-D warnings" -e CARGO_TERM_COLOR=always rust:1.91-bookworm bash -c '\ mkdir /w && tar -xf - -C /w && cd /w && cargo --version >/dev/null && \ apt-get update -qq >/dev/null && apt-get install -y -qq nodejs >/dev/null && node --version && \ @@ -87,7 +90,7 @@ ci-test-elastos: # Accurate local replica of the CI test-capsules job (same container recipe). ci-test-capsules: - tar --no-xattrs --no-mac-metadata --no-fflags --exclude='.git' --exclude='target' --exclude='target-capsules' --exclude='*/target' --exclude='capsules/*/target' -cf - . | \ + tar --no-xattrs --no-mac-metadata --no-fflags --exclude='.git' --exclude='target' --exclude='target-capsules' --exclude='target-build' --exclude='*/target' --exclude='capsules/*/target' -cf - . | \ docker run --rm -i -e RUSTFLAGS="-D warnings" -e CARGO_TERM_COLOR=always rust:1.91-bookworm bash -c '\ mkdir /w && tar -xf - -C /w && cd /w && cargo --version >/dev/null && \ useradd -m ci && chown -R ci:ci /w && \ @@ -95,14 +98,14 @@ ci-test-capsules: for lock in capsules/*/Cargo.lock; do \ capsule=\$(dirname \$lock); \ echo \"== testing \$capsule ==\"; \ - (cd \$capsule && cargo test --target-dir /w/target-capsules) || exit 1; \ + (cd \$capsule && cargo test) || exit 1; \ done"' # Accurate local replica of the CI source-home-linux job. arch selects the # matrix leg: arm64 = ubuntu-24.04-arm (native on Apple silicon), # amd64 = ubuntu-latest (emulated, much slower). ci-source-home-linux arch='arm64': - tar --no-xattrs --no-mac-metadata --no-fflags --exclude='.git' --exclude='target' --exclude='target-capsules' --exclude='*/target' --exclude='capsules/*/target' -cf - . | \ + tar --no-xattrs --no-mac-metadata --no-fflags --exclude='.git' --exclude='target' --exclude='target-capsules' --exclude='target-build' --exclude='*/target' --exclude='capsules/*/target' -cf - . | \ docker run --rm -i --platform linux/{{arch}} -e CARGO_TERM_COLOR=never rust:1.91-bookworm bash -c '\ mkdir /w && tar -xf - -C /w && cd /w && \ apt-get update -qq >/dev/null && apt-get install -y -qq coturn e2fsprogs ffmpeg nodejs git jq >/dev/null && \ diff --git a/scripts/build/clean.sh b/scripts/build/clean.sh index 08a977c5a..4d2957207 100644 --- a/scripts/build/clean.sh +++ b/scripts/build/clean.sh @@ -77,8 +77,9 @@ if [ "$DRY_RUN" = true ] || [ "$CLEAN_DATA_ONLY" = false ]; then BUILD_SIZE=$((BUILD_SIZE + s)) fi - # Capsule targets - for dir in elastos/capsules/*/target capsules/*/target; do + # Capsule targets, the shared intermediate build dir, and the shared + # capsule test dir + for dir in elastos/capsules/*/target capsules/*/target target-build target-capsules; do if [ -d "$dir" ]; then BUILD_DIRS+=("$dir") s=$(du -sm "$dir" 2>/dev/null | cut -f1) @@ -135,8 +136,9 @@ if [ "$CLEAN_DATA_ONLY" = false ]; then ok "elastos/target" fi - # Capsule targets - for dir in elastos/capsules/*/target capsules/*/target; do + # Capsule targets, the shared intermediate build dir, and the shared + # capsule test dir + for dir in elastos/capsules/*/target capsules/*/target target-build target-capsules; do if [ -d "$dir" ]; then rm -rf "$dir" ok "$dir" From 9c0706e89c9be2996d89ee05971b735c65ad6ac0 Mon Sep 17 00:00:00 2001 From: Irzhy Ranaivoarivony Date: Wed, 2 Sep 2026 14:04:18 +0300 Subject: [PATCH 004/535] chore(protected-content): prepare provisioning dependencies and document the operator surface Promote custody-provider from dev-dependency to dependency so operator provisioning calls the same provision_state_root the provider binary uses; refresh object-provider's lockfile and build-script permissions for the shared-artifacts layout; document the elastos protected-content-config command surface in docs/PROTECTED_CONTENT.md. --- capsules/object-provider/Cargo.lock | 265 ++++++++++++++++++++--- docs/PROTECTED_CONTENT.md | 10 + elastos/crates/elastos-server/Cargo.toml | 2 +- scripts/build/build-llama-server.sh | 0 scripts/build/build-vm-smoke-rootfs.sh | 0 scripts/build/clean.sh | 0 6 files changed, 245 insertions(+), 32 deletions(-) mode change 100644 => 100755 scripts/build/build-llama-server.sh mode change 100644 => 100755 scripts/build/build-vm-smoke-rootfs.sh mode change 100644 => 100755 scripts/build/clean.sh diff --git a/capsules/object-provider/Cargo.lock b/capsules/object-provider/Cargo.lock index c8edcb267..4a67c0033 100644 --- a/capsules/object-provider/Cargo.lock +++ b/capsules/object-provider/Cargo.lock @@ -52,7 +52,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" dependencies = [ "crypto-common 0.1.7", - "generic-array", + "generic-array 0.14.7", ] [[package]] @@ -64,6 +64,7 @@ dependencies = [ "cfg-if", "cipher", "cpufeatures 0.2.17", + "zeroize", ] [[package]] @@ -78,6 +79,7 @@ dependencies = [ "ctr", "ghash", "subtle", + "zeroize", ] [[package]] @@ -140,7 +142,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -151,7 +153,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -494,7 +496,7 @@ version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" dependencies = [ - "generic-array", + "generic-array 0.14.7", ] [[package]] @@ -832,6 +834,12 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "cpubits" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15b85f9c39137c3a891689859392b1bd49812121d0d61c9caf00d46ed5ce06ae" + [[package]] name = "cpufeatures" version = "0.2.17" @@ -1064,19 +1072,35 @@ version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" dependencies = [ - "generic-array", + "generic-array 0.14.7", "rand_core 0.6.4", "subtle", "zeroize", ] +[[package]] +name = "crypto-bigint" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a52aa3fcda4e6302a9f48734f234d35d4721b96f8fe07d073f07ce9df4f0271" +dependencies = [ + "cpubits", + "ctutils", + "hybrid-array", + "num-traits", + "rand_core 0.10.1", + "serdect", + "subtle", + "zeroize", +] + [[package]] name = "crypto-common" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" dependencies = [ - "generic-array", + "generic-array 0.14.7", "rand_core 0.6.4", "typenum", ] @@ -1087,6 +1111,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" dependencies = [ + "getrandom 0.4.3", "hybrid-array", "rand_core 0.10.1", ] @@ -1107,6 +1132,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" dependencies = [ "cmov", + "subtle", ] [[package]] @@ -1154,6 +1180,23 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "custody-provider" +version = "0.1.0" +dependencies = [ + "ed25519-dalek 2.2.0", + "elastos-protected-content-contracts", + "elastos-protected-content-custody", + "elastos-protected-content-provider-contracts", + "elastos-protected-content-rights", + "getrandom 0.2.17", + "nix", + "serde", + "serde_json", + "sha2 0.10.9", + "zeroize", +] + [[package]] name = "data-encoding" version = "2.11.1" @@ -1220,7 +1263,7 @@ dependencies = [ "asn1-rs", "displaydoc", "nom", - "num-bigint", + "num-bigint 0.4.8", "num-traits", "rusticata-macros", ] @@ -1424,7 +1467,7 @@ checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" dependencies = [ "der 0.7.10", "digest 0.10.7", - "elliptic-curve", + "elliptic-curve 0.13.8", "rfc6979", "signature 2.2.0", "spki 0.7.3", @@ -1631,6 +1674,32 @@ dependencies = [ "x-wing", ] +[[package]] +name = "elastos-protected-content-custody" +version = "0.7.0" +dependencies = [ + "aes", + "aes-gcm", + "ctr", + "ed25519-dalek 2.2.0", + "elastos-protected-content-contracts", + "elastos-protected-content-provider-contracts", + "ghash", + "hkdf", + "hpke", + "nix", + "rand 0.10.2", + "rand 0.9.5", + "rand_chacha 0.3.1", + "rand_core 0.6.4", + "sha2 0.10.9", + "subtle", + "thiserror 1.0.69", + "vsss-rs", + "x-wing", + "zeroize", +] + [[package]] name = "elastos-protected-content-provider-contracts" version = "0.7.0" @@ -1712,6 +1781,7 @@ dependencies = [ "bs58", "cid", "clap", + "custody-provider", "data-encoding", "dirs", "distributed-topic-tracker", @@ -1813,20 +1883,51 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" dependencies = [ "base16ct 0.2.0", - "crypto-bigint", + "crypto-bigint 0.5.5", "digest 0.10.7", - "ff", - "generic-array", - "group", + "ff 0.13.1", + "generic-array 0.14.7", + "group 0.13.0", "hkdf", "pem-rfc7468 0.7.0", "pkcs8 0.10.2", "rand_core 0.6.4", - "sec1", + "sec1 0.7.3", "subtle", "zeroize", ] +[[package]] +name = "elliptic-curve" +version = "0.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d65aa39b3a5c1c9c1b745c9a019234bb7a21b77abcb4f4d266d706e2d577d65" +dependencies = [ + "base16ct 1.0.0", + "crypto-bigint 0.7.5", + "crypto-common 0.2.2", + "ff 0.14.0", + "group 0.14.0", + "hybrid-array", + "pkcs8 0.11.0", + "rand_core 0.10.1", + "sec1 0.8.1", + "subtle", + "zeroize", +] + +[[package]] +name = "elliptic-curve-tools" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a0d5e534f103b079a71ef1d66c6e62c89413f1b62709ca11f744429b4afe5b4" +dependencies = [ + "elliptic-curve 0.14.1", + "heapless 0.9.3", + "serde", + "serdect", +] + [[package]] name = "embedded-io" version = "0.4.0" @@ -1872,7 +1973,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1897,6 +1998,16 @@ dependencies = [ "subtle", ] +[[package]] +name = "ff" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1f686ab92a9fb0eaf188f6c6c87b89490baa6fdb0db4544ba4dc47f7942489f" +dependencies = [ + "rand_core 0.10.1", + "subtle", +] + [[package]] name = "fiat-crypto" version = "0.2.9" @@ -2170,6 +2281,16 @@ dependencies = [ "zeroize", ] +[[package]] +name = "generic-array" +version = "1.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "337d46834ee672ab3e48caca2cb0c78cc174fb12b3a68d0d88f99a0519a5e36e" +dependencies = [ + "rustversion", + "typenum", +] + [[package]] name = "getrandom" version = "0.2.17" @@ -2219,6 +2340,7 @@ checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" dependencies = [ "opaque-debug", "polyval", + "zeroize", ] [[package]] @@ -2250,11 +2372,22 @@ version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" dependencies = [ - "ff", + "ff 0.13.1", "rand_core 0.6.4", "subtle", ] +[[package]] +name = "group" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fd1a1c7a5206c5b7a3f5a0d7ccd3ff85d0c8f5133d62a02680255b0004af5f4" +dependencies = [ + "ff 0.14.0", + "rand_core 0.10.1", + "subtle", +] + [[package]] name = "h2" version = "0.4.19" @@ -2294,6 +2427,15 @@ dependencies = [ "byteorder", ] +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + [[package]] name = "hashbrown" version = "0.15.5" @@ -2322,13 +2464,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cdc6457c0eb62c71aac4bc17216026d8410337c4126773b9c5daba343f17964f" dependencies = [ "atomic-polyfill", - "hash32", + "hash32 0.2.1", "rustc_version", "serde", "spin 0.9.9", "stable_deref_trait", ] +[[package]] +name = "heapless" +version = "0.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25ba4bd83f9415b58b4ed8dc5714c76e626a105be4646c02630ad730ad3b5aa4" +dependencies = [ + "hash32 0.3.1", + "stable_deref_trait", +] + [[package]] name = "heck" version = "0.5.0" @@ -2446,7 +2598,7 @@ dependencies = [ "aes-gcm", "chacha20poly1305", "digest 0.10.7", - "generic-array", + "generic-array 0.14.7", "hkdf", "hmac", "p256", @@ -2515,7 +2667,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" dependencies = [ "ctutils", + "subtle", "typenum", + "zeroize", ] [[package]] @@ -2757,7 +2911,7 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" dependencies = [ - "generic-array", + "generic-array 0.14.7", ] [[package]] @@ -3166,7 +3320,7 @@ checksum = "f6e3919bbaa2945715f0bb6d3934a173d1e9a59ac23767fbaaef277265a7411b" dependencies = [ "cfg-if", "ecdsa", - "elliptic-curve", + "elliptic-curve 0.13.8", "once_cell", "sha2 0.10.9", "signature 2.2.0", @@ -3433,6 +3587,7 @@ dependencies = [ "module-lattice", "rand_core 0.10.1", "sha3 0.11.0", + "zeroize", ] [[package]] @@ -3444,6 +3599,7 @@ dependencies = [ "ctutils", "hybrid-array", "num-traits", + "zeroize", ] [[package]] @@ -3773,7 +3929,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -3786,6 +3942,16 @@ dependencies = [ "num-traits", ] +[[package]] +name = "num-bigint" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93e7820bc0a80a0238e650327316f929ba18d5be054b647490a3a6a339f3e7c0" +dependencies = [ + "num-integer", + "num-traits", +] + [[package]] name = "num-conv" version = "0.2.2" @@ -3993,7 +4159,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" dependencies = [ "ecdsa", - "elliptic-curve", + "elliptic-curve 0.13.8", "primeorder", "sha2 0.10.9", ] @@ -4233,7 +4399,7 @@ dependencies = [ "cobs", "embedded-io 0.4.0", "embedded-io 0.6.1", - "heapless", + "heapless 0.7.17", "postcard-derive", "serde", ] @@ -4290,7 +4456,7 @@ version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" dependencies = [ - "elliptic-curve", + "elliptic-curve 0.13.8", ] [[package]] @@ -4401,7 +4567,7 @@ dependencies = [ "once_cell", "socket2", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4745,7 +4911,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4804,7 +4970,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4875,12 +5041,26 @@ checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" dependencies = [ "base16ct 0.2.0", "der 0.7.10", - "generic-array", + "generic-array 0.14.7", "pkcs8 0.10.2", "subtle", "zeroize", ] +[[package]] +name = "sec1" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56d437c2f19203ce5f7122e507831de96f3d2d4d3be5af44a0b0a09d8a80e4d" +dependencies = [ + "base16ct 1.0.0", + "ctutils", + "der 0.8.1", + "hybrid-array", + "subtle", + "zeroize", +] + [[package]] name = "security-framework" version = "3.7.0" @@ -5212,7 +5392,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -5420,10 +5600,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.3.4", + "getrandom 0.4.3", "once_cell", "rustix", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -5986,6 +6166,27 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" +[[package]] +name = "vsss-rs" +version = "6.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bfc736cfd88115aedb95ba84bc2d428fe351e92a56f69fce090af301402d91" +dependencies = [ + "crypto-bigint 0.7.5", + "elliptic-curve 0.14.1", + "elliptic-curve-tools", + "ff 0.14.0", + "generic-array 1.4.5", + "hex", + "hybrid-array", + "num-bigint 0.5.1", + "num-traits", + "rand_core 0.10.1", + "serde", + "subtle", + "zeroize", +] + [[package]] name = "walkdir" version = "2.5.0" @@ -6512,7 +6713,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -7043,6 +7244,7 @@ dependencies = [ "sha3 0.12.0", "shake", "x25519-dalek 3.0.0", + "zeroize", ] [[package]] @@ -7065,6 +7267,7 @@ checksum = "e7e8131a03190127fb2263afc72b322ecadae46b6ff8c6f399ff5d02f5559af6" dependencies = [ "curve25519-dalek 5.0.0", "rand_core 0.10.1", + "zeroize", ] [[package]] diff --git a/docs/PROTECTED_CONTENT.md b/docs/PROTECTED_CONTENT.md index f3b362c36..004c79198 100644 --- a/docs/PROTECTED_CONTENT.md +++ b/docs/PROTECTED_CONTENT.md @@ -178,6 +178,16 @@ configuration passed admission. Runtime startup, signed custody validation, live Chain evidence, replication and repair, and mint-buy-play remain active proof. +Operator provisioning of that configuration is one explicit command surface, +`elastos protected-content-config`. It creates the policy authority key, +provisions each custody host's inactive state root and exports its node +descriptor, prints the Runtime operation issuer custody hosts must trust, +assembles and signs the owner-only 2-of-3 custody composition from three node +descriptors, and installs the private multi-source Chain configuration. A +generated composition or Chain configuration is proven against the Runtime +loader before success is reported, and an installed composition can be +re-verified in place. The installer never creates this operator state. + The macOS and Linux source-home restart helpers validate the installation receipt and current clean source before any stop or migration. They select only the stable installed Runtime, own one exact PID file, stop only the proven diff --git a/elastos/crates/elastos-server/Cargo.toml b/elastos/crates/elastos-server/Cargo.toml index 5a25bb03c..b9601c07b 100644 --- a/elastos/crates/elastos-server/Cargo.toml +++ b/elastos/crates/elastos-server/Cargo.toml @@ -114,6 +114,7 @@ distributed-topic-tracker = "=0.3.5" futures-lite = "2.6" getrandom = "0.2" data-encoding = "2.6" +custody-provider = { path = "../../../capsules/custody-provider" } [dev-dependencies] tokio = { version = "1.0", features = ["full", "test-util"] } @@ -123,7 +124,6 @@ wasm-encoder = "=0.236.1" elastos-auth = { path = "../elastos-auth" } elastos-protected-content-custody = { path = "../elastos-protected-content-custody" } x-wing = { version = "=0.1.0", features = ["hazmat", "zeroize"] } -custody-provider = { path = "../../../capsules/custody-provider" } [features] default = [] diff --git a/scripts/build/build-llama-server.sh b/scripts/build/build-llama-server.sh old mode 100644 new mode 100755 diff --git a/scripts/build/build-vm-smoke-rootfs.sh b/scripts/build/build-vm-smoke-rootfs.sh old mode 100644 new mode 100755 diff --git a/scripts/build/clean.sh b/scripts/build/clean.sh old mode 100644 new mode 100755 From 4d688cc5aef73dc0d2f88e81b4c4a4fb34744720 Mon Sep 17 00:00:00 2001 From: Irzhy Ranaivoarivony Date: Wed, 2 Sep 2026 14:04:19 +0300 Subject: [PATCH 005/535] feat(protected-content): provision installed custody and chain prerequisites One explicit operator command surface, elastos protected-content-config: create the policy authority key; provision each custody host's inactive state root (the owner-only directory boot registration requires but nothing created) and export its node descriptor; print the Runtime issuer custody hosts must trust; assemble and sign the owner-only 2-of-3 custody composition from three descriptors with canonical ordering, derived operator/failure-domain ids, and random owner state roots; and install the private multi-source Chain configuration with the deployment-proven Base gateway and selector defaults, enforcing the documented 2..=5 distinct-origin evidence rule. Generated files are proven against the Runtime loaders before success is reported, and a group-readable data root stays provisionable since owner-only modes are enforced from protected-content down. Boot now wires reconcile_runtime_custody_viewers_after_decrypt_registration into the decrypt-registration success arm through a warn-and-continue hook, settling viewer records stranded before decrypt came up, and names each absent protected-content provider binary instead of skipping it silently. --- .../src/collaboration_config.rs | 13 +- elastos/crates/elastos-server/src/lib.rs | 1 + elastos/crates/elastos-server/src/main.rs | 9 + .../src/protected_content_config.rs | 1568 +++++++++++++++++ .../src/protected_content_runtime.rs | 66 +- .../src/protected_content_runtime/tests.rs | 150 ++ .../crates/elastos-server/src/server_infra.rs | 31 +- 7 files changed, 1806 insertions(+), 32 deletions(-) create mode 100644 elastos/crates/elastos-server/src/protected_content_config.rs diff --git a/elastos/crates/elastos-server/src/collaboration_config.rs b/elastos/crates/elastos-server/src/collaboration_config.rs index 40b5481c2..ff90a552c 100644 --- a/elastos/crates/elastos-server/src/collaboration_config.rs +++ b/elastos/crates/elastos-server/src/collaboration_config.rs @@ -282,7 +282,11 @@ fn read_authority_key(path: &Path) -> anyhow::Result { Ok(SigningKey::from_bytes(&key)) } -fn read_owner_only_file(path: &Path, max_bytes: usize, label: &str) -> anyhow::Result> { +pub(crate) fn read_owner_only_file( + path: &Path, + max_bytes: usize, + label: &str, +) -> anyhow::Result> { validate_owner_only_parent(path)?; let metadata = fs::symlink_metadata(path).with_context(|| format!("failed to inspect {label}"))?; @@ -312,7 +316,7 @@ fn read_owner_only_file(path: &Path, max_bytes: usize, label: &str) -> anyhow::R Ok(bytes) } -fn create_owner_only_file(path: &Path, bytes: &[u8], label: &str) -> anyhow::Result<()> { +pub(crate) fn create_owner_only_file(path: &Path, bytes: &[u8], label: &str) -> anyhow::Result<()> { let parent = validate_owner_only_parent(path)?; if fs::symlink_metadata(path).is_ok() { anyhow::bail!("{label} output already exists"); @@ -400,7 +404,10 @@ fn sha256_label(bytes: &[u8]) -> String { format!("sha256:{}", hex::encode(Sha256::digest(bytes))) } -fn write_receipt(output: &mut dyn Write, receipt: &impl Serialize) -> anyhow::Result<()> { +pub(crate) fn write_receipt( + output: &mut dyn Write, + receipt: &impl Serialize, +) -> anyhow::Result<()> { serde_json::to_writer(&mut *output, receipt)?; output.write_all(b"\n")?; Ok(()) diff --git a/elastos/crates/elastos-server/src/lib.rs b/elastos/crates/elastos-server/src/lib.rs index 78a0b6db3..6a6570eca 100644 --- a/elastos/crates/elastos-server/src/lib.rs +++ b/elastos/crates/elastos-server/src/lib.rs @@ -50,6 +50,7 @@ pub mod local_http; pub mod notifications; pub mod operator_control; pub mod ownership; +pub mod protected_content_config; pub mod protected_content_runtime; pub mod provider_resource; pub mod resource_bridge; diff --git a/elastos/crates/elastos-server/src/main.rs b/elastos/crates/elastos-server/src/main.rs index 4baf44833..5653e165a 100755 --- a/elastos/crates/elastos-server/src/main.rs +++ b/elastos/crates/elastos-server/src/main.rs @@ -312,6 +312,10 @@ enum Commands { #[command(subcommand)] CollaborationConfig(elastos_server::collaboration_config::CollaborationConfigCommand), + /// Offline operator provisioning for protected-content custody and chain prerequisites + #[command(subcommand)] + ProtectedContentConfig(elastos_server::protected_content_config::ProtectedContentConfigCommand), + /// Show and manage the local DID-backed profile #[command(subcommand)] Identity(IdentityCommand), @@ -1302,6 +1306,11 @@ async fn main() -> anyhow::Result<()> { elastos_server::collaboration_config::run_collaboration_config_command(cmd).await?; } + Commands::ProtectedContentConfig(cmd) => { + elastos_server::protected_content_config::run_protected_content_config_command(cmd) + .await?; + } + Commands::Identity(cmd) => { return identity_cmd::run_identity(cmd).await; } diff --git a/elastos/crates/elastos-server/src/protected_content_config.rs b/elastos/crates/elastos-server/src/protected_content_config.rs new file mode 100644 index 000000000..9cda57611 --- /dev/null +++ b/elastos/crates/elastos-server/src/protected_content_config.rs @@ -0,0 +1,1568 @@ +//! Offline operator provisioning for the protected-content installed +//! prerequisites: the policy authority key, per-host custody node state, +//! the signed owner-only 2-of-3 custody composition, and the private +//! Chain provider configuration. + +use std::io::Write; +use std::path::{Path, PathBuf}; + +use anyhow::Context; +use base64::Engine as _; +use clap::Subcommand; +use ed25519_dalek::{Signer as _, SigningKey}; +use elastos_protected_content_contracts::{ + CanonicalContract, CustodyApprovedSuitesV1, CustodyCommitteeAuthorizationStatementV1, + CustodyEpochIssuerKeyV1, CustodyEpochStatementV1, CustodyNodeIdentityV1, + CustodyPoolFailureDomainIdV1, CustodyPoolMemberStateV1, CustodyPoolMemberV1, + CustodyPoolOperatorIdV1, CustodyPoolStatementV1, NodeCustodyPublicKeyV1, NodePublicKey, + RuntimeOperationIssuerKeyV1, ShareCoordinateV1, SignedCustodyCommitteeAuthorizationV1, + SignedCustodyEpochV1, SignedCustodyPoolV1, ThresholdV1, CUSTODY_X_WING_AES256GCM_SUITE_ID_V1, + PQ_HYBRID_WRAP_PUBLIC_KEY_BYTES, +}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use crate::collaboration_config::{create_owner_only_file, read_owner_only_file, write_receipt}; +use crate::protected_content_runtime::{ + decode_canonical_peer_did, inactive_custody_state_root, protected_content_root, + runtime_custody_composition_config_path, validate_owner_only_directory, + RuntimeCustodyCompositionConfigFile, RuntimeCustodyRouteBindingConfig, + RuntimeCustodyRouteTransportConfig, CUSTODY_COMPOSITION_SCHEMA_V1, +}; + +const POLICY_AUTHORITY_KEY_RECEIPT_SCHEMA: &str = + "elastos.protected-content.policy-authority-key-created/v1"; +const CUSTODY_NODE_PROVISION_RECEIPT_SCHEMA: &str = + "elastos.protected-content.custody-node-provisioned/v1"; +pub(crate) const CUSTODY_NODE_DESCRIPTOR_SCHEMA_V1: &str = + "elastos.protected-content.custody-node-descriptor/v1"; +const CUSTODY_COMPOSITION_RECEIPT_SCHEMA: &str = + "elastos.protected-content.custody-composition-generated/v1"; +const CUSTODY_COMPOSITION_VERIFIED_RECEIPT_SCHEMA: &str = + "elastos.protected-content.custody-composition-verified/v1"; +const RUNTIME_ISSUER_RECEIPT_SCHEMA: &str = "elastos.protected-content.runtime-issuer/v1"; +/// Mirrors the Runtime loader's byte cap for the composition file. +const MAX_CUSTODY_COMPOSITION_FILE_BYTES: usize = 64 * 1024; +const CUSTODY_OPERATOR_ID_DOMAIN: &str = "elastos.protected-content.custody-operator-id/v1"; +const CUSTODY_FAILURE_DOMAIN_ID_DOMAIN: &str = + "elastos.protected-content.custody-failure-domain-id/v1"; +const MAX_CUSTODY_NODE_LABEL_BYTES: usize = 64; +const MAX_CUSTODY_NODE_DESCRIPTOR_BYTES: usize = 8 * 1024; +/// The composition validators hard-code a 2-of-3 committee; the ceremony keeps +/// the threshold in one place so a future k-of-n only changes this pair. +const CUSTODY_THRESHOLD_REQUIRED: u8 = 2; +const CUSTODY_THRESHOLD_TOTAL: u8 = 3; +const CHAIN_CONFIG_RECEIPT_SCHEMA: &str = + "elastos.protected-content.chain-provider-config-generated/v1"; +/// Facts proven in the deployment review for Base (chain id 8453); see +/// docs/CHAIN_PROVIDER.md ("has_access_by_content_id") and docs/PROTECTED_CONTENT.md. +const PROVEN_BASE_AUTHORITY_GATEWAY: &str = "0x09dbe796f40eceffeaccf243c3d758c4c1d8d87d"; +const PROVEN_HAS_ACCESS_SELECTOR: &str = "0x54d42821"; +/// Evidence corroboration needs 2..=5 independent sources; documented as the +/// product contract in docs/PROTECTED_CONTENT.md and enforced again by the +/// chain-provider capsule at Init. +const MIN_EVIDENCE_RPC_SOURCES: usize = 2; +const MAX_EVIDENCE_RPC_SOURCES: usize = 5; + +#[derive(Debug, Subcommand)] +pub enum ProtectedContentConfigCommand { + /// Create a new dedicated 32-byte protected-content policy authority key. + CreatePolicyAuthorityKey { + #[arg(long)] + key: PathBuf, + }, + /// Provision this host's inactive custody provider state root and export + /// its custody node descriptor for the composition ceremony. + ProvisionCustodyNode { + #[arg(long)] + data_dir: PathBuf, + /// Runtime operation issuer this custody node will trust (0x + 64 lowercase hex). + #[arg(long)] + trusted_runtime_issuer: String, + /// Operator label for this node (distinct across the three nodes). + #[arg(long)] + operator: String, + /// Failure-domain label for this node (distinct across the three nodes). + #[arg(long)] + failure_domain: String, + /// Carrier peer DID route to this node; omit for the single local route. + #[arg(long)] + transport_peer_did: Option, + #[arg(long)] + output: PathBuf, + }, + /// Assemble and sign the owner-only 2-of-3 custody composition from three + /// custody node descriptors, installing it into the Runtime data dir. + GenerateCustodyComposition { + #[arg(long)] + authority_key: PathBuf, + /// Custody node descriptor path; pass exactly three. + #[arg(long = "node")] + nodes: Vec, + #[arg(long)] + data_dir: PathBuf, + /// Days the pool members stay active from now. + #[arg(long, default_value_t = 365)] + valid_days: u16, + }, + /// Purely verify the installed custody composition through the Runtime loader. + VerifyCustodyComposition { + #[arg(long)] + data_dir: PathBuf, + }, + /// Print the Runtime operation issuer custody nodes must trust for this data dir. + ShowRuntimeIssuer { + #[arg(long)] + data_dir: PathBuf, + }, + /// Install the private multi-source Chain provider configuration for the + /// protected-content rights, mint, and market methods. + GenerateChainConfig { + #[arg(long)] + data_dir: PathBuf, + #[arg(long, default_value = "base-mainnet")] + id: String, + #[arg(long, default_value = "Base")] + display_name: String, + #[arg(long, default_value_t = 8453)] + chain_id: u64, + #[arg(long, default_value = "ETH")] + native_symbol: String, + #[arg(long, default_value_t = true)] + mainnet: bool, + /// Primary JSON-RPC endpoint for transactions and reads. + #[arg(long)] + rpc_url: String, + /// Independent evidence JSON-RPC endpoint (repeat 2..=5 times, distinct origins). + #[arg(long = "evidence-rpc-url")] + evidence_rpc_urls: Vec, + /// Rights gateway contract proven for this deployment. + #[arg(long, default_value = PROVEN_BASE_AUTHORITY_GATEWAY)] + rights_contract: String, + /// has_access_by_content_id selector proven for this deployment. + #[arg(long, default_value = PROVEN_HAS_ACCESS_SELECTOR)] + rights_selector: String, + #[arg(long, default_value = PROVEN_BASE_AUTHORITY_GATEWAY)] + authority_gateway_contract: String, + #[arg(long)] + mint_ledger: String, + #[arg(long)] + mint_pay_token: String, + #[arg(long)] + mint_asset_created_emitter: String, + }, +} + +#[derive(Serialize)] +struct PolicyAuthorityKeyReceipt { + schema: &'static str, + created: bool, +} + +#[derive(Serialize)] +struct CustodyNodeProvisionReceipt { + schema: &'static str, + node_public_key_hex: String, + descriptor_sha256: String, + created: bool, +} + +#[derive(Serialize)] +struct CustodyCompositionReceipt { + schema: &'static str, + node_public_key_hexes: Vec, + config_sha256: String, + created: bool, +} + +#[derive(Serialize)] +struct CustodyCompositionVerifiedReceipt { + schema: &'static str, + node_public_key_hexes: Vec, + config_sha256: String, + verified: bool, +} + +#[derive(Serialize)] +struct RuntimeIssuerReceipt { + schema: &'static str, + trusted_runtime_issuer: String, +} + +#[derive(Serialize)] +struct ChainConfigReceipt { + schema: &'static str, + network_id: String, + chain_id: u64, + evidence_rpc_sources: usize, + config_sha256: String, + created: bool, +} + +/// One custody node's public identity plus its operator-declared placement, +/// produced on the custody host and consumed by the composition ceremony. +#[derive(Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct CustodyNodeDescriptorV1 { + pub(crate) schema: String, + pub(crate) node_public_key_hex: String, + pub(crate) node_custody_public_key_hex: String, + pub(crate) operator: String, + pub(crate) failure_domain: String, + pub(crate) transport: CustodyNodeDescriptorTransportV1, +} + +#[derive(Debug, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] +pub(crate) enum CustodyNodeDescriptorTransportV1 { + Local, + CarrierPeerDid { peer_did: String }, +} + +pub async fn run_protected_content_config_command( + command: ProtectedContentConfigCommand, +) -> anyhow::Result<()> { + let stdout = std::io::stdout(); + let mut output = stdout.lock(); + run_protected_content_config_command_with_writer(command, &mut output).await +} + +async fn run_protected_content_config_command_with_writer( + command: ProtectedContentConfigCommand, + output: &mut dyn Write, +) -> anyhow::Result<()> { + match command { + ProtectedContentConfigCommand::CreatePolicyAuthorityKey { key } => { + let mut bytes = [0u8; 32]; + getrandom::getrandom(&mut bytes) + .context("OS randomness unavailable for protected-content policy authority key")?; + create_owner_only_file(&key, &bytes, "protected-content policy authority key")?; + write_receipt( + output, + &PolicyAuthorityKeyReceipt { + schema: POLICY_AUTHORITY_KEY_RECEIPT_SCHEMA, + created: true, + }, + ) + } + ProtectedContentConfigCommand::ProvisionCustodyNode { + data_dir, + trusted_runtime_issuer, + operator, + failure_domain, + transport_peer_did, + output: output_path, + } => { + let issuer = parse_runtime_issuer_hex(&trusted_runtime_issuer)?; + let operator = validated_custody_node_label(&operator, "operator")?; + let failure_domain = validated_custody_node_label(&failure_domain, "failure domain")?; + let transport = match transport_peer_did { + None => CustodyNodeDescriptorTransportV1::Local, + Some(peer_did) => CustodyNodeDescriptorTransportV1::CarrierPeerDid { + peer_did: decode_canonical_peer_did(&peer_did)?, + }, + }; + let inactive_root = provisioned_inactive_custody_root(&data_dir)?; + let provisioned = custody_provider::provision_state_root(&inactive_root, issuer) + .map_err(|error| { + anyhow::anyhow!("custody provider state provisioning failed: {error:?}") + })?; + let descriptor = CustodyNodeDescriptorV1 { + schema: CUSTODY_NODE_DESCRIPTOR_SCHEMA_V1.to_string(), + node_public_key_hex: format!( + "0x{}", + hex::encode(provisioned.node_public_key.as_bytes()) + ), + node_custody_public_key_hex: format!( + "0x{}", + hex::encode(provisioned.node_custody_public_key.as_bytes()) + ), + operator, + failure_domain, + transport, + }; + let descriptor_bytes = serde_json::to_vec(&serde_json::to_value(&descriptor)?)?; + create_owner_only_file(&output_path, &descriptor_bytes, "custody node descriptor")?; + write_receipt( + output, + &CustodyNodeProvisionReceipt { + schema: CUSTODY_NODE_PROVISION_RECEIPT_SCHEMA, + node_public_key_hex: descriptor.node_public_key_hex, + descriptor_sha256: format!( + "sha256:{}", + hex::encode(Sha256::digest(&descriptor_bytes)) + ), + created: true, + }, + ) + } + ProtectedContentConfigCommand::GenerateCustodyComposition { + authority_key, + nodes, + data_dir, + valid_days, + } => { + let authority = read_policy_authority_key(&authority_key)?; + let descriptors = read_distinct_custody_node_descriptors(&nodes)?; + let config = signed_custody_composition_config(&authority, descriptors, valid_days)?; + let config_bytes = serde_json::to_vec(&serde_json::to_value(&config)?)?; + + validate_existing_data_dir(&data_dir)?; + ensure_owner_only_dir(&protected_content_root(&data_dir), "protected-content root")?; + let config_path = runtime_custody_composition_config_path(&data_dir); + create_owner_only_file(&config_path, &config_bytes, "custody composition")?; + + // The Runtime loader is the single validation authority; prove the + // installed file end to end before reporting success. + if let Err(error) = crate::protected_content_runtime::load_runtime_custody_composition( + &data_dir, + std::sync::Arc::new(elastos_runtime::provider::ProviderRegistry::new()), + ) { + let _ = std::fs::remove_file(&config_path); + return Err(error.context("generated custody composition failed loader validation")); + } + + write_receipt( + output, + &CustodyCompositionReceipt { + schema: CUSTODY_COMPOSITION_RECEIPT_SCHEMA, + node_public_key_hexes: config + .routes + .iter() + .map(|route| { + Ok(format!( + "0x{}", + hex::encode(decode_standard_base64(&route.node_public_key_base64)?) + )) + }) + .collect::>>()?, + config_sha256: format!("sha256:{}", hex::encode(Sha256::digest(&config_bytes))), + created: true, + }, + ) + } + ProtectedContentConfigCommand::VerifyCustodyComposition { data_dir } => { + crate::protected_content_runtime::load_runtime_custody_composition( + &data_dir, + std::sync::Arc::new(elastos_runtime::provider::ProviderRegistry::new()), + )? + .context("custody composition is missing; generate and install it first")?; + let config_bytes = read_owner_only_file( + &runtime_custody_composition_config_path(&data_dir), + MAX_CUSTODY_COMPOSITION_FILE_BYTES, + "custody composition", + )?; + let config: RuntimeCustodyCompositionConfigFile = + serde_json::from_slice(&config_bytes)?; + write_receipt( + output, + &CustodyCompositionVerifiedReceipt { + schema: CUSTODY_COMPOSITION_VERIFIED_RECEIPT_SCHEMA, + node_public_key_hexes: config + .routes + .iter() + .map(|route| { + Ok(format!( + "0x{}", + hex::encode(decode_standard_base64(&route.node_public_key_base64)?) + )) + }) + .collect::>>()?, + config_sha256: format!("sha256:{}", hex::encode(Sha256::digest(&config_bytes))), + verified: true, + }, + ) + } + ProtectedContentConfigCommand::ShowRuntimeIssuer { data_dir } => { + let device_key_path = elastos_identity::device_key_path(&data_dir); + if std::fs::symlink_metadata(&device_key_path).is_err() { + anyhow::bail!( + "Runtime device key is missing at {}; start the Runtime on this data dir once, then retry", + device_key_path.display() + ); + } + let device_key = elastos_identity::load_or_create_device_key(&data_dir)?; + let issuer = crate::protected_content_runtime::derive_protected_content_runtime_issuer( + &device_key, + )?; + write_receipt( + output, + &RuntimeIssuerReceipt { + schema: RUNTIME_ISSUER_RECEIPT_SCHEMA, + trusted_runtime_issuer: format!("0x{}", hex::encode(issuer.as_bytes())), + }, + ) + } + ProtectedContentConfigCommand::GenerateChainConfig { + data_dir, + id, + display_name, + chain_id, + native_symbol, + mainnet, + rpc_url, + evidence_rpc_urls, + rights_contract, + rights_selector, + authority_gateway_contract, + mint_ledger, + mint_pay_token, + mint_asset_created_emitter, + } => { + validate_private_rpc_url(&rpc_url, "rpc url")?; + validate_evidence_rpc_urls(&evidence_rpc_urls)?; + let rights_contract = validated_contract_address(&rights_contract, "rights contract")?; + let authority_gateway_contract = validated_contract_address( + &authority_gateway_contract, + "authority gateway contract", + )?; + let mint_ledger = validated_contract_address(&mint_ledger, "mint ledger")?; + let mint_pay_token = validated_contract_address(&mint_pay_token, "mint pay token")?; + let mint_asset_created_emitter = validated_contract_address( + &mint_asset_created_emitter, + "mint asset-created emitter", + )?; + validated_selector(&rights_selector)?; + + let network = serde_json::json!({ + "id": id, + "display_name": display_name, + "kind": "evm_json_rpc", + "chain_id": chain_id, + "native_symbol": native_symbol, + "provider": "operator", + "mainnet": mainnet, + "explorer_url": null, + "rpc_url": rpc_url, + "rights_methods": [{ + "id": "has_access_by_content_id", + "contract": rights_contract, + "abi": "has_access_by_content_id_address_bytes16", + "selector": rights_selector, + "protected_content_policies": [{ + "action": "view", + "evidence_rpc_urls": evidence_rpc_urls, + }], + }], + "protected_content_creator_mint": { + "ledger": mint_ledger, + "pay_token": mint_pay_token, + "asset_created_emitter": mint_asset_created_emitter, + "abi": "elacity_mint_v1", + }, + "protected_content_market": { + "authority_gateway_contract": authority_gateway_contract, + "evidence_rpc_urls": evidence_rpc_urls, + }, + }); + let config_bytes = serde_json::to_vec(&serde_json::json!({ + "schema": crate::protected_content_runtime::PROTECTED_CONTENT_CHAIN_PROVIDER_CONFIG_SCHEMA_V1, + "protected_content_network": network, + }))?; + + validate_existing_data_dir(&data_dir)?; + ensure_owner_only_dir(&protected_content_root(&data_dir), "protected-content root")?; + let config_path = protected_content_root(&data_dir).join("chain-provider.json"); + create_owner_only_file( + &config_path, + &config_bytes, + "protected-content chain config", + )?; + + // Prove the installed file passes the Runtime loader before + // reporting success; the chain-provider capsule re-validates the + // network body at Init and stays the live admission authority. + if let Err(error) = + crate::protected_content_runtime::load_runtime_protected_content_chain_provider_config( + &data_dir, + ) + { + let _ = std::fs::remove_file(&config_path); + return Err(error.context("generated chain config failed loader validation")); + } + + write_receipt( + output, + &ChainConfigReceipt { + schema: CHAIN_CONFIG_RECEIPT_SCHEMA, + network_id: id, + chain_id, + evidence_rpc_sources: evidence_rpc_urls.len(), + config_sha256: format!("sha256:{}", hex::encode(Sha256::digest(&config_bytes))), + created: true, + }, + ) + } + } +} + +/// Doc-backed shape rule for private chain endpoints (see the chain-provider +/// capsule's admission checks): https, or http against loopback only, and no +/// credentials in the URL. +fn validate_private_rpc_url(value: &str, label: &str) -> anyhow::Result { + let parsed = url::Url::parse(value).with_context(|| format!("{label} is not a valid URL"))?; + let loopback_http = parsed.scheme() == "http" + && matches!(parsed.host_str(), Some("127.0.0.1") | Some("localhost")); + if parsed.scheme() != "https" && !loopback_http { + anyhow::bail!("{label} must use https (or http against 127.0.0.1/localhost)"); + } + if !parsed.username().is_empty() || parsed.password().is_some() { + anyhow::bail!("{label} must not embed credentials"); + } + Ok(parsed) +} + +fn validate_evidence_rpc_urls(urls: &[String]) -> anyhow::Result<()> { + if !(MIN_EVIDENCE_RPC_SOURCES..=MAX_EVIDENCE_RPC_SOURCES).contains(&urls.len()) { + anyhow::bail!( + "evidence corroboration requires {MIN_EVIDENCE_RPC_SOURCES}..={MAX_EVIDENCE_RPC_SOURCES} independent evidence RPC sources" + ); + } + let mut seen_urls = std::collections::BTreeSet::new(); + let mut seen_origins = std::collections::BTreeSet::new(); + for value in urls { + let parsed = validate_private_rpc_url(value, "evidence rpc url")?; + if !seen_urls.insert(parsed.to_string()) { + anyhow::bail!("evidence RPC sources must be distinct URLs"); + } + if !seen_origins.insert(parsed.origin().ascii_serialization()) { + anyhow::bail!("evidence RPC sources must come from distinct origins"); + } + } + Ok(()) +} + +fn validated_contract_address(value: &str, label: &str) -> anyhow::Result { + let hex_part = value + .strip_prefix("0x") + .with_context(|| format!("{label} must start with 0x"))?; + if hex_part.len() != 40 + || !hex_part + .bytes() + .all(|byte| matches!(byte, b'0'..=b'9' | b'a'..=b'f')) + { + anyhow::bail!("{label} must be 0x plus 40 lowercase hex characters"); + } + Ok(value.to_string()) +} + +fn validated_selector(value: &str) -> anyhow::Result<()> { + let hex_part = value + .strip_prefix("0x") + .context("selector must start with 0x")?; + if hex_part.len() != 8 + || !hex_part + .bytes() + .all(|byte| matches!(byte, b'0'..=b'9' | b'a'..=b'f')) + { + anyhow::bail!("selector must be 0x plus 8 lowercase hex characters"); + } + Ok(()) +} + +struct ValidatedCustodyNodeDescriptor { + node_public_key: NodePublicKey, + custody_public_key: NodeCustodyPublicKeyV1, + operator: String, + failure_domain: String, + transport: RuntimeCustodyRouteTransportConfig, +} + +fn read_policy_authority_key(path: &Path) -> anyhow::Result { + let bytes = read_owner_only_file(path, 32, "protected-content policy authority key")?; + let key: [u8; 32] = bytes.try_into().map_err(|_| { + anyhow::anyhow!("protected-content policy authority key must contain exactly 32 bytes") + })?; + Ok(SigningKey::from_bytes(&key)) +} + +fn read_distinct_custody_node_descriptors( + paths: &[PathBuf], +) -> anyhow::Result> { + if paths.len() != usize::from(CUSTODY_THRESHOLD_TOTAL) { + anyhow::bail!( + "custody composition requires exactly {CUSTODY_THRESHOLD_TOTAL} --node descriptors" + ); + } + let mut descriptors = paths + .iter() + .map(|path| read_custody_node_descriptor(path)) + .collect::>>()?; + require_distinct( + descriptors + .iter() + .map(|d| d.node_public_key.as_bytes().to_vec()), + "node public keys", + )?; + require_distinct( + descriptors + .iter() + .map(|d| d.custody_public_key.as_bytes().to_vec()), + "node custody public keys", + )?; + require_distinct( + descriptors.iter().map(|d| d.operator.clone().into_bytes()), + "operator labels", + )?; + require_distinct( + descriptors + .iter() + .map(|d| d.failure_domain.clone().into_bytes()), + "failure-domain labels", + )?; + let local_routes = descriptors + .iter() + .filter(|d| matches!(d.transport, RuntimeCustodyRouteTransportConfig::Local)) + .count(); + if local_routes > 1 { + anyhow::bail!("custody composition allows at most one local transport route"); + } + require_distinct( + descriptors.iter().filter_map(|d| match &d.transport { + RuntimeCustodyRouteTransportConfig::CarrierPeerDid { peer_did } => { + Some(peer_did.clone().into_bytes()) + } + RuntimeCustodyRouteTransportConfig::Local => None, + }), + "carrier peer DIDs", + )?; + // The epoch statement orders nodes by node public key; keep every derived + // artifact (members, routes) in that one canonical order. + descriptors.sort_by(|a, b| { + a.node_public_key + .as_bytes() + .cmp(b.node_public_key.as_bytes()) + }); + Ok(descriptors) +} + +fn read_custody_node_descriptor(path: &Path) -> anyhow::Result { + let bytes = read_owner_only_file( + path, + MAX_CUSTODY_NODE_DESCRIPTOR_BYTES, + "custody node descriptor", + )?; + let descriptor: CustodyNodeDescriptorV1 = serde_json::from_slice(&bytes) + .with_context(|| format!("invalid custody node descriptor at {}", path.display()))?; + if serde_json::to_vec(&serde_json::to_value(&descriptor)?)? != bytes { + anyhow::bail!( + "custody node descriptor at {} is not canonical JSON", + path.display() + ); + } + if descriptor.schema != CUSTODY_NODE_DESCRIPTOR_SCHEMA_V1 { + anyhow::bail!( + "custody node descriptor at {} carries an unsupported schema", + path.display() + ); + } + let node_public_key = NodePublicKey::new(parse_prefixed_hex::<32>( + &descriptor.node_public_key_hex, + "node public key", + )?) + .map_err(|error| anyhow::anyhow!("custody node public key is invalid: {error:?}"))?; + let custody_public_key = NodeCustodyPublicKeyV1::new(parse_prefixed_hex::< + PQ_HYBRID_WRAP_PUBLIC_KEY_BYTES, + >( + &descriptor.node_custody_public_key_hex, + "node custody public key", + )?) + .map_err(|error| anyhow::anyhow!("custody node custody public key is invalid: {error:?}"))?; + let operator = validated_custody_node_label(&descriptor.operator, "operator")?; + let failure_domain = + validated_custody_node_label(&descriptor.failure_domain, "failure domain")?; + let transport = match descriptor.transport { + CustodyNodeDescriptorTransportV1::Local => RuntimeCustodyRouteTransportConfig::Local, + CustodyNodeDescriptorTransportV1::CarrierPeerDid { peer_did } => { + RuntimeCustodyRouteTransportConfig::CarrierPeerDid { + peer_did: decode_canonical_peer_did(&peer_did)?, + } + } + }; + Ok(ValidatedCustodyNodeDescriptor { + node_public_key, + custody_public_key, + operator, + failure_domain, + transport, + }) +} + +fn signed_custody_composition_config( + authority: &SigningKey, + descriptors: Vec, + valid_days: u16, +) -> anyhow::Result { + let issuer = CustodyEpochIssuerKeyV1::new(authority.verifying_key().to_bytes()) + .map_err(|error| anyhow::anyhow!("policy authority key is invalid: {error:?}"))?; + let suites = CustodyApprovedSuitesV1::new( + CUSTODY_X_WING_AES256GCM_SUITE_ID_V1, + CUSTODY_X_WING_AES256GCM_SUITE_ID_V1, + CUSTODY_X_WING_AES256GCM_SUITE_ID_V1, + ) + .map_err(|error| anyhow::anyhow!("approved custody suites are invalid: {error:?}"))?; + + let node_identities = descriptors + .iter() + .enumerate() + .map(|(index, descriptor)| { + let coordinate = ShareCoordinateV1::new(u8::try_from(index + 1)?) + .map_err(|error| anyhow::anyhow!("share coordinate is invalid: {error:?}"))?; + CustodyNodeIdentityV1::new( + descriptor.node_public_key, + descriptor.custody_public_key, + coordinate, + ) + .map_err(|error| anyhow::anyhow!("custody node identity is invalid: {error:?}")) + }) + .collect::>>()?; + let epoch_statement = CustodyEpochStatementV1::new( + issuer, + suites.clone(), + ThresholdV1::new(CUSTODY_THRESHOLD_REQUIRED, CUSTODY_THRESHOLD_TOTAL) + .map_err(|error| anyhow::anyhow!("custody threshold is invalid: {error:?}"))?, + node_identities, + ) + .map_err(|error| anyhow::anyhow!("custody epoch statement is invalid: {error:?}"))?; + let signed_epoch = SignedCustodyEpochV1::new( + epoch_statement.clone(), + authority + .sign(&epoch_statement.canonical_bytes().map_err(|error| { + anyhow::anyhow!("custody epoch statement cannot be encoded: {error:?}") + })?) + .to_bytes() + .to_vec(), + ) + .map_err(|error| anyhow::anyhow!("signed custody epoch is invalid: {error:?}"))?; + + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .context("system clock is before the Unix epoch")? + .as_secs(); + let active_window = ( + now.saturating_sub(60), + now.checked_add(u64::from(valid_days) * 86_400) + .context("custody validity window overflows")?, + ); + let members = descriptors + .iter() + .map(|descriptor| { + CustodyPoolMemberV1::new( + descriptor.node_public_key, + descriptor.custody_public_key, + CustodyPoolOperatorIdV1::new(custody_label_id( + CUSTODY_OPERATOR_ID_DOMAIN, + &descriptor.operator, + )), + CustodyPoolFailureDomainIdV1::new(custody_label_id( + CUSTODY_FAILURE_DOMAIN_ID_DOMAIN, + &descriptor.failure_domain, + )), + suites.clone(), + active_window, + CustodyPoolMemberStateV1::Active, + ) + .map_err(|error| anyhow::anyhow!("custody pool member is invalid: {error:?}")) + }) + .collect::>>()?; + let pool_statement = CustodyPoolStatementV1::new(issuer, members) + .map_err(|error| anyhow::anyhow!("custody pool statement is invalid: {error:?}"))?; + let signed_pool = SignedCustodyPoolV1::new( + pool_statement.clone(), + authority + .sign(&pool_statement.canonical_bytes().map_err(|error| { + anyhow::anyhow!("custody pool statement cannot be encoded: {error:?}") + })?) + .to_bytes() + .to_vec(), + ) + .map_err(|error| anyhow::anyhow!("signed custody pool is invalid: {error:?}"))?; + + let authorization_statement = CustodyCommitteeAuthorizationStatementV1::new( + issuer, + signed_pool + .pool_identity() + .map_err(|error| anyhow::anyhow!("custody pool identity is invalid: {error:?}"))?, + signed_epoch + .epoch_identity() + .map_err(|error| anyhow::anyhow!("custody epoch identity is invalid: {error:?}"))?, + ) + .map_err(|error| { + anyhow::anyhow!("custody committee authorization statement is invalid: {error:?}") + })?; + let signed_authorization = SignedCustodyCommitteeAuthorizationV1::new( + authorization_statement.clone(), + authority + .sign(&authorization_statement.canonical_bytes().map_err(|error| { + anyhow::anyhow!("committee authorization cannot be encoded: {error:?}") + })?) + .to_bytes() + .to_vec(), + ) + .map_err(|error| anyhow::anyhow!("signed committee authorization is invalid: {error:?}"))?; + + let routes = descriptors + .iter() + .map(|descriptor| { + Ok(RuntimeCustodyRouteBindingConfig { + node_public_key_base64: encode_standard_base64( + descriptor.node_public_key.as_bytes(), + ), + owner_state_root_base64: encode_standard_base64(&random_owner_state_root()?), + transport: descriptor.transport.clone(), + }) + }) + .collect::>>()?; + + Ok(RuntimeCustodyCompositionConfigFile { + schema: CUSTODY_COMPOSITION_SCHEMA_V1.to_string(), + expected_policy_authority_base64: encode_standard_base64( + &authority.verifying_key().to_bytes(), + ), + expected_committee_authorization_identity_base64: canonical_contract_base64( + &signed_authorization + .authorization_identity() + .map_err(|error| { + anyhow::anyhow!("committee authorization identity is invalid: {error:?}") + })?, + )?, + signed_pool_base64: canonical_contract_base64(&signed_pool)?, + signed_epoch_base64: canonical_contract_base64(&signed_epoch)?, + signed_committee_authorization_base64: canonical_contract_base64(&signed_authorization)?, + routes, + }) +} + +fn require_distinct(values: impl Iterator>, label: &str) -> anyhow::Result<()> { + let mut seen = std::collections::BTreeSet::new(); + for value in values { + if !seen.insert(value) { + anyhow::bail!("custody composition requires distinct {label} across nodes"); + } + } + Ok(()) +} + +fn custody_label_id(domain: &str, label: &str) -> [u8; 32] { + let mut hasher = Sha256::new(); + hasher.update(domain.as_bytes()); + hasher.update([0u8]); + hasher.update(label.as_bytes()); + hasher.finalize().into() +} + +fn random_owner_state_root() -> anyhow::Result<[u8; 32]> { + let mut bytes = [0u8; 32]; + getrandom::getrandom(&mut bytes).context("OS randomness unavailable for owner state root")?; + if bytes == [0u8; 32] { + anyhow::bail!("owner state root randomness returned all zeroes"); + } + Ok(bytes) +} + +fn parse_prefixed_hex(value: &str, label: &str) -> anyhow::Result<[u8; LEN]> { + let hex_part = value + .strip_prefix("0x") + .with_context(|| format!("{label} must start with 0x"))?; + if hex_part.len() != LEN * 2 + || !hex_part + .bytes() + .all(|byte| matches!(byte, b'0'..=b'9' | b'a'..=b'f')) + { + anyhow::bail!( + "{label} must be 0x plus {} lowercase hex characters", + LEN * 2 + ); + } + hex::decode(hex_part)? + .try_into() + .map_err(|_| anyhow::anyhow!("{label} must encode {LEN} bytes")) +} + +fn encode_standard_base64(bytes: &[u8]) -> String { + base64::engine::general_purpose::STANDARD.encode(bytes) +} + +fn decode_standard_base64(value: &str) -> anyhow::Result> { + base64::engine::general_purpose::STANDARD + .decode(value) + .context("invalid base64 value") +} + +fn canonical_contract_base64(value: &T) -> anyhow::Result { + Ok(encode_standard_base64(&value.canonical_bytes().map_err( + |error| anyhow::anyhow!("canonical contract encoding failed: {error:?}"), + )?)) +} + +/// Create (if missing) and validate the owner-only directory chain that must +/// exist before the inactive custody provider state root can be provisioned +/// or registered at boot. +fn provisioned_inactive_custody_root(data_dir: &Path) -> anyhow::Result { + validate_existing_data_dir(data_dir)?; + let protected_root = protected_content_root(data_dir); + ensure_owner_only_dir(&protected_root, "protected-content root")?; + let inactive_root = inactive_custody_state_root(data_dir); + let custody_dir = inactive_root + .parent() + .context("inactive custody provider root has no parent directory")?; + ensure_owner_only_dir(custody_dir, "custody provider root")?; + Ok(inactive_root) +} + +/// The Runtime enforces owner-only modes from `protected-content` down; the +/// data root itself only needs to be a real directory. +fn validate_existing_data_dir(data_dir: &Path) -> anyhow::Result<()> { + let metadata = std::fs::symlink_metadata(data_dir) + .with_context(|| format!("data dir {} is unavailable", data_dir.display()))?; + if metadata.file_type().is_symlink() || !metadata.is_dir() { + anyhow::bail!("data dir {} must be a real directory", data_dir.display()); + } + Ok(()) +} + +fn ensure_owner_only_dir(path: &Path, label: &str) -> anyhow::Result<()> { + match std::fs::symlink_metadata(path) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + std::fs::DirBuilder::new() + .mode(0o700) + .create(path) + .with_context(|| format!("failed to create {label}"))?; + } + #[cfg(not(unix))] + anyhow::bail!("{label} provisioning requires a Unix host"); + } + Err(error) => { + return Err(error).with_context(|| format!("failed to inspect {label}")); + } + Ok(_) => {} + } + validate_owner_only_directory(path, label) +} + +fn parse_runtime_issuer_hex(value: &str) -> anyhow::Result { + let hex_part = value + .strip_prefix("0x") + .context("trusted runtime issuer must start with 0x")?; + if hex_part.len() != 64 + || !hex_part + .bytes() + .all(|byte| matches!(byte, b'0'..=b'9' | b'a'..=b'f')) + { + anyhow::bail!("trusted runtime issuer must be 0x plus 64 lowercase hex characters"); + } + let bytes: [u8; 32] = hex::decode(hex_part)? + .try_into() + .map_err(|_| anyhow::anyhow!("trusted runtime issuer must encode 32 bytes"))?; + RuntimeOperationIssuerKeyV1::new(bytes) + .map_err(|_| anyhow::anyhow!("trusted runtime issuer is not a valid operation issuer key")) +} + +fn validated_custody_node_label(value: &str, label: &str) -> anyhow::Result { + if value.is_empty() || value.trim() != value || value.len() > MAX_CUSTODY_NODE_LABEL_BYTES { + anyhow::bail!( + "custody node {label} label must be 1..={MAX_CUSTODY_NODE_LABEL_BYTES} bytes without surrounding whitespace" + ); + } + Ok(value.to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn create_policy_authority_key_creates_owner_only_key_and_receipt() { + let dir = tempfile::tempdir().unwrap(); + let key_path = dir.path().join("policy-authority.key"); + let mut out = Vec::new(); + + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::CreatePolicyAuthorityKey { + key: key_path.clone(), + }, + &mut out, + ) + .await + .unwrap(); + + let metadata = std::fs::symlink_metadata(&key_path).unwrap(); + assert_eq!(metadata.len(), 32); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!(metadata.permissions().mode() & 0o777, 0o600); + } + let receipt: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!( + receipt["schema"], + "elastos.protected-content.policy-authority-key-created/v1" + ); + assert_eq!(receipt["created"], true); + } + + #[cfg(unix)] + fn owner_only_dir(path: &std::path::Path) { + use std::os::unix::fs::DirBuilderExt; + std::fs::DirBuilder::new().mode(0o700).create(path).unwrap(); + } + + /// Custody provisioning validates every ancestor's owner and mode, and the + /// std temp root is structurally unsafe on some hosts (see + /// `safe_ancestor_tempdir` in the protected-content runtime tests); anchor + /// fixtures under the build target dir instead. + #[cfg(unix)] + fn safe_ancestor_tempdir() -> tempfile::TempDir { + let root = option_env!("CARGO_TARGET_DIR") + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from(concat!(env!("CARGO_MANIFEST_DIR"), "/../../target"))); + let base = root.join("protected-content-config-test-fixtures"); + std::fs::create_dir_all(&base).unwrap(); + // Custody provisioning also rejects `..` path components, so hand it a + // canonical base. + tempfile::tempdir_in(base.canonicalize().unwrap()).unwrap() + } + + #[cfg(unix)] + fn test_runtime_issuer_hex() -> String { + use elastos_runtime::signature::SigningKey; + let key = SigningKey::from_bytes(&[0x42; 32]); + format!("0x{}", hex::encode(key.verifying_key().to_bytes())) + } + + #[cfg(unix)] + #[tokio::test] + async fn provision_custody_node_provisions_inactive_root_and_writes_descriptor() { + use std::os::unix::fs::PermissionsExt; + + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + let descriptor_path = dir.path().join("node-a.descriptor.json"); + let mut out = Vec::new(); + + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ProvisionCustodyNode { + data_dir: data_dir.clone(), + trusted_runtime_issuer: test_runtime_issuer_hex(), + operator: "operator-a".to_string(), + failure_domain: "failure-domain-a".to_string(), + transport_peer_did: None, + output: descriptor_path.clone(), + }, + &mut out, + ) + .await + .unwrap(); + + let inactive_root = data_dir.join("protected-content/custody-provider/inactive"); + let metadata = std::fs::symlink_metadata(&inactive_root).unwrap(); + assert!(metadata.is_dir()); + assert_eq!(metadata.permissions().mode() & 0o777, 0o700); + + let descriptor_bytes = std::fs::read(&descriptor_path).unwrap(); + let descriptor: serde_json::Value = serde_json::from_slice(&descriptor_bytes).unwrap(); + assert_eq!( + descriptor["schema"], + "elastos.protected-content.custody-node-descriptor/v1" + ); + let node_public_key = descriptor["node_public_key_hex"].as_str().unwrap(); + assert_eq!(node_public_key.len(), 2 + 64); + assert!(node_public_key.starts_with("0x")); + let custody_public_key = descriptor["node_custody_public_key_hex"].as_str().unwrap(); + assert_eq!(custody_public_key.len(), 2 + 1216 * 2); + assert_eq!(descriptor["operator"], "operator-a"); + assert_eq!(descriptor["failure_domain"], "failure-domain-a"); + assert_eq!(descriptor["transport"]["kind"], "local"); + + let receipt: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!( + receipt["schema"], + "elastos.protected-content.custody-node-provisioned/v1" + ); + assert_eq!(receipt["node_public_key_hex"], node_public_key); + assert_eq!(receipt["created"], true); + } + + /// The Runtime validates owner-only modes from `protected-content` down, + /// not on the data root itself; a 0755 data dir must stay provisionable. + #[cfg(unix)] + #[tokio::test] + async fn provision_custody_node_accepts_a_group_readable_data_dir() { + use std::os::unix::fs::PermissionsExt; + + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + std::fs::create_dir(&data_dir).unwrap(); + std::fs::set_permissions(&data_dir, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ProvisionCustodyNode { + data_dir: data_dir.clone(), + trusted_runtime_issuer: test_runtime_issuer_hex(), + operator: "operator-a".to_string(), + failure_domain: "failure-domain-a".to_string(), + transport_peer_did: None, + output: dir.path().join("node.descriptor.json"), + }, + &mut out, + ) + .await + .unwrap(); + + let protected_root = data_dir.join("protected-content"); + let metadata = std::fs::symlink_metadata(&protected_root).unwrap(); + assert_eq!(metadata.permissions().mode() & 0o777, 0o700); + } + + #[cfg(unix)] + #[tokio::test] + async fn provision_custody_node_reprovision_keeps_the_same_node_identity() { + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + let issuer = test_runtime_issuer_hex(); + + let mut first_out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ProvisionCustodyNode { + data_dir: data_dir.clone(), + trusted_runtime_issuer: issuer.clone(), + operator: "operator-a".to_string(), + failure_domain: "failure-domain-a".to_string(), + transport_peer_did: None, + output: dir.path().join("first.descriptor.json"), + }, + &mut first_out, + ) + .await + .unwrap(); + + let mut second_out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ProvisionCustodyNode { + data_dir: data_dir.clone(), + trusted_runtime_issuer: issuer, + operator: "operator-a".to_string(), + failure_domain: "failure-domain-a".to_string(), + transport_peer_did: None, + output: dir.path().join("second.descriptor.json"), + }, + &mut second_out, + ) + .await + .unwrap(); + + let first: serde_json::Value = serde_json::from_slice(&first_out).unwrap(); + let second: serde_json::Value = serde_json::from_slice(&second_out).unwrap(); + assert_eq!(first["node_public_key_hex"], second["node_public_key_hex"]); + } + + /// Provision three distinct custody hosts and return their descriptor paths. + #[cfg(unix)] + async fn provision_three_node_descriptors( + dir: &std::path::Path, + issuer_hex: &str, + transports: [Option; 3], + ) -> Vec { + let mut descriptors = Vec::new(); + for (index, transport_peer_did) in transports.into_iter().enumerate() { + let host_data_dir = dir.join(format!("host-{index}/data")); + std::fs::create_dir_all(host_data_dir.parent().unwrap()).unwrap(); + owner_only_dir(&host_data_dir); + let descriptor_path = dir.join(format!("node-{index}.descriptor.json")); + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ProvisionCustodyNode { + data_dir: host_data_dir, + trusted_runtime_issuer: issuer_hex.to_string(), + operator: format!("operator-{index}"), + failure_domain: format!("failure-domain-{index}"), + transport_peer_did, + output: descriptor_path.clone(), + }, + &mut out, + ) + .await + .unwrap(); + descriptors.push(descriptor_path); + } + descriptors + } + + #[cfg(unix)] + fn test_peer_did(seed: u8) -> String { + use elastos_runtime::signature::SigningKey; + crate::crypto::encode_did_key(&SigningKey::from_bytes(&[seed; 32]).verifying_key()).unwrap() + } + + #[cfg(unix)] + #[tokio::test] + async fn generate_custody_composition_output_passes_the_runtime_loader() { + use elastos_runtime::provider::ProviderRegistry; + use std::sync::Arc; + + let dir = safe_ancestor_tempdir(); + let runtime_data_dir = dir.path().join("runtime/data"); + std::fs::create_dir_all(runtime_data_dir.parent().unwrap()).unwrap(); + owner_only_dir(&runtime_data_dir); + + let authority_key_path = dir.path().join("policy-authority.key"); + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::CreatePolicyAuthorityKey { + key: authority_key_path.clone(), + }, + &mut out, + ) + .await + .unwrap(); + + let descriptors = provision_three_node_descriptors( + dir.path(), + &test_runtime_issuer_hex(), + [None, Some(test_peer_did(0x51)), Some(test_peer_did(0x52))], + ) + .await; + + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::GenerateCustodyComposition { + authority_key: authority_key_path, + nodes: descriptors, + data_dir: runtime_data_dir.clone(), + valid_days: 365, + }, + &mut out, + ) + .await + .unwrap(); + + // The loader's [_; 3] node array plus signature validation make this + // the whole done-check: parse, canonical bytes, trust anchors, routes. + crate::protected_content_runtime::load_runtime_custody_composition( + &runtime_data_dir, + Arc::new(ProviderRegistry::new()), + ) + .unwrap() + .expect("generated composition must load"); + + let receipt: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!( + receipt["schema"], + "elastos.protected-content.custody-composition-generated/v1" + ); + assert_eq!(receipt["created"], true); + assert_eq!( + receipt["node_public_key_hexes"].as_array().unwrap().len(), + 3 + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn generate_custody_composition_rejects_duplicate_operator_labels() { + let dir = safe_ancestor_tempdir(); + let runtime_data_dir = dir.path().join("runtime/data"); + std::fs::create_dir_all(runtime_data_dir.parent().unwrap()).unwrap(); + owner_only_dir(&runtime_data_dir); + + let authority_key_path = dir.path().join("policy-authority.key"); + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::CreatePolicyAuthorityKey { + key: authority_key_path.clone(), + }, + &mut out, + ) + .await + .unwrap(); + + let descriptors = provision_three_node_descriptors( + dir.path(), + &test_runtime_issuer_hex(), + [None, Some(test_peer_did(0x51)), Some(test_peer_did(0x52))], + ) + .await; + // Rewrite the third descriptor with the first one's operator label. + let mut third: serde_json::Value = + serde_json::from_slice(&std::fs::read(&descriptors[2]).unwrap()).unwrap(); + third["operator"] = serde_json::Value::String("operator-0".to_string()); + std::fs::write(&descriptors[2], serde_json::to_vec(&third).unwrap()).unwrap(); + + let mut out = Vec::new(); + let error = run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::GenerateCustodyComposition { + authority_key: authority_key_path, + nodes: descriptors, + data_dir: runtime_data_dir.clone(), + valid_days: 365, + }, + &mut out, + ) + .await + .unwrap_err(); + assert!(error.to_string().contains("distinct"), "{error}"); + assert!(!runtime_data_dir + .join("protected-content/custody-composition.json") + .exists()); + } + + #[cfg(unix)] + async fn generate_test_composition(dir: &tempfile::TempDir) -> PathBuf { + let runtime_data_dir = dir.path().join("runtime/data"); + std::fs::create_dir_all(runtime_data_dir.parent().unwrap()).unwrap(); + owner_only_dir(&runtime_data_dir); + let authority_key_path = dir.path().join("policy-authority.key"); + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::CreatePolicyAuthorityKey { + key: authority_key_path.clone(), + }, + &mut out, + ) + .await + .unwrap(); + let descriptors = provision_three_node_descriptors( + dir.path(), + &test_runtime_issuer_hex(), + [None, Some(test_peer_did(0x51)), Some(test_peer_did(0x52))], + ) + .await; + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::GenerateCustodyComposition { + authority_key: authority_key_path, + nodes: descriptors, + data_dir: runtime_data_dir.clone(), + valid_days: 365, + }, + &mut out, + ) + .await + .unwrap(); + runtime_data_dir + } + + #[cfg(unix)] + #[tokio::test] + async fn verify_custody_composition_reports_the_installed_composition() { + let dir = safe_ancestor_tempdir(); + let runtime_data_dir = generate_test_composition(&dir).await; + + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::VerifyCustodyComposition { + data_dir: runtime_data_dir, + }, + &mut out, + ) + .await + .unwrap(); + + let receipt: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!( + receipt["schema"], + "elastos.protected-content.custody-composition-verified/v1" + ); + assert_eq!(receipt["verified"], true); + assert_eq!( + receipt["node_public_key_hexes"].as_array().unwrap().len(), + 3 + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn verify_custody_composition_fails_closed_when_the_composition_is_missing() { + let dir = safe_ancestor_tempdir(); + let runtime_data_dir = dir.path().join("runtime/data"); + std::fs::create_dir_all(runtime_data_dir.parent().unwrap()).unwrap(); + owner_only_dir(&runtime_data_dir); + + let mut out = Vec::new(); + let error = run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::VerifyCustodyComposition { + data_dir: runtime_data_dir, + }, + &mut out, + ) + .await + .unwrap_err(); + assert!(error.to_string().contains("missing"), "{error}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn show_runtime_issuer_prints_the_derived_issuer_for_an_existing_device_key() { + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + let device_key = elastos_identity::load_or_create_device_key(&data_dir).unwrap(); + let expected = + crate::protected_content_runtime::derive_protected_content_runtime_issuer(&device_key) + .unwrap(); + + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ShowRuntimeIssuer { + data_dir: data_dir.clone(), + }, + &mut out, + ) + .await + .unwrap(); + + let receipt: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!( + receipt["schema"], + "elastos.protected-content.runtime-issuer/v1" + ); + assert_eq!( + receipt["trusted_runtime_issuer"], + format!("0x{}", hex::encode(expected.as_bytes())) + ); + } + + #[cfg(unix)] + fn chain_config_command( + data_dir: &std::path::Path, + evidence_rpc_urls: Vec, + ) -> ProtectedContentConfigCommand { + ProtectedContentConfigCommand::GenerateChainConfig { + data_dir: data_dir.to_path_buf(), + id: "base-mainnet".to_string(), + display_name: "Base".to_string(), + chain_id: 8453, + native_symbol: "ETH".to_string(), + mainnet: true, + rpc_url: "https://private-primary.example.invalid".to_string(), + evidence_rpc_urls, + rights_contract: "0x09dbe796f40eceffeaccf243c3d758c4c1d8d87d".to_string(), + rights_selector: "0x54d42821".to_string(), + authority_gateway_contract: "0x09dbe796f40eceffeaccf243c3d758c4c1d8d87d".to_string(), + mint_ledger: "0x0000000000000000000000000000000000000022".to_string(), + mint_pay_token: "0x0000000000000000000000000000000000000033".to_string(), + mint_asset_created_emitter: "0x0000000000000000000000000000000000000044".to_string(), + } + } + + #[cfg(unix)] + #[tokio::test] + async fn generate_chain_config_installs_a_loadable_owner_only_config() { + use std::os::unix::fs::PermissionsExt; + + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + + let mut out = Vec::new(); + run_protected_content_config_command_with_writer( + chain_config_command( + &data_dir, + vec![ + "https://evidence-a.example.invalid".to_string(), + "https://evidence-b.example.invalid".to_string(), + ], + ), + &mut out, + ) + .await + .unwrap(); + + let loaded = + crate::protected_content_runtime::load_runtime_protected_content_chain_provider_config( + &data_dir, + ) + .unwrap() + .expect("generated chain config must load"); + let network = loaded.protected_content_network(); + assert_eq!(network["chain_id"], 8453); + assert_eq!(network["rights_methods"][0]["selector"], "0x54d42821"); + assert_eq!( + network["rights_methods"][0]["protected_content_policies"][0]["evidence_rpc_urls"] + .as_array() + .unwrap() + .len(), + 2 + ); + + let path = data_dir.join("protected-content/chain-provider.json"); + let metadata = std::fs::symlink_metadata(&path).unwrap(); + assert_eq!(metadata.permissions().mode() & 0o777, 0o600); + + let receipt: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!( + receipt["schema"], + "elastos.protected-content.chain-provider-config-generated/v1" + ); + assert_eq!(receipt["created"], true); + } + + #[cfg(unix)] + #[tokio::test] + async fn generate_chain_config_rejects_a_single_evidence_source() { + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + + let mut out = Vec::new(); + let error = run_protected_content_config_command_with_writer( + chain_config_command( + &data_dir, + vec!["https://evidence-a.example.invalid".to_string()], + ), + &mut out, + ) + .await + .unwrap_err(); + assert!(error.to_string().contains("evidence"), "{error}"); + assert!(!data_dir + .join("protected-content/chain-provider.json") + .exists()); + } + + #[cfg(unix)] + #[tokio::test] + async fn generate_chain_config_rejects_duplicate_evidence_origins() { + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + + let mut out = Vec::new(); + let error = run_protected_content_config_command_with_writer( + chain_config_command( + &data_dir, + vec![ + "https://evidence-a.example.invalid/one".to_string(), + "https://evidence-a.example.invalid/two".to_string(), + ], + ), + &mut out, + ) + .await + .unwrap_err(); + assert!(error.to_string().contains("origin"), "{error}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn show_runtime_issuer_fails_closed_without_a_device_key() { + let dir = safe_ancestor_tempdir(); + let data_dir = dir.path().join("data"); + owner_only_dir(&data_dir); + + let mut out = Vec::new(); + let error = run_protected_content_config_command_with_writer( + ProtectedContentConfigCommand::ShowRuntimeIssuer { data_dir }, + &mut out, + ) + .await + .unwrap_err(); + assert!(error.to_string().contains("device key"), "{error}"); + } +} diff --git a/elastos/crates/elastos-server/src/protected_content_runtime.rs b/elastos/crates/elastos-server/src/protected_content_runtime.rs index 62dec7c04..9642bea41 100644 --- a/elastos/crates/elastos-server/src/protected_content_runtime.rs +++ b/elastos/crates/elastos-server/src/protected_content_runtime.rs @@ -171,7 +171,8 @@ pub(crate) const RUNTIME_PURCHASE_SCHEMA_V1: &str = "elastos.library.runtime-cus static RUNTIME_VIEWER_LIFECYCLE_GUARDS: OnceLock< StdMutex>>>, > = OnceLock::new(); -const CUSTODY_COMPOSITION_SCHEMA_V1: &str = "elastos.protected-content.custody-composition/v1"; +pub(crate) const CUSTODY_COMPOSITION_SCHEMA_V1: &str = + "elastos.protected-content.custody-composition/v1"; const MAX_CUSTODY_COMPOSITION_BYTES: usize = 64 * 1024; const MAX_CUSTODY_COMPOSITION_BLOB_BYTES: usize = 16 * 1024; const MAX_CUSTODY_COMPOSITION_PEER_DID_BYTES: usize = 256; @@ -312,29 +313,29 @@ struct RuntimePreparedMediaProviderOutput { #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] -enum RuntimeCustodyRouteTransportConfig { +pub(crate) enum RuntimeCustodyRouteTransportConfig { Local, CarrierPeerDid { peer_did: String }, } #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] -struct RuntimeCustodyRouteBindingConfig { - node_public_key_base64: String, - owner_state_root_base64: String, - transport: RuntimeCustodyRouteTransportConfig, +pub(crate) struct RuntimeCustodyRouteBindingConfig { + pub(crate) node_public_key_base64: String, + pub(crate) owner_state_root_base64: String, + pub(crate) transport: RuntimeCustodyRouteTransportConfig, } #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] -struct RuntimeCustodyCompositionConfigFile { - schema: String, - expected_policy_authority_base64: String, - expected_committee_authorization_identity_base64: String, - signed_pool_base64: String, - signed_epoch_base64: String, - signed_committee_authorization_base64: String, - routes: Vec, +pub(crate) struct RuntimeCustodyCompositionConfigFile { + pub(crate) schema: String, + pub(crate) expected_policy_authority_base64: String, + pub(crate) expected_committee_authorization_identity_base64: String, + pub(crate) signed_pool_base64: String, + pub(crate) signed_epoch_base64: String, + pub(crate) signed_committee_authorization_base64: String, + pub(crate) routes: Vec, } struct RuntimeValidatedCustodyRouteBinding { @@ -1466,15 +1467,25 @@ async fn register_inactive_custody_runtime_provider_target( .await } -fn inactive_custody_state_root(data_dir: &Path) -> std::path::PathBuf { +/// Derive the Runtime operation issuer custody and decrypt providers must +/// trust for the Runtime rooted at this device key. +pub fn derive_protected_content_runtime_issuer( + device_key: &[u8; 32], +) -> anyhow::Result { + let (runtime_signing_key, _) = elastos_identity::derive_did(device_key); + RuntimeOperationIssuerKeyV1::new(runtime_signing_key.verifying_key().to_bytes()) + .map_err(|_| anyhow::anyhow!("active Runtime operation issuer is invalid")) +} + +pub(crate) fn inactive_custody_state_root(data_dir: &Path) -> std::path::PathBuf { data_dir.join(INACTIVE_CUSTODY_ROOT) } -fn protected_content_root(data_dir: &Path) -> PathBuf { +pub(crate) fn protected_content_root(data_dir: &Path) -> PathBuf { data_dir.join(PROTECTED_CONTENT_ROOT) } -fn runtime_custody_composition_config_path(data_dir: &Path) -> PathBuf { +pub(crate) fn runtime_custody_composition_config_path(data_dir: &Path) -> PathBuf { data_dir.join(CUSTODY_COMPOSITION_CONFIG_FILE) } @@ -1486,7 +1497,7 @@ fn invalid_inactive_custody_config(reason: impl std::fmt::Display) -> anyhow::Er anyhow::anyhow!("inactive custody provider configuration is missing or unsafe: {reason}") } -fn validate_owner_only_directory(path: &Path, label: &str) -> anyhow::Result<()> { +pub(crate) fn validate_owner_only_directory(path: &Path, label: &str) -> anyhow::Result<()> { let metadata = fs::symlink_metadata(path).map_err(|error| { let _ = error; invalid_inactive_custody_config(format!("{label} is unavailable")) @@ -2296,7 +2307,7 @@ fn decode_node_public_key_base64(encoded: &str, field: &str) -> anyhow::Result anyhow::Result { +pub(crate) fn decode_canonical_peer_did(peer_did: &str) -> anyhow::Result { if peer_did.is_empty() || peer_did.len() > MAX_CUSTODY_COMPOSITION_PEER_DID_BYTES { anyhow::bail!( "{}", @@ -6770,6 +6781,23 @@ pub async fn reconcile_runtime_custody_viewers_after_decrypt_registration( Ok(()) } +/// Boot-path hook after successful Runtime-only decrypt registration. +/// Viewer records created before decrypt came up are the state this reconcile +/// exists for. Boot keeps serving when reconciliation fails closed. +pub async fn reconcile_runtime_custody_viewers_after_decrypt_boot( + data_dir: &Path, + registry: Arc, +) { + if let Err(error) = + reconcile_runtime_custody_viewers_after_decrypt_registration(data_dir, registry).await + { + tracing::warn!( + error = %error, + "Runtime custody viewer reconciliation after decrypt registration failed closed" + ); + } +} + fn reconstructed_buy_receipt( draft: &RuntimeMintDraft, availability: &RuntimeVerifiedContentAvailability, diff --git a/elastos/crates/elastos-server/src/protected_content_runtime/tests.rs b/elastos/crates/elastos-server/src/protected_content_runtime/tests.rs index 2808f0ebb..31d140921 100644 --- a/elastos/crates/elastos-server/src/protected_content_runtime/tests.rs +++ b/elastos/crates/elastos-server/src/protected_content_runtime/tests.rs @@ -11967,6 +11967,156 @@ async fn runtime_custody_viewer_restart_reconciliation_rejects_invalid_active_re assert!(error.to_string().contains("invalid")); } +/// The boot hook must delegate to the restart reconciliation once decrypt +/// registration succeeds; the per-lifecycle settlement semantics are covered +/// by the `runtime_custody_viewer_restart_reconciliation_*` tests. +#[cfg(unix)] +#[tokio::test] +async fn runtime_custody_viewer_decrypt_boot_settles_open_pending_record_after_registration() { + let harness = runtime_custody_prebuy_availability_harness( + 0x61, + ContentAvailabilityTestConfig::accepted(), + ) + .await; + let principal_id = "person:local:runtime-custody-viewer-decrypt-boot"; + let (proof_binding_id, _) = + install_profile_authority_keeping_device_key(&harness.data_dir, principal_id); + let mint = runtime_mint_journal(&harness.data_dir) + .load(harness.mint_id) + .unwrap(); + let purchase = persist_runtime_custody_purchase_for_mint( + &harness.data_dir, + &mint, + principal_id, + &load_profile_did_for_test(&harness.data_dir, principal_id), + crate::auth::now_ts(), + ); + persist_runtime_custody_open_pending_viewer_for_purchase( + &harness.data_dir, + &mint, + &purchase, + &proof_binding_id, + "runtime-session:alpha", + "grant:alpha", + opaque_handle(0x36), + crate::auth::now_ts() + 60, + ); + let registry = Arc::new(ProviderRegistry::new()); + let decrypt = SequencedProvider::new( + "decrypt", + vec![ + Ok(ok_provider_response( + serde_json::to_value( + DecryptProviderResponseV1::new_viewer_session_already_absent( + RuntimeReleaseAuditIdV1::new(digest(0x91)).unwrap(), + opaque_handle(0x36), + ) + .unwrap(), + ) + .unwrap(), + )), + Ok(ok_provider_response( + serde_json::to_value( + DecryptProviderResponseV1::new_cancelled_prepared_recipient( + RuntimeReleaseAuditIdV1::new(digest(0x91)).unwrap(), + opaque_handle(0x36), + ) + .unwrap(), + ) + .unwrap(), + )), + ], + ); + registry + .register_runtime_provider_target(PROTECTED_CONTENT_DECRYPT_PROVIDER_ID, decrypt) + .await + .unwrap(); + + super::reconcile_runtime_custody_viewers_after_decrypt_boot(&harness.data_dir, registry).await; + + let record = super::load_runtime_custody_viewer_record( + &harness.data_dir, + principal_id, + mint.draft().mint_id(), + ) + .unwrap() + .unwrap(); + assert_eq!( + record.lifecycle_status, + super::RuntimeCustodyViewerLifecycleStatus::AlreadyAbsent + ); +} + +/// The boot hook swallows the reconcile's hard failure into a warning so boot +/// keeps serving; the underlying `Err` path is proven by +/// `runtime_custody_viewer_restart_reconciliation_rejects_invalid_active_record`. +#[cfg(unix)] +#[tokio::test] +async fn runtime_custody_viewer_decrypt_boot_keeps_serving_when_reconciliation_fails_closed() { + let harness = runtime_custody_prebuy_availability_harness( + 0x61, + ContentAvailabilityTestConfig::accepted(), + ) + .await; + let principal_id = "person:local:runtime-custody-viewer-decrypt-boot-invalid"; + let (proof_binding_id, _) = + install_profile_authority_keeping_device_key(&harness.data_dir, principal_id); + let current_profile_did = load_profile_did_for_test(&harness.data_dir, principal_id); + let mint = runtime_mint_journal(&harness.data_dir) + .load(harness.mint_id) + .unwrap(); + let purchase = persist_runtime_custody_purchase_for_mint( + &harness.data_dir, + &mint, + principal_id, + ¤t_profile_did, + crate::auth::now_ts(), + ); + persist_runtime_custody_active_viewer_for_purchase( + &harness.data_dir, + &mint, + &purchase, + &proof_binding_id, + "runtime-session:alpha", + "grant:alpha", + opaque_handle(0x32), + crate::auth::now_ts() + 60, + ); + let mut record = super::load_runtime_custody_viewer_record( + &harness.data_dir, + principal_id, + mint.draft().mint_id(), + ) + .unwrap() + .unwrap(); + record.viewer_session_handle = hex::encode([0u8; MAX_PROVIDER_OPAQUE_HANDLE_BYTES_V1]); + super::persist_runtime_custody_viewer_record( + &harness.data_dir, + principal_id, + mint.draft().mint_id(), + &record, + ) + .unwrap(); + + super::reconcile_runtime_custody_viewers_after_decrypt_boot( + &harness.data_dir, + Arc::new(ProviderRegistry::new()), + ) + .await; + + let persisted = super::load_runtime_custody_viewer_record( + &harness.data_dir, + principal_id, + mint.draft().mint_id(), + ) + .unwrap() + .unwrap(); + assert_eq!( + persisted.lifecycle_status, + super::RuntimeCustodyViewerLifecycleStatus::Active + ); +} + #[cfg(unix)] #[tokio::test] async fn runtime_custody_viewer_lifecycle_guard_serializes_same_key_and_leaves_other_keys_independent( diff --git a/elastos/crates/elastos-server/src/server_infra.rs b/elastos/crates/elastos-server/src/server_infra.rs index c6381daaf..0dc8340f3 100644 --- a/elastos/crates/elastos-server/src/server_infra.rs +++ b/elastos/crates/elastos-server/src/server_infra.rs @@ -102,15 +102,7 @@ fn model_provider_bridge_config(data_dir: &Path) -> anyhow::Result anyhow::Result { - let (runtime_signing_key, _) = elastos_identity::derive_did(device_key); - elastos_protected_content_contracts::RuntimeOperationIssuerKeyV1::new( - runtime_signing_key.verifying_key().to_bytes(), - ) - .map_err(|_| anyhow::anyhow!("active Runtime operation issuer is invalid")) -} +use elastos_server::protected_content_runtime::derive_protected_content_runtime_issuer; fn chain_provider_bridge_config_without_protected_network( runtime_operation_issuer: &elastos_protected_content_contracts::RuntimeOperationIssuerKeyV1, @@ -943,7 +935,9 @@ async fn setup_server_infrastructure_impl( Ok(None) => tracing::info!("media-provider is installed but unconfigured"), Err(_) => tracing::warn!("Skipping media-provider due to invalid private config"), }, - Ok(None) => {} + Ok(None) => tracing::warn!( + "media-provider binary is not installed; protected-content media sessions will fail closed" + ), Err(_) => tracing::warn!("Skipping media-provider due to verification failure"), } @@ -1317,7 +1311,16 @@ async fn setup_server_infrastructure_impl( tracing::info!( "protected-content-decrypt-provider registered on Runtime-only target protected-content-decrypt; provisional decrypt-provider remains on elastos://decrypt" ); + elastos_server::protected_content_runtime::reconcile_runtime_custody_viewers_after_decrypt_boot( + &data_dir, + provider_registry.clone(), + ) + .await; } + } else { + tracing::warn!( + "protected-content-decrypt-provider binary is not installed; protected-content open and play will fail closed" + ); } if let Some(path) = crate::find_installed_provider_binary("protected-content-protect-provider") @@ -1340,6 +1343,10 @@ async fn setup_server_infrastructure_impl( path.display() ); } + } else { + tracing::warn!( + "protected-content-protect-provider binary is not installed; protected-content mint will fail closed" + ); } if let Some(path) = crate::find_installed_provider_binary("custody-provider") { @@ -1365,6 +1372,10 @@ async fn setup_server_infrastructure_impl( "custody-provider registered as inactive Runtime custody route; provisional key-provider remains the product path" ); } + } else { + tracing::warn!( + "custody-provider binary is not installed; the inactive Runtime custody route will be unavailable" + ); } // Built-in Carrier node — ALWAYS starts, not conditional on spawn_host_providers. From 5675b883765a0ac56b9a012b0287f09b1bd36506 Mon Sep 17 00:00:00 2001 From: SashaMIT Date: Wed, 2 Sep 2026 22:08:07 +0100 Subject: [PATCH 006/535] feat(home): first run seeds an empty desktop and pins Marketplace New Homes start with nothing on the desktop: apps are in the dock and the Apps face, content is reached through Library. Marketplace joins the default dock pins after Library so the way to get more is one click away. Only affects the first-run layout seed; saved layouts are untouched. Co-authored-by: Cursor --- capsules/home-gui/browser/shell-core.js | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/capsules/home-gui/browser/shell-core.js b/capsules/home-gui/browser/shell-core.js index c05ae3374..2ddd3ce41 100644 --- a/capsules/home-gui/browser/shell-core.js +++ b/capsules/home-gui/browser/shell-core.js @@ -660,7 +660,15 @@ function normalizeTaskbarLayout(taskbar, summary) { return normalized; } -const DEFAULT_TASKBAR_PINS = ["browser", "library", "wallet", "documents", "chat-room", "system"]; +const DEFAULT_TASKBAR_PINS = [ + "browser", + "library", + "marketplace", + "wallet", + "documents", + "chat-room", + "system", +]; function defaultTaskbarPins(summary) { const knownTargets = new Set(allVisibleTargets(summary).map((target) => target.target)); @@ -683,10 +691,11 @@ function normalizeDesktopLabels(labels, summary) { return normalized; } +/* First run: an empty desktop. Apps live in the dock and the Apps face; + content (games, models) is reached through Library. Anything the person + drags out to the desktop afterwards is respected by the saved layout. */ function defaultHiddenDesktopTargets(summary) { - return allVisibleTargets(summary) - .filter((target) => target.target_kind !== "object") - .map((target) => target.target); + return allVisibleTargets(summary).map((target) => target.target); } function normalizeDesktopHiddenTargets(targetIds, summary) { From 568bb04f5033850cc8b62b6b780b8cd76979eee2 Mon Sep 17 00:00:00 2001 From: SashaMIT Date: Wed, 2 Sep 2026 22:16:45 +0100 Subject: [PATCH 007/535] test(home): regression smoke expects the empty first-run desktop Co-authored-by: Cursor --- scripts/home-shell-regression-smoke.mjs | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/scripts/home-shell-regression-smoke.mjs b/scripts/home-shell-regression-smoke.mjs index c493db206..40ba0af02 100644 --- a/scripts/home-shell-regression-smoke.mjs +++ b/scripts/home-shell-regression-smoke.mjs @@ -488,13 +488,14 @@ assert( "browser", "system", "documents", + "object-target", ]), - "fresh layout did not hide all non-object visible targets and keep object targets visible", + "fresh layout did not hide every visible target (first run is an empty desktop)", shellCore.shellState.shellLayoutState.desktopHidden, ); assert( - shellCore.isTargetOnDesktop("object-target") === true, - "fresh layout hid an object target from the desktop", + shellCore.isTargetOnDesktop("object-target") === false, + "fresh layout left an object target on the desktop", shellCore.shellState.shellLayoutState.desktopHidden, ); assert( @@ -503,10 +504,8 @@ assert( shellCore.shellState.shellLayoutState.desktopHidden, ); assert( - JSON.stringify(shellCore.shellState.shellLayoutState.desktop) === JSON.stringify({ - "object-target": { x: 12, y: 12 }, - }), - "fresh layout did not seed only the visible desktop entry positions", + JSON.stringify(shellCore.shellState.shellLayoutState.desktop) === JSON.stringify({}), + "fresh layout seeded desktop entry positions for an empty desktop", shellCore.shellState.shellLayoutState.desktop, ); From d8c37cb1d13cec919b4e3540d2e1dfad66e9a616 Mon Sep 17 00:00:00 2001 From: SashaMIT Date: Wed, 2 Sep 2026 22:45:51 +0100 Subject: [PATCH 008/535] fix(apps): clear the unified-sidebar chrome overlay in Documents, Inbox and Chat Room The Home unified-sidebar chrome paints the traffic lights over the first 52px of the capsule frame. Library, Marketplace, System, People and Services already reserve that band; Documents, Inbox and Chat Room did not, so the lights sat on top of the first sidebar control (Documents' New button overlapped outright). Documents follows the Library convention (--window-chrome-safe-top, minus its own 0.75rem page inset). Inbox and Chat Room use a capsule-local 52px, matching System and the entropy gate's rule that those two keep capsule-local top padding only. Co-authored-by: Cursor --- capsules/chat-room/browser/style.css | 4 +++- capsules/documents/browser/index.html | 3 +++ capsules/inbox/browser/index.html | 4 +++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/capsules/chat-room/browser/style.css b/capsules/chat-room/browser/style.css index e2ee27f9a..fa3f41526 100644 --- a/capsules/chat-room/browser/style.css +++ b/capsules/chat-room/browser/style.css @@ -135,7 +135,9 @@ p { min-height: 0; flex-direction: column; gap: 0.45rem; - padding: 12px 10px 14px; + /* Unified-sidebar chrome paints the traffic lights over the first 52px; + capsule-local value on purpose (see home-entropy-check: Chat/Inbox). */ + padding: 52px 10px 14px; overflow: auto; border-right: 1px solid color-mix(in srgb, var(--line) 55%, transparent); background: var(--sidebar-bg); diff --git a/capsules/documents/browser/index.html b/capsules/documents/browser/index.html index 859313444..3403daa56 100644 --- a/capsules/documents/browser/index.html +++ b/capsules/documents/browser/index.html @@ -115,6 +115,9 @@ min-height: 0; overflow: hidden; padding: 0.8rem; + /* Unified-sidebar chrome paints the traffic lights over this corner; + .page-shell already insets 0.75rem, so the remainder lands content at safe-top. */ + padding-top: calc(var(--window-chrome-safe-top, 52px) - 0.75rem); background: var(--sidebar-bg); border-color: var(--line); color: var(--sidebar-text); diff --git a/capsules/inbox/browser/index.html b/capsules/inbox/browser/index.html index df17fba74..3bc8b2642 100644 --- a/capsules/inbox/browser/index.html +++ b/capsules/inbox/browser/index.html @@ -95,7 +95,9 @@ display: flex; flex-direction: column; gap: 4px; - padding: 12px 12px 16px; + /* Unified-sidebar chrome paints the traffic lights over the first 52px; + capsule-local value on purpose (see home-entropy-check: Chat/Inbox). */ + padding: 52px 12px 16px; border: 0; border-right: 1px solid var(--line); border-radius: 0; From 90d7f08f58ad168ab6874d5331c0ad7a76690ec2 Mon Sep 17 00:00:00 2001 From: SashaMIT Date: Wed, 2 Sep 2026 22:50:21 +0100 Subject: [PATCH 009/535] fix(inbox): request rows hug their content instead of stretching to the column height Co-authored-by: Cursor --- capsules/inbox/browser/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/capsules/inbox/browser/index.html b/capsules/inbox/browser/index.html index 3bc8b2642..d576fb2d4 100644 --- a/capsules/inbox/browser/index.html +++ b/capsules/inbox/browser/index.html @@ -269,6 +269,7 @@ .entry-rows { display: grid; + align-content: start; gap: 2px; overflow: auto; padding: 12px; From b26f5ee446bce1313ae8068b0b7e4310c5e0fb8b Mon Sep 17 00:00:00 2001 From: SashaMIT Date: Wed, 2 Sep 2026 23:01:04 +0100 Subject: [PATCH 010/535] =?UTF-8?q?feat(home):=20lock=20face=20=E2=80=94?= =?UTF-8?q?=20wordmark,=20full-bleed=20ground=20and=20vignette,=20large=20?= =?UTF-8?q?clock?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings the URUX lock face visual onto the Home host: ELASTOS wordmark, wallpaper as a full-bleed ground with a vignette, date and clock at their intended size, compact avatar with the name beneath. Two defects in the existing face go away with it: the generic `.home-unlock p` rule outranked `.home-unlock-date/-time` so the clock rendered at 13px grey, and the picker-flow entrance animation left a transform on the face, making it the containing block for the fixed ground and clipping the wallpaper to a centred column. Date/time rules are now scoped under .home-unlock and the entrance animates the brand/clock/account instead of the face. No behaviour change: shell-auth.js, the explicit-click passkey flow, and the person/person-name ordering are as before; no account directory. Co-authored-by: Cursor --- capsules/home/browser/index.html | 34 +++-- capsules/home/browser/style.css | 220 ++++++++++++++++++++++--------- 2 files changed, 177 insertions(+), 77 deletions(-) diff --git a/capsules/home/browser/index.html b/capsules/home/browser/index.html index 36230abf1..d5c63dc7f 100644 --- a/capsules/home/browser/index.html +++ b/capsules/home/browser/index.html @@ -62,21 +62,27 @@

Home didn't open

- + diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index abce07a95..755c5f5be 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -1850,7 +1850,7 @@ assert( ) && browserStyle.includes('.browser-status[data-visible="true"][data-copyable="true"]') && browserStyle.includes(".browser-status-copy") && - browser.includes("browser.js?v=browser-20260731b") && + browser.includes("browser.js?v=browser-20260907a") && !browser.includes("browser.js?v=browser-20260731a") && !browser.includes("browser.js?v=browser-20260730a") && !browser.includes("browser.js?v=browser-20260728a") && @@ -1896,8 +1896,8 @@ assert( ); assert( - browserJs.includes("browser-status.js?v=browser-20260730b") && - browserRemoteDisplay.includes("browser-status.js?v=browser-20260730b") && + browserJs.includes("browser-status.js?v=browser-20260907a") && + browserRemoteDisplay.includes("browser-status.js?v=browser-20260907a") && !browserJs.includes("browser-status.js?v=browser-20260711c") && !browserRemoteDisplay.includes("browser-status.js?v=browser-20260711c") && !browserJs.includes("browser-status.js?v=browser-20260626e") && @@ -1912,7 +1912,7 @@ assert( ); assert( - browserJs.includes("browser-remote-display.js?v=browser-20260731a") && + browserJs.includes("browser-remote-display.js?v=browser-20260907a") && !browserJs.includes("browser-remote-display.js?v=browser-20260730b") && !browserJs.includes("browser-remote-display.js?v=browser-20260728a") && !browserJs.includes("browser-remote-display.js?v=browser-20260727a") && diff --git a/state.md b/state.md index 5afde4fe2..74ad6920c 100644 --- a/state.md +++ b/state.md @@ -76,6 +76,8 @@ qualification matrix is in [docs/BROWSER_SUPPORT.md](docs/BROWSER_SUPPORT.md). The initial compatibility slice shares Engine protocol and request types, validates declared Engine capabilities before launch, and preserves explicit operator selection. Installed acceptance of this slice remains open. +Browser settings retain a missing selected Engine or Exit as unavailable until +the operator changes it. Automatic Engine selection remains with Runtime. Mac and Linux host adapters now query their virtualization APIs for eligibility. The Mac unit check ran locally; the Linux-specific KVM query still needs a Linux test run. @@ -95,7 +97,9 @@ Browser maturity changes. Existing installed observations retain their original source bindings and verdicts. The existing Mac VM artifact preflight rejected the image because its SHA-256 -differs from the sidecar manifest. The active test Home also lacks the required +differs from the sidecar manifest. Direct read-only inspection with `debugfs` +passed the VM file and script contract; that result leaves the stale artifact +receipt unresolved. The active test Home also lacks the required VM helper and image set. Neither installation supplies a current B01 product acceptance receipt. This identifies local repair work; it does not establish the cause of another operator's startup failure. From 013bd48809a3366b86086e3618fcc2db219fb4b7 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 17:57:35 -0500 Subject: [PATCH 070/535] feat(content): verify signed model catalog metadata Add a bounded passive first-Qwen metadata profile and an operator-pinned raw CIDv1 signed catalog snapshot through the existing inventory and authenticated catalog. Reuse ContentObjectManifest for file-list/digest consistency, retain verified publisher identity, preserve ordinary installed inventory, and project all model claims as unprepared. Include mechanical constructor closure and current source-truth documentation. Proof: expected unknown-field RED, then common metadata 3/3 and full common 95/95; server model_catalog 6/6 and final capsule_catalog 25/25 independently pass under Rust 1.91, RUSTFLAGS=-D warnings, locked/offline and the existing shared target. Workspace/Chain fmt, Home/public-copy entropy, and cached/working diff checks pass. Staging preserves all 19 Rust file bytes and the e221 Next section. Full package CID/payload verification, bounded transfer, atomic preparation, provider readiness and installed acceptance remain open. --- TASKS.md | 4 +- docs/CONTENT_CAPSULE_DISTRIBUTION.md | 49 ++- docs/MODEL_PROVIDER.md | 6 +- elastos/crates/elastos-common/src/lib.rs | 9 +- elastos/crates/elastos-common/src/manifest.rs | 260 +++++++++++ elastos/crates/elastos-crosvm/src/config.rs | 2 + .../elastos-runtime/src/capsule/manager.rs | 1 + .../elastos-runtime/src/signature/verifier.rs | 1 + .../src/api/capsule_inventory.rs | 415 +++++++++++++++++- .../api/gateway_capsule_catalog/read_model.rs | 149 +++++++ .../src/collaboration_startup.rs | 1 + elastos/crates/elastos-server/src/content.rs | 119 +++++ elastos/crates/elastos-server/src/publish.rs | 1 + elastos/crates/elastos-server/src/runtime.rs | 1 + elastos/crates/elastos-server/src/setup.rs | 18 + .../crates/elastos-server/src/supervisor.rs | 7 + .../src/bin/browser-vz-engine-supervisor.rs | 1 + elastos/crates/elastos-vz/src/config.rs | 1 + elastos/crates/elastos-vz/src/provider.rs | 1 + elastos/crates/elastos-vz/src/vm.rs | 1 + .../elastos-vz/tests/concurrent_launch.rs | 1 + elastos/crates/elastos-vz/tests/smoke.rs | 1 + state.md | 26 +- 23 files changed, 1051 insertions(+), 24 deletions(-) diff --git a/TASKS.md b/TASKS.md index 43805cb82..e2962f9d6 100644 --- a/TASKS.md +++ b/TASKS.md @@ -70,8 +70,8 @@ installed onboarding and window acceptance do not block CID source work. or release local retention. Runtime owns transfer and atomic admission behind these actions, rather than a traditional download/install or file-picker UI. All projections use the same Runtime catalog, admission and inventory facts. - First close native backend bounded-transfer, manifest and engine-availability - prerequisites; a stream label or installed catalog row is insufficient. + First close native backend bounded-transfer, full closure verification and + engine-availability prerequisites; a stream label or catalog row is insufficient. Prove progress, cancel, retry, failure/offline state, readiness, incompatibility, busy-safe retention release, restart and idempotence, with at least 10% free disk and bounded partial cleanup. Catalog selection may diff --git a/docs/CONTENT_CAPSULE_DISTRIBUTION.md b/docs/CONTENT_CAPSULE_DISTRIBUTION.md index 9a3170e0f..614df5147 100644 --- a/docs/CONTENT_CAPSULE_DISTRIBUTION.md +++ b/docs/CONTENT_CAPSULE_DISTRIBUTION.md @@ -7,15 +7,15 @@ key release remain in [PROTECTED_CONTENT.md](PROTECTED_CONTENT.md), while replication policy and availability receipts remain in [CONTENT_AVAILABILITY.md](CONTENT_AVAILABILITY.md). -Runtime currently projects installed capsules. The signed network catalog, -selection-to-preparation admission, and model-content packaging described here -remain planned work; this contract does not claim that those features are implemented. +Runtime source projects installed capsules and can verify one locally supplied, +operator-pinned signed model catalog snapshot. Selection-to-preparation, +model-content packaging and network catalog updates remain planned work. The implemented content plane already provides `elastos://content` publish, fetch, status, ensure, repair, and unpublish operations. It records signed local availability receipts, keeps raw IPFS access system-only, and serves verified gateway CID reads. These form the package-delivery foundation. The signed -network catalog and install contract remain open. +network delivery and package admission contract remain open. ## Decision @@ -163,14 +163,46 @@ in the same inventory records. The signed catalog identity remains visible with its actual availability after local eviction. Unpinning local bytes says nothing about whether another provider retains the CID. +## Implemented catalog metadata profile + +The optional `CapsuleManifest.model_content` block describes passive +`role=content`, `type=data` content. Its first-Qwen profile supports GGUF, +Q4_K_M, llama.cpp and `elastos.provider.model` version `0.1.0`. It requires +Apache-2.0 license references for the base and quantized model, a provenance +notice, bounded owner/repository identifiers and exact 40-character lowercase +Git revisions. The declared memory floor is 1 to 1,048,576 MiB. This profile +grants no execution, viewer, storage or provider capability. Other formats, +licenses and provenance schemes remain outside this closeout. + +The existing `_elastos_object.json` owns file paths, exact sizes and SHA-256 +digests. Catalog validation checks its ordered unique file list and content +digest, canonical `capsule.json` bytes, entrypoint and notice references. It +rejects traversal, aliases and missing references. Limits are 32 files, 256-byte +relative paths, a 64 KiB capsule manifest, 1 MiB per auxiliary file and 16 GiB +total declared content. Payload bytes and the complete package CID still need +verification during preparation. + +Operator configuration in `components.json.model_catalog` pins the exact raw +CIDv1/SHA-256 head and one to eight trusted publisher DIDs. Runtime reads the +fixed local `model-catalog.json` snapshot with a 128 KiB bound, verifies that +head and signature independently from entry claims, and accepts one current +signed entry. The entry names a canonical DAG-PB/SHA-256 package closure CID. +The existing authenticated catalog exposes verified publisher identity, +declared CID/size and model metadata as `unprepared`, with installed and +launchable both false. A same-name local directory does not establish admission. +Absent model configuration preserves ordinary installed inventory; invalid +model configuration marks only the model catalog unavailable. Snapshot +verification provides publisher metadata, while transfer, atomic admission, +provider readiness and deployed availability require their own proof. + ## Source prerequisites and bounded implementation plan The following are source limitations, not completed large-model support: | Existing surface | Verified limitation and required extension | | --- | --- | -| `elastos/crates/elastos-common/src/manifest.rs` | Strict `CapsuleManifest` has generic resources, requirements and an optional signature, but no typed model format/quantization/license/provenance block or signed catalog-root admission. Add bounded model-content metadata and canonical signed closure/catalog validation; retain strict unknown-field and path checks. | -| `elastos/crates/elastos-server/src/api/capsule_inventory.rs` and `gateway_capsule_catalog/read_model.rs` | Active inventory comes from installed `components.json` external entries and valid capsule directories. Extend this inventory's admission/receipt ownership and its catalog projection; a source-directory scan or a separate model store is insufficient. | +| `elastos/crates/elastos-common/src/manifest.rs` | The bounded passive metadata profile above is implemented. Preparation must verify its declared facts against the complete fetched package before admission. | +| `elastos/crates/elastos-server/src/api/capsule_inventory.rs` and `gateway_capsule_catalog/read_model.rs` | The catalog projects installed inventory plus verified, unprepared model metadata. Extend existing inventory admission/receipt ownership for preparation; source-directory presence and a signed metadata row alone are insufficient. | | `elastos/crates/elastos-server/src/content.rs` | `fetch_bytes_via_provider` calls `drain_to_vec`; `materialize_data_capsule` repeats this per file. `import_exact` and aggregate `import_object` bytes are capped at 64 MiB; import permits at most 512 files. These whole-buffer import/materialization paths are not the multi-gigabyte model path. | | `elastos/crates/elastos-runtime/src/provider/registry.rs` | `open_provider_stream` decodes the full response into `ProviderStreamSession.bytes`; 64 KiB `read_next` chunks slice that buffer. Consumer chunking alone does not bound producer memory or cancel network transfer. | | `capsules/ipfs-provider/src/main.rs` | Runtime's registered native IPFS backend handles `Cat` with CID/path only. `cat` and `cat_to_path` fetch the entire file with `read_to_end` before encoding or writing. Implement bounded backend reads and actual cancellation before using this path for Qwen. | @@ -199,9 +231,8 @@ transfer. Cancellation and cleanup leave other packages and user data intact. Proposed local commit order, after the separate onboarding/recovery and window policy slices: -1. **Package and catalog admission contract.** Extend the existing manifest, - inventory/catalog and Runtime authority types for one Qwen package. Signed - catalog entries can exist before local materialization. Add narrowly typed +1. **Package admission contract.** Build on the verified catalog metadata and + existing inventory/Runtime ownership for one Qwen package. Add narrowly typed preparation, status/cancel and retention intents at this Runtime boundary: the current catalog is read-only and content fetch does not authorize an installation. Inputs identify the exact catalog entry/CID or owned operation; diff --git a/docs/MODEL_PROVIDER.md b/docs/MODEL_PROVIDER.md index 947648653..ec11d65cd 100644 --- a/docs/MODEL_PROVIDER.md +++ b/docs/MODEL_PROVIDER.md @@ -222,7 +222,11 @@ not require a new secret store. ## Local content selection and retention The current closeout includes the complete path from trusted model discovery -to a real local reply for one verified Qwen package. It is pending implementation. +to a real local reply for one verified Qwen package. Runtime source now verifies +the bounded signed catalog metadata described in +[Content capsule distribution](CONTENT_CAPSULE_DISTRIBUTION.md#implemented-catalog-metadata-profile). +Its model rows remain unprepared. Transfer, package admission, offer binding and +the following UI flow remain pending implementation. Marketplace adds Models browse/details/Use within the existing app; System manages the same model records and local retention. Assistant and Home Agent keep their existing pickers and typed run lifecycle. These are projections of diff --git a/elastos/crates/elastos-common/src/lib.rs b/elastos/crates/elastos-common/src/lib.rs index b0973cf68..8a86cd66d 100644 --- a/elastos/crates/elastos-common/src/lib.rs +++ b/elastos/crates/elastos-common/src/lib.rs @@ -11,10 +11,11 @@ mod types; pub use error::{ElastosError, Result}; pub use manifest::{ - elastos_bus_v1_wit_sha256, AffordanceApprovalMode, AffordanceAuditMode, AffordanceRisk, - CapsuleAffordanceDescriptor, CapsuleExecution, CapsuleInterfaceDescriptor, CapsuleManifest, - CapsuleProjection, CapsuleRequirement, CapsuleRole, CapsuleRuntimeAbi, CapsuleType, - CapsuleWindowPolicy, MicroVmConfig, Permissions, ProviderAuthority, ProviderCapabilitySchema, + elastos_bus_v1_wit_sha256, validate_model_content_path, AffordanceApprovalMode, + AffordanceAuditMode, AffordanceRisk, CapsuleAffordanceDescriptor, CapsuleExecution, + CapsuleInterfaceDescriptor, CapsuleManifest, CapsuleProjection, CapsuleRequirement, + CapsuleRole, CapsuleRuntimeAbi, CapsuleType, CapsuleWindowPolicy, MicroVmConfig, + ModelContentMetadata, Permissions, ProviderAuthority, ProviderCapabilitySchema, RequirementKind, ResourceLimits, ELASTOS_BUS_V1_CONTRACT, ELASTOS_BUS_V1_WORLD, SCHEMA_V1, }; use sha2::{Digest, Sha256}; diff --git a/elastos/crates/elastos-common/src/manifest.rs b/elastos/crates/elastos-common/src/manifest.rs index da2e3ce6e..e206eced3 100644 --- a/elastos/crates/elastos-common/src/manifest.rs +++ b/elastos/crates/elastos-common/src/manifest.rs @@ -66,6 +66,11 @@ pub struct CapsuleManifest { #[serde(default, skip_serializing_if = "Option::is_none")] pub window_policy: Option, + /// Descriptive model metadata. The content object manifest owns file sizes + /// and digests; Runtime separately verifies publisher trust and admission. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub model_content: Option, + pub entrypoint: String, /// Typed requirements needed by this capsule. @@ -127,6 +132,144 @@ pub struct CapsuleManifest { pub signature: Option, } +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ModelContentMetadata { + pub format: String, + pub quantization: String, + pub engine: String, + pub consumer_interface: String, + pub consumer_interface_version: String, + pub minimum_memory_mb: u32, + pub license: ModelContentLicense, + pub provenance: ModelContentProvenance, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ModelContentLicense { + pub spdx_id: String, + pub path: String, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ModelContentProvenance { + pub base_repository: String, + pub base_revision: String, + pub base_license: ModelContentLicense, + pub quantized_repository: String, + pub quantized_revision: String, + pub path: String, +} + +impl ModelContentMetadata { + fn validate(&self, manifest: &CapsuleManifest) -> Result<(), String> { + if manifest.role != CapsuleRole::Content + || manifest.capsule_type != CapsuleType::Data + || manifest.runtime_abi.is_some() + || manifest.execution.is_some() + || manifest.bus_contract.is_some() + || manifest.wit_world_sha256.is_some() + || manifest.viewer.is_some() + || manifest.microvm.is_some() + || manifest.providers.is_some() + || manifest.authority.is_some() + || manifest.provides.is_some() + || !manifest.requires.is_empty() + || !manifest.capabilities.is_empty() + || !manifest.interfaces.is_empty() + || manifest.permissions.host_process + || manifest.permissions.guest_network + || !manifest.permissions.storage.is_empty() + || !manifest.permissions.messaging.is_empty() + { + return Err( + "model_content requires passive content/data without execution authority".into(), + ); + } + if self.format != "gguf" + || self.quantization != "Q4_K_M" + || self.engine != "llama.cpp" + || self.consumer_interface != "elastos.provider.model" + || self.consumer_interface_version != "0.1.0" + || !(1..=1_048_576).contains(&self.minimum_memory_mb) + { + return Err( + "unsupported model_content format, engine, interface or resource requirement" + .into(), + ); + } + validate_model_content_path(&manifest.entrypoint)?; + validate_descriptor_id("model_content name", &manifest.name)?; + for (value, limit) in [ + (manifest.version.as_str(), 32), + (manifest.description.as_deref().unwrap_or(""), 1024), + (manifest.author.as_deref().unwrap_or(""), 128), + ] { + if value.len() > limit || value.chars().any(char::is_control) { + return Err("model_content display facts exceed their bounds".into()); + } + } + if !manifest.entrypoint.ends_with(".gguf") { + return Err("model_content entrypoint must name its GGUF file".into()); + } + for license in [&self.license, &self.provenance.base_license] { + if license.spdx_id != "Apache-2.0" { + return Err("model_content first profile requires Apache-2.0 license facts".into()); + } + validate_model_content_path(&license.path)?; + } + validate_model_content_path(&self.provenance.path)?; + for repository in [ + &self.provenance.base_repository, + &self.provenance.quantized_repository, + ] { + validate_model_content_path(repository)?; + if repository.split('/').count() != 2 { + return Err( + "model_content provenance requires an owner/repository identifier".into(), + ); + } + } + for revision in [ + &self.provenance.base_revision, + &self.provenance.quantized_revision, + ] { + if revision.len() != 40 + || !revision + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return Err( + "model_content provenance requires an exact lowercase Git revision".into(), + ); + } + } + Ok(()) + } +} + +/// Portable canonical model-package path; file identity stays in the existing +/// content object manifest. Reject aliases before materialization on any host. +pub fn validate_model_content_path(path: &str) -> Result<(), String> { + if path.is_empty() + || path.len() > 256 + || path.split('/').any(|part| { + part.is_empty() + || part == "." + || part == ".." + || part.ends_with('.') + || !part + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-')) + }) + { + return Err("model_content path must be a bounded canonical relative path".into()); + } + Ok(()) +} + /// A single typed capsule requirement. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(deny_unknown_fields)] @@ -287,6 +430,9 @@ impl CapsuleManifest { /// Validate manifest fields after deserialization. pub fn validate(&self) -> Result<(), String> { + if let Some(model) = &self.model_content { + model.validate(self)?; + } if self.schema != SCHEMA_V1 { return Err(format!( "unsupported schema \"{}\", expected \"{}\"", @@ -995,6 +1141,120 @@ mod tests { // ── Core parse/validation tests (strict v1 schema) ────────────── + fn model_content_fixture() -> Value { + // Synthetic metadata only; this fixture is not a published model package. + serde_json::json!({ + "schema": "elastos.capsule/v1", "version": "0.1.0", + "name": "model-fixture", "role": "content", "type": "data", + "entrypoint": "weights.gguf", "projections": ["content"], + "model_content": { + "format": "gguf", "quantization": "Q4_K_M", "engine": "llama.cpp", + "consumer_interface": "elastos.provider.model", + "consumer_interface_version": "0.1.0", "minimum_memory_mb": 8192, + "license": {"spdx_id": "Apache-2.0", "path": "LICENSE"}, + "provenance": { + "base_repository": "fixture/base", "base_revision": "a".repeat(40), + "base_license": {"spdx_id": "Apache-2.0", "path": "LICENSE.base"}, + "quantized_repository": "fixture/quantized", + "quantized_revision": "b".repeat(40), "path": "PROVENANCE.md" + } + } + }) + } + + #[test] + fn model_content_metadata_accepts_bounded_gguf() { + let value = model_content_fixture(); + let manifest: CapsuleManifest = serde_json::from_value(value.clone()).unwrap(); + manifest.validate().unwrap(); + assert_eq!( + serde_json::to_value(manifest).unwrap()["model_content"], + value["model_content"] + ); + } + + #[test] + fn model_content_metadata_rejects_unknown_fields_and_authority() { + for field in ["extra", "payload_sha256", "payload_size", "provider_route"] { + let mut value = model_content_fixture(); + value["model_content"][field] = serde_json::json!("untrusted"); + assert!(serde_json::from_value::(value).is_err()); + } + for (field, invalid) in [ + ("role", serde_json::json!("app")), + ("type", serde_json::json!("wasm")), + ("viewer", serde_json::json!("assistant")), + ("capabilities", serde_json::json!(["elastos://model/*"])), + ( + "permissions", + serde_json::json!({"storage": ["localhost://*"]}), + ), + ] { + let mut value = model_content_fixture(); + value[field] = invalid; + assert!(serde_json::from_value::(value) + .unwrap() + .validate() + .is_err()); + } + } + + #[test] + fn model_content_metadata_rejects_paths_compatibility_and_unbounded_facts() { + for (pointer, invalid) in [ + ("/entrypoint", serde_json::json!("../weights.gguf")), + ("/entrypoint", serde_json::json!("models//weights.gguf")), + ("/model_content/license/path", serde_json::json!("/LICENSE")), + ( + "/model_content/provenance/path", + serde_json::json!("docs/./notice"), + ), + ( + "/model_content/provenance/base_revision", + serde_json::json!("main"), + ), + ( + "/model_content/provenance/base_repository", + serde_json::json!("a".repeat(257)), + ), + ( + "/model_content/license/spdx_id", + serde_json::json!("unknown"), + ), + ( + "/model_content/consumer_interface", + serde_json::json!("elastos.provider.other"), + ), + ( + "/model_content/consumer_interface_version", + serde_json::json!("9.0.0"), + ), + ("/model_content/engine", serde_json::json!("ambient")), + ("/model_content/format", serde_json::json!("executable")), + ("/model_content/quantization", serde_json::json!("unknown")), + ("/model_content/minimum_memory_mb", serde_json::json!(0)), + ( + "/model_content/minimum_memory_mb", + serde_json::json!(1_048_577), + ), + ("/description", serde_json::json!("x".repeat(1025))), + ] { + let mut value = model_content_fixture(); + if pointer == "/description" { + value["description"] = invalid; + } else { + *value.pointer_mut(pointer).unwrap() = invalid; + } + assert!( + serde_json::from_value::(value) + .unwrap() + .validate() + .is_err(), + "{pointer}" + ); + } + } + #[test] fn window_policy_validates_presentation_contract() { let base = serde_json::json!({ diff --git a/elastos/crates/elastos-crosvm/src/config.rs b/elastos/crates/elastos-crosvm/src/config.rs index 753f919c5..c007cbc87 100644 --- a/elastos/crates/elastos-crosvm/src/config.rs +++ b/elastos/crates/elastos-crosvm/src/config.rs @@ -441,6 +441,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, }; @@ -504,6 +505,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, }; diff --git a/elastos/crates/elastos-runtime/src/capsule/manager.rs b/elastos/crates/elastos-runtime/src/capsule/manager.rs index c5e744782..0a7b92251 100644 --- a/elastos/crates/elastos-runtime/src/capsule/manager.rs +++ b/elastos/crates/elastos-runtime/src/capsule/manager.rs @@ -414,6 +414,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-runtime/src/signature/verifier.rs b/elastos/crates/elastos-runtime/src/signature/verifier.rs index bcfb1faab..2aef3ab04 100644 --- a/elastos/crates/elastos-runtime/src/signature/verifier.rs +++ b/elastos/crates/elastos-runtime/src/signature/verifier.rs @@ -196,6 +196,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory.rs b/elastos/crates/elastos-server/src/api/capsule_inventory.rs index edd4e7767..725d5fabc 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory.rs @@ -1,9 +1,194 @@ use std::collections::{BTreeMap, BTreeSet}; +use std::io::Read; use std::path::{Path, PathBuf}; use elastos_common::CapsuleManifest; +use serde::Deserialize; +use serde_json::Value; +use sha2::{Digest, Sha256}; const DEV_CAPSULES_ROOT: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../../../capsules"); +const MODEL_CATALOG_FILE: &str = "model-catalog.json"; +const MODEL_CATALOG_DOMAIN: &str = "elastos.model.catalog.v1"; +const MAX_MODEL_CATALOG_BYTES: usize = 128 * 1024; + +pub(crate) struct VerifiedModelCatalogEntry { + pub cid: String, + pub publisher_did: String, + pub manifest: CapsuleManifest, + pub size_bytes: u64, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct SignedModelCatalog { + payload: ModelCatalogPayload, + signature: String, + signer_did: String, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct ModelCatalogPayload { + schema: String, + published_at: u64, + expires_at: u64, + entries: Vec, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct ModelCatalogEntry { + cid: String, + capsule_manifest: Value, + object_manifest: Value, +} + +fn read_model_catalog_file(data_dir: &Path, name: &str, limit: usize) -> anyhow::Result> { + let root = std::fs::symlink_metadata(data_dir)?; + if !root.is_dir() || root.file_type().is_symlink() { + anyhow::bail!("model catalog requires a real data directory"); + } + let mut options = std::fs::OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK); + } + let path = data_dir.join(name); + let link_metadata = std::fs::symlink_metadata(&path)?; + if !link_metadata.is_file() || link_metadata.file_type().is_symlink() { + anyhow::bail!("model catalog input must be a regular file"); + } + let file = options.open(path)?; + let metadata = file.metadata()?; + if !metadata.is_file() || metadata.len() > limit as u64 { + anyhow::bail!("model catalog input exceeds its file bound"); + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + let uid = unsafe { libc::geteuid() }; + if root.uid() != uid + || root.mode() & 0o022 != 0 + || metadata.uid() != uid + || metadata.mode() & 0o022 != 0 + || metadata.nlink() != 1 + { + anyhow::bail!( + "model catalog input must remain operator-owned and non-writable by others" + ); + } + } + let mut bytes = Vec::new(); + file.take(limit as u64 + 1).read_to_end(&mut bytes)?; + if bytes.len() > limit { + anyhow::bail!("model catalog input exceeds its byte bound"); + } + Ok(bytes) +} + +pub(crate) fn model_catalog_entries( + data_dir: &Path, +) -> anyhow::Result>> { + let config: crate::setup::ComponentsManifest = serde_json::from_slice( + &read_model_catalog_file(data_dir, "components.json", 4 * 1024 * 1024)?, + )?; + let Some(trust) = config.model_catalog else { + return Ok(None); + }; + let bytes = read_model_catalog_file(data_dir, MODEL_CATALOG_FILE, MAX_MODEL_CATALOG_BYTES)?; + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH)? + .as_secs(); + verify_model_catalog(&trust, &bytes, now).map(Some) +} + +fn verify_model_catalog( + trust: &crate::setup::ModelCatalogConfig, + bytes: &[u8], + now: u64, +) -> anyhow::Result> { + // The shared verifier treats an empty expected set as unrestricted trust. + if trust.publisher_dids.is_empty() + || trust.publisher_dids.len() > 8 + || trust + .publisher_dids + .iter() + .any(|did| did.is_empty() || did.len() > 128) + || trust.publisher_dids.iter().collect::>().len() != trust.publisher_dids.len() + || trust.head_cid.len() > 128 + || bytes.len() > MAX_MODEL_CATALOG_BYTES + { + anyhow::bail!("model catalog trust or byte bound is invalid"); + } + let head = cid::Cid::try_from(trust.head_cid.as_str())?; + if head.version() != cid::Version::V1 + || head.to_string() != trust.head_cid + || head.codec() != 0x55 + || head.hash().code() != 0x12 + || head.hash().digest() != &Sha256::digest(bytes)[..] + { + anyhow::bail!("model catalog head must match the pinned raw SHA-256 CIDv1"); + } + let signed: SignedModelCatalog = serde_json::from_slice(bytes)?; + for publisher in &trust.publisher_dids { + crate::crypto::decode_did_key(publisher)?; + } + if signed.signature.len() != 128 || signed.signer_did.len() > 128 { + anyhow::bail!("model catalog signature fields exceed their bounds"); + } + crate::crypto::verify_signed_json_envelope_against_dids( + bytes, + MODEL_CATALOG_DOMAIN, + &trust.publisher_dids, + )?; + let payload = signed.payload; + if payload.schema != "elastos.model.catalog/v1" + || payload.published_at > now + || payload.expires_at <= now + || payload.expires_at <= payload.published_at + || payload.entries.len() != 1 + { + anyhow::bail!("model catalog requires one current signed entry"); + } + let mut verified = Vec::with_capacity(1); + for entry in payload.entries { + if entry.cid.len() > 128 { + anyhow::bail!("model package CID exceeds its bound"); + } + let cid = cid::Cid::try_from(entry.cid.as_str())?; + if cid.version() != cid::Version::V1 + || cid.to_string() != entry.cid + || cid.codec() != 0x70 + || cid.hash().code() != 0x12 + || cid.hash().digest().len() != 32 + { + anyhow::bail!("model package requires a canonical DAG-PB SHA-256 closure CIDv1"); + } + if entry + .object_manifest + .get("publisher_did") + .is_some_and(|publisher| { + !publisher.is_null() && publisher.as_str() != Some(signed.signer_did.as_str()) + }) + { + anyhow::bail!("model closure publisher must match its configured catalog signer"); + } + let (manifest, size_bytes) = crate::content::validate_model_content_closure_metadata( + &entry.capsule_manifest, + &entry.object_manifest, + )?; + verified.push(VerifiedModelCatalogEntry { + cid: entry.cid, + publisher_did: signed.signer_did.clone(), + manifest, + size_bytes, + }); + } + Ok(verified) +} pub(crate) fn installed_capsules_root(data_dir: &Path) -> PathBuf { data_dir.join("capsules") @@ -15,7 +200,11 @@ pub(crate) fn development_capsules_root() -> PathBuf { fn components_manifest(data_dir: &Path) -> Option { let bytes = std::fs::read(data_dir.join("components.json")).ok()?; - serde_json::from_slice(&bytes).ok() + let mut value: Value = serde_json::from_slice(&bytes).ok()?; + // Model trust errors close only the model catalog. Ordinary installed + // inventory does not consume that separate optional configuration field. + value.as_object_mut()?.remove("model_catalog"); + serde_json::from_value(value).ok() } pub(crate) fn active_component_names(data_dir: &Path) -> Option> { @@ -129,9 +318,231 @@ pub(crate) fn list_development_capsule_manifests() -> Vec { } #[cfg(test)] -mod tests { +pub(crate) mod tests { use super::*; + pub(crate) fn model_catalog_fixture() -> Value { + // Synthetic signed metadata; no model bytes or production publisher. + let capsule = serde_json::json!({ + "schema": "elastos.capsule/v1", "version": "0.1.0", "name": "model-fixture", + "role": "content", "type": "data", "entrypoint": "weights.gguf", + "projections": ["content"], + "model_content": { + "format": "gguf", "quantization": "Q4_K_M", "engine": "llama.cpp", + "consumer_interface": "elastos.provider.model", "consumer_interface_version": "0.1.0", + "minimum_memory_mb": 8192, + "license": {"spdx_id": "Apache-2.0", "path": "LICENSE"}, + "provenance": { + "base_repository": "fixture/base", "base_revision": "a".repeat(40), + "base_license": {"spdx_id": "Apache-2.0", "path": "LICENSE.base"}, + "quantized_repository": "fixture/quantized", "quantized_revision": "b".repeat(40), + "path": "PROVENANCE.md" + } + } + }); + let capsule_bytes = serde_json::to_vec(&capsule).unwrap(); + let files = [ + ("LICENSE", b"fixture license".as_slice()), + ("LICENSE.base", b"fixture base license".as_slice()), + ("PROVENANCE.md", b"fixture provenance".as_slice()), + ("capsule.json", capsule_bytes.as_slice()), + ("weights.gguf", b"GGUF fixture metadata only".as_slice()), + ] + .into_iter() + .map(|(path, bytes)| { + serde_json::json!({ + "path": path, "size": bytes.len(), "sha256": format!("{:x}", Sha256::digest(bytes)) + }) + }) + .collect::>(); + let mut digest = Sha256::new(); + for file in &files { + digest.update(file["path"].as_str().unwrap().as_bytes()); + digest.update(b"\0"); + digest.update(file["sha256"].as_str().unwrap().as_bytes()); + digest.update(b"\0"); + digest.update(file["size"].to_string().as_bytes()); + digest.update(b"\0"); + } + serde_json::json!({ + "schema": "elastos.model.catalog/v1", "published_at": 1, "expires_at": 4_000_000_000_u64, + "entries": [{ + "cid": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi", + "capsule_manifest": capsule, + "object_manifest": { + "schema": "elastos.content.object.manifest/v1", "kind": "capsule", + "content_digest": format!("sha256:{:x}", digest.finalize()), "files": files + } + }] + }) + } + + fn head_cid(bytes: &[u8]) -> String { + let hash = cid::multihash::Multihash::<64>::wrap(0x12, &Sha256::digest(bytes)).unwrap(); + cid::Cid::new_v1(0x55, hash).to_string() + } + + fn sign_model_catalog(payload: &Value) -> (crate::setup::ModelCatalogConfig, Vec) { + let key = elastos_runtime::signature::SigningKey::from_bytes(&[7; 32]); + let (signature, signer_did) = crate::crypto::domain_separated_sign( + &key, + MODEL_CATALOG_DOMAIN, + &serde_json::to_vec(payload).unwrap(), + ); + let bytes = serde_json::to_vec(&serde_json::json!({ + "payload": payload, "signature": signature, "signer_did": signer_did + })) + .unwrap(); + ( + crate::setup::ModelCatalogConfig { + head_cid: head_cid(&bytes), + publisher_dids: vec![signer_did], + }, + bytes, + ) + } + + pub(crate) fn write_model_catalog_fixture(data_dir: &Path, payload: &Value) { + let (trust, bytes) = sign_model_catalog(payload); + std::fs::write( + data_dir.join("components.json"), + serde_json::to_vec(&serde_json::json!({ + "external": {}, "capsules": {}, "profiles": {}, "model_catalog": trust + })) + .unwrap(), + ) + .unwrap(); + std::fs::write(data_dir.join(MODEL_CATALOG_FILE), bytes).unwrap(); + } + + #[test] + fn model_catalog_accepts_exact_pinned_signed_metadata() { + let root = tempfile::tempdir().unwrap(); + write_model_catalog_fixture(root.path(), &model_catalog_fixture()); + let entries = model_catalog_entries(root.path()).unwrap().unwrap(); + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].manifest.name, "model-fixture"); + assert!(entries[0].size_bytes > 0); + assert!(!root.path().join("capsules/model-fixture").exists()); + } + + #[test] + fn model_catalog_rejects_empty_untrusted_publishers_and_wrong_head() { + let (trust, bytes) = sign_model_catalog(&model_catalog_fixture()); + for publishers in [ + Vec::new(), + vec!["did:key:untrusted".into()], + vec![trust.publisher_dids[0].clone(); 2], + vec!["x".repeat(129)], + ] { + let mut changed = trust.clone(); + changed.publisher_dids = publishers; + assert!(verify_model_catalog(&changed, &bytes, 2).is_err()); + } + let mut changed = trust.clone(); + changed.head_cid = head_cid(b"different"); + assert!(verify_model_catalog(&changed, &bytes, 2).is_err()); + let mut envelope: Value = serde_json::from_slice(&bytes).unwrap(); + envelope["payload"]["entries"][0]["capsule_manifest"]["name"] = "tampered".into(); + let tampered = serde_json::to_vec(&envelope).unwrap(); + changed.head_cid = head_cid(&tampered); + assert!( + verify_model_catalog(&changed, &tampered, 2).is_err(), + "even a repinned payload needs the configured signature" + ); + assert!(verify_model_catalog(&trust, &vec![0; MAX_MODEL_CATALOG_BYTES + 1], 2).is_err()); + } + + #[test] + fn model_catalog_rejects_invalid_or_incomplete_closures() { + for (pointer, value) in [ + ("/schema", serde_json::json!("unknown")), + ("/published_at", serde_json::json!(3)), + ("/expires_at", serde_json::json!(2)), + ( + "/entries/0/cid", + serde_json::json!(head_cid(b"raw is not a directory")), + ), + ( + "/entries/0/object_manifest/content_digest", + serde_json::json!("sha256:wrong"), + ), + ( + "/entries/0/object_manifest/files/0/path", + serde_json::json!("../LICENSE"), + ), + ( + "/entries/0/object_manifest/files/1/path", + serde_json::json!("license"), + ), + ( + "/entries/0/object_manifest/files/0/size", + serde_json::json!(0), + ), + ( + "/entries/0/object_manifest/files/4/size", + serde_json::json!(17_u64 * 1024 * 1024 * 1024), + ), + ( + "/entries/0/object_manifest/files/3/sha256", + serde_json::json!("0".repeat(64)), + ), + ( + "/entries/0/capsule_manifest/model_content/provenance/path", + serde_json::json!("missing.md"), + ), + ] { + let mut payload = model_catalog_fixture(); + *payload.pointer_mut(pointer).unwrap() = value; + let (trust, bytes) = sign_model_catalog(&payload); + assert!( + verify_model_catalog(&trust, &bytes, 2).is_err(), + "{pointer}" + ); + } + for pointer in [ + "", + "/entries/0", + "/entries/0/object_manifest", + "/entries/0/object_manifest/files/0", + ] { + let mut payload = model_catalog_fixture(); + payload.pointer_mut(pointer).unwrap()["extra"] = true.into(); + let (trust, bytes) = sign_model_catalog(&payload); + assert!( + verify_model_catalog(&trust, &bytes, 2).is_err(), + "extra field at {pointer}" + ); + } + let mut payload = model_catalog_fixture(); + let entry = payload["entries"][0].clone(); + payload["entries"].as_array_mut().unwrap().push(entry); + let (trust, bytes) = sign_model_catalog(&payload); + assert!(verify_model_catalog(&trust, &bytes, 2).is_err()); + } + + #[cfg(unix)] + #[test] + fn model_catalog_rejects_symlink_hardlink_and_writable_snapshot() { + use std::os::unix::fs::{symlink, PermissionsExt}; + for name in [MODEL_CATALOG_FILE, "components.json"] { + let root = tempfile::tempdir().unwrap(); + write_model_catalog_fixture(root.path(), &model_catalog_fixture()); + let path = root.path().join(name); + let original = root.path().join("original"); + std::fs::rename(&path, &original).unwrap(); + symlink(&original, &path).unwrap(); + assert!(model_catalog_entries(root.path()).is_err()); + std::fs::remove_file(&path).unwrap(); + std::fs::hard_link(&original, &path).unwrap(); + assert!(model_catalog_entries(root.path()).is_err()); + std::fs::remove_file(&path).unwrap(); + std::fs::rename(&original, &path).unwrap(); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o666)).unwrap(); + assert!(model_catalog_entries(root.path()).is_err()); + } + } + #[test] fn product_inventory_excludes_source_only_capsules() { let data_dir = tempfile::tempdir().unwrap(); diff --git a/elastos/crates/elastos-server/src/api/gateway_capsule_catalog/read_model.rs b/elastos/crates/elastos-server/src/api/gateway_capsule_catalog/read_model.rs index cad90b5af..e88fcb284 100644 --- a/elastos/crates/elastos-server/src/api/gateway_capsule_catalog/read_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_capsule_catalog/read_model.rs @@ -24,6 +24,8 @@ pub(in crate::api::gateway) fn capsule_catalog_summary( let mut capsules = crate::api::capsule_inventory::list_active_capsule_manifests(data_dir) .into_iter() + // A directory or component entry cannot authenticate model content. + .filter(|manifest| manifest.model_content.is_none()) .map(|manifest| { catalog_capsule_summary( manifest, @@ -33,6 +35,45 @@ pub(in crate::api::gateway) fn capsule_catalog_summary( ) }) .collect::>(); + let model_catalog_state = match crate::api::capsule_inventory::model_catalog_entries(data_dir) { + Ok(None) => "unconfigured", + Ok(Some(entries)) + if entries.iter().all(|entry| { + !capsules + .iter() + .any(|capsule| capsule.name == entry.manifest.name) + }) => + { + for entry in entries { + let mut summary = catalog_capsule_summary( + entry.manifest, + &BTreeMap::new(), + &BTreeMap::new(), + &BTreeMap::new(), + ); + summary.state = "unprepared".into(); + summary.installed = false; + summary.launchable = false; + summary.cid = Some(entry.cid); + summary.publisher_did = Some(entry.publisher_did); + summary.cid_state = "catalog-declared".into(); + summary.signature_state = "catalog-signature-verified".into(); + summary.trust_state = "publisher-verified-admission-pending".into(); + summary.source = "signed-model-catalog".into(); + summary.content_size_bytes = Some(entry.size_bytes); + summary.icon.clear(); + summary.projection.audit_mirror.note = Some( + "Catalog publisher verified; content admission and provider readiness remain pending.".into()); + capsules.push(summary); + } + "verified" + } + Ok(Some(_)) => "unavailable", + Err(err) => { + tracing::debug!(error = %format!("{err:#}"), "model catalog unavailable"); + "unavailable" + } + }; capsules.sort_by(|left, right| { capsule_category_order(&left.category) .cmp(&capsule_category_order(&right.category)) @@ -69,6 +110,7 @@ pub(in crate::api::gateway) fn capsule_catalog_summary( CapsuleCatalogResponse { schema: CAPSULE_CATALOG_SCHEMA.to_string(), + model_catalog_state: model_catalog_state.to_string(), counts, capsules, policy: CapsuleCatalogPolicy { @@ -221,6 +263,9 @@ fn catalog_capsule_summary( CapsuleSummary { name: name.clone(), + model_content: manifest.model_content, + publisher_did: None, + content_size_bytes: None, version: manifest.version, title: target .map(|target| target.title.clone()) @@ -612,6 +657,103 @@ fn load_capsule_components(data_dir: &std::path::Path) -> BTreeMap 0); + for field in ["route", "launch_target", "install_path", "release_path"] { + assert!(model.get(field).is_none(), "{field}"); + } + assert!(!serde_json::to_string(&summary) + .unwrap() + .contains(&root.path().display().to_string())); + } + + #[test] + fn model_catalog_absent_or_malformed_keeps_ordinary_installed_inventory() { + let root = tempfile::tempdir().unwrap(); + let dir = root.path().join("capsules/system"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("capsule.json"), + include_bytes!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../../capsules/system/capsule.json" + )), + ) + .unwrap(); + for trust in [ + None, + Some(serde_json::json!("malformed")), + Some(serde_json::json!({ + "head_cid": "invalid", "publisher_dids": [] + })), + ] { + let mut components = serde_json::json!({ + "external": {"system": {"install_path": "capsules/system", "platforms": {}}}, + "capsules": {}, "profiles": {} + }); + let expected = if let Some(trust) = trust { + components["model_catalog"] = trust; + "unavailable" + } else { + "unconfigured" + }; + std::fs::write( + root.path().join("components.json"), + serde_json::to_vec(&components).unwrap(), + ) + .unwrap(); + let summary = serde_json::to_value(capsule_catalog_summary(root.path())).unwrap(); + assert_eq!(summary["model_catalog_state"], expected); + let ordinary = summary["capsules"] + .as_array() + .unwrap() + .iter() + .find(|capsule| capsule["name"] == "system") + .expect("ordinary System remains visible"); + assert_eq!(ordinary["installed"], true); + assert!(ordinary.get("model_content").is_none()); + } + } + #[test] fn capsule_title_preserves_product_names() { assert_eq!(capsule_title("wallet-metamask"), "MetaMask"); @@ -662,6 +804,7 @@ mod tests { #[derive(Serialize)] pub(in crate::api::gateway) struct CapsuleCatalogResponse { pub(in crate::api::gateway) schema: String, + pub(in crate::api::gateway) model_catalog_state: String, pub(in crate::api::gateway) counts: CapsuleCatalogCounts, pub(in crate::api::gateway) capsules: Vec, pub(in crate::api::gateway) policy: CapsuleCatalogPolicy, @@ -695,6 +838,12 @@ pub(in crate::api::gateway) struct CapsuleCatalogPolicy { pub(in crate::api::gateway) struct CapsuleSummary { pub(in crate::api::gateway) name: String, #[serde(skip_serializing_if = "Option::is_none")] + pub(in crate::api::gateway) model_content: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(in crate::api::gateway) publisher_did: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub(in crate::api::gateway) content_size_bytes: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub(in crate::api::gateway) window_policy: Option, pub(in crate::api::gateway) version: String, pub(in crate::api::gateway) title: String, diff --git a/elastos/crates/elastos-server/src/collaboration_startup.rs b/elastos/crates/elastos-server/src/collaboration_startup.rs index e05057abb..f997689c9 100644 --- a/elastos/crates/elastos-server/src/collaboration_startup.rs +++ b/elastos/crates/elastos-server/src/collaboration_startup.rs @@ -1175,6 +1175,7 @@ mod tests { let mut supervisor = crate::supervisor::Supervisor::new( temp.path().to_path_buf(), crate::setup::ComponentsManifest { + model_catalog: None, external: std::collections::HashMap::new(), capsules: std::collections::HashMap::new(), profiles: std::collections::HashMap::new(), diff --git a/elastos/crates/elastos-server/src/content.rs b/elastos/crates/elastos-server/src/content.rs index ce59256c0..5fc93b484 100644 --- a/elastos/crates/elastos-server/src/content.rs +++ b/elastos/crates/elastos-server/src/content.rs @@ -2657,6 +2657,125 @@ pub async fn fetch_bytes_via_provider( .map_err(|err| anyhow::anyhow!("content provider stream read failed: {err}")) } +/// Validate signed catalog metadata using the existing content closure list. +/// This proves descriptive consistency only; payload verification and admission +/// require the later bounded content transfer. +pub(crate) fn validate_model_content_closure_metadata( + capsule: &Value, + object: &Value, +) -> anyhow::Result<(elastos_common::CapsuleManifest, u64)> { + let capsule_bytes = serde_json::to_vec(capsule)?; + if capsule_bytes.len() > 64 * 1024 { + anyhow::bail!("model capsule manifest exceeds its byte limit"); + } + let manifest: elastos_common::CapsuleManifest = serde_json::from_value(capsule.clone())?; + manifest.validate().map_err(anyhow::Error::msg)?; + let model = manifest + .model_content + .as_ref() + .ok_or_else(|| anyhow::anyhow!("model catalog requires model_content metadata"))?; + let fields = object + .as_object() + .ok_or_else(|| anyhow::anyhow!("invalid content closure"))?; + if fields.keys().any(|key| { + !matches!( + key.as_str(), + "schema" + | "kind" + | "content_digest" + | "files" + | "links" + | "object_did" + | "publisher_did" + ) + }) { + anyhow::bail!("unknown model content closure field"); + } + let files = object + .get("files") + .and_then(Value::as_array) + .ok_or_else(|| anyhow::anyhow!("missing model content closure files"))?; + if files.is_empty() || files.len() > 32 { + anyhow::bail!("model content closure file count exceeds its limit"); + } + for file in files { + if file.as_object().is_none_or(|fields| { + fields.len() != 3 + || fields + .keys() + .any(|key| !matches!(key.as_str(), "path" | "sha256" | "size")) + }) { + anyhow::bail!("invalid model content file fields"); + } + } + let closure = parse_content_object_manifest("model-catalog", &serde_json::to_vec(object)?)?; + if closure.kind != "capsule" || !closure.links.is_empty() || closure.object_did.is_some() { + anyhow::bail!("model catalog requires a self-contained capsule closure"); + } + let mut total = 0_u64; + let mut paths = BTreeSet::new(); + let mut previous = ""; + let mut digest = sha2::Sha256::new(); + for file in &closure.files { + elastos_common::validate_model_content_path(&file.path).map_err(anyhow::Error::msg)?; + if file.path.as_str() <= previous + || !paths.insert(file.path.to_ascii_lowercase()) + || file.path.eq_ignore_ascii_case(CONTENT_OBJECT_MANIFEST_PATH) + || file.size == 0 + || file.size > 16 * 1024 * 1024 * 1024 + || (file.path != manifest.entrypoint && file.size > 1024 * 1024) + || file.sha256.len() != 64 + || !file + .sha256 + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + anyhow::bail!("invalid, aliased or unbounded model content file"); + } + previous = &file.path; + total = total + .checked_add(file.size) + .ok_or_else(|| anyhow::anyhow!("model size overflow"))?; + digest.update(file.path.as_bytes()); + digest.update(b"\0"); + digest.update(file.sha256.as_bytes()); + digest.update(b"\0"); + digest.update(file.size.to_string().as_bytes()); + digest.update(b"\0"); + } + if total > 16 * 1024 * 1024 * 1024 + || closure.content_digest != format!("sha256:{:x}", digest.finalize()) + { + anyhow::bail!("model closure size or digest mismatch"); + } + for path in [ + &manifest.entrypoint, + &model.license.path, + &model.provenance.base_license.path, + &model.provenance.path, + ] { + if !closure.files.iter().any(|file| &file.path == path) { + anyhow::bail!("model content reference is absent from the closure"); + } + } + for path in [ + &model.license.path, + &model.provenance.base_license.path, + &model.provenance.path, + ] { + if path == &manifest.entrypoint || path == "capsule.json" { + anyhow::bail!("model notice must be a distinct closure file"); + } + } + let capsule_file = closure + .files + .iter() + .find(|file| file.path == "capsule.json") + .ok_or_else(|| anyhow::anyhow!("model closure lacks capsule.json"))?; + verify_content_object_file("model-catalog", capsule_file, &capsule_bytes)?; + Ok((manifest, total)) +} + pub async fn fetch_content_object_manifest( registry: &ProviderRegistry, cid: &str, diff --git a/elastos/crates/elastos-server/src/publish.rs b/elastos/crates/elastos-server/src/publish.rs index c9d90d90b..7e7036328 100644 --- a/elastos/crates/elastos-server/src/publish.rs +++ b/elastos/crates/elastos-server/src/publish.rs @@ -1616,6 +1616,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-server/src/runtime.rs b/elastos/crates/elastos-server/src/runtime.rs index 69347e1b7..ae08a2ed0 100644 --- a/elastos/crates/elastos-server/src/runtime.rs +++ b/elastos/crates/elastos-server/src/runtime.rs @@ -651,6 +651,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-server/src/setup.rs b/elastos/crates/elastos-server/src/setup.rs index 4405dce8a..1f0d9f054 100644 --- a/elastos/crates/elastos-server/src/setup.rs +++ b/elastos/crates/elastos-server/src/setup.rs @@ -34,6 +34,18 @@ pub struct ComponentsManifest { pub capsules: HashMap, pub profiles: HashMap, + + /// Operator-pinned signed model catalog. Absence keeps installed inventory + /// unchanged; catalog entries cannot supply their own trust configuration. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub model_catalog: Option, +} + +#[derive(Deserialize, Serialize, Clone)] +#[serde(deny_unknown_fields)] +pub struct ModelCatalogConfig { + pub head_cid: String, + pub publisher_dids: Vec, } /// A setup-materialized component. @@ -3776,6 +3788,7 @@ mod tests { fs::create_dir_all(binary_path.parent().unwrap()).unwrap(); fs::write(&binary_path, b"object-provider").unwrap(); let manifest = ComponentsManifest { + model_catalog: None, external: HashMap::new(), capsules: HashMap::new(), profiles: HashMap::new(), @@ -3864,11 +3877,13 @@ mod tests { )); let old_manifest = ComponentsManifest { + model_catalog: None, external: HashMap::from([("object-provider".to_string(), old_component)]), capsules: HashMap::new(), profiles: HashMap::new(), }; let new_manifest = ComponentsManifest { + model_catalog: None, external: HashMap::from([("object-provider".to_string(), new_component)]), capsules: HashMap::new(), profiles: HashMap::new(), @@ -3894,11 +3909,13 @@ mod tests { let mut old_component = new_component.clone(); old_component.capsule_metadata = None; let old_manifest = ComponentsManifest { + model_catalog: None, external: HashMap::from([("object-provider".to_string(), old_component)]), capsules: HashMap::new(), profiles: HashMap::new(), }; let new_manifest = ComponentsManifest { + model_catalog: None, external: HashMap::from([("object-provider".to_string(), new_component)]), capsules: HashMap::new(), profiles: HashMap::new(), @@ -4131,6 +4148,7 @@ mod tests { platforms, }; let manifest = ComponentsManifest { + model_catalog: None, external: HashMap::new(), capsules: HashMap::new(), profiles: HashMap::new(), diff --git a/elastos/crates/elastos-server/src/supervisor.rs b/elastos/crates/elastos-server/src/supervisor.rs index 943ede18f..fbafd7c5f 100644 --- a/elastos/crates/elastos-server/src/supervisor.rs +++ b/elastos/crates/elastos-server/src/supervisor.rs @@ -1655,6 +1655,7 @@ mod tests { let supervisor = Supervisor::new( data_dir.to_path_buf(), ComponentsManifest { + model_catalog: None, external: std::collections::HashMap::new(), capsules, profiles: std::collections::HashMap::new(), @@ -1855,6 +1856,7 @@ mod tests { Supervisor::new( tempfile::tempdir().unwrap().keep(), ComponentsManifest { + model_catalog: None, external: std::collections::HashMap::new(), capsules: std::collections::HashMap::new(), profiles: std::collections::HashMap::new(), @@ -2002,6 +2004,7 @@ mod tests { let supervisor = Supervisor::new( data_dir.to_path_buf(), ComponentsManifest { + model_catalog: None, external, capsules: std::collections::HashMap::new(), profiles: std::collections::HashMap::new(), @@ -2057,6 +2060,7 @@ mod tests { let supervisor = Supervisor::new( data_dir.to_path_buf(), ComponentsManifest { + model_catalog: None, external, capsules: std::collections::HashMap::new(), profiles: std::collections::HashMap::new(), @@ -2100,6 +2104,7 @@ mod tests { let supervisor = Supervisor::new( data_dir.to_path_buf(), ComponentsManifest { + model_catalog: None, external, capsules: std::collections::HashMap::new(), profiles: std::collections::HashMap::new(), @@ -2143,6 +2148,7 @@ mod tests { let supervisor = Supervisor::new( data_dir.to_path_buf(), ComponentsManifest { + model_catalog: None, external: std::collections::HashMap::new(), capsules, profiles: std::collections::HashMap::new(), @@ -2187,6 +2193,7 @@ mod tests { let supervisor = Supervisor::new( data_dir.to_path_buf(), ComponentsManifest { + model_catalog: None, external: std::collections::HashMap::new(), capsules, profiles: std::collections::HashMap::new(), diff --git a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs index 73f4b4da3..e4e720773 100644 --- a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs +++ b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs @@ -2498,6 +2498,7 @@ fn browser_vm_manifest(memory_mib: u32, vcpu_count: u8) -> CapsuleManifest { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-vz/src/config.rs b/elastos/crates/elastos-vz/src/config.rs index b4751ee89..ca7d6ba02 100644 --- a/elastos/crates/elastos-vz/src/config.rs +++ b/elastos/crates/elastos-vz/src/config.rs @@ -662,6 +662,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-vz/src/provider.rs b/elastos/crates/elastos-vz/src/provider.rs index 060647834..968b9bd88 100644 --- a/elastos/crates/elastos-vz/src/provider.rs +++ b/elastos/crates/elastos-vz/src/provider.rs @@ -537,6 +537,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-vz/src/vm.rs b/elastos/crates/elastos-vz/src/vm.rs index 9a3107a62..cc89cfa6c 100644 --- a/elastos/crates/elastos-vz/src/vm.rs +++ b/elastos/crates/elastos-vz/src/vm.rs @@ -584,6 +584,7 @@ mod tests { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-vz/tests/concurrent_launch.rs b/elastos/crates/elastos-vz/tests/concurrent_launch.rs index 0b839b86c..e94a24715 100644 --- a/elastos/crates/elastos-vz/tests/concurrent_launch.rs +++ b/elastos/crates/elastos-vz/tests/concurrent_launch.rs @@ -83,6 +83,7 @@ fn microvm_manifest(name: &str) -> CapsuleManifest { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/elastos/crates/elastos-vz/tests/smoke.rs b/elastos/crates/elastos-vz/tests/smoke.rs index c10420591..687f1c70d 100644 --- a/elastos/crates/elastos-vz/tests/smoke.rs +++ b/elastos/crates/elastos-vz/tests/smoke.rs @@ -66,6 +66,7 @@ fn microvm_manifest(name: &str) -> CapsuleManifest { icon: None, viewer: None, window_policy: None, + model_content: None, signature: None, } } diff --git a/state.md b/state.md index c36855abf..6a292981d 100644 --- a/state.md +++ b/state.md @@ -215,11 +215,25 @@ The integrated source includes these durable facts: accepted local retention only; independent network replication needs its own evidence. Gateway CID reads serve verified content through the Runtime-owned route. -- The Runtime catalog currently projects installed capsules. It has no signed - network catalog entry or root, model selection-to-preparation admission, or - GGUF content-capsule closure. Marketplace and System show installed inventory; - model-provider consumes static private artifact/offer configuration. Model - and Assistant Truth below records the current Qwen operator bootstrap. +- The current Runtime catalog source projects installed capsules and optionally + verifies one operator-pinned signed model catalog snapshot. The existing + `components.json` config supplies a raw CIDv1/SHA-256 head and a nonempty + trusted publisher set independently from the entry. The authenticated catalog + projects the verified publisher DID, declared package CID, exact declared + size and bounded model facts as `unprepared`, with installed and launchable + both false. Same-name local files do not establish admission or readiness; + malformed model trust leaves ordinary installed inventory available. +- The first-Qwen metadata profile and limits are in + [Content capsule distribution](docs/CONTENT_CAPSULE_DISTRIBUTION.md#implemented-catalog-metadata-profile). + Source tests pass 3 focused metadata tests, all 95 common tests, 6 model + catalog tests and the final 25-test server capsule-catalog regression. + Workspace/Chain formatting, Home/public-copy entropy and diff checks pass. + These prove metadata consistency and publisher verification, + including rejection paths; full package CID and payload verification await + preparation. Installed Homes remain unchanged. Model-provider still consumes + static private artifact/offer configuration, and Marketplace/System model + selection remains future work. Model and Assistant Truth below records the + existing Qwen operator bootstrap. - The signed, complete-closure CID model path is required in the current closeout, alongside onboarding and window policy. People select or use a model; Runtime prepares the exact content behind that action. Settings local @@ -235,7 +249,7 @@ The integrated source includes these durable facts: encoding or writing. Stream/range labels do not prove bounded backend reads or network cancellation. The 64 MiB content import bound and 100 MiB gateway file bound are not a multi-gigabyte model delivery path. Bounded native - transfer, strict package metadata, signed catalog admission and inventory-to- + transfer, full closure verification, atomic package admission and inventory-to- offer binding remain implementation prerequisites, not installed support. ## Protected-content Contract Truth From 6aff190c1351656c6f91a31cdd2deda5c41bb8b3 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 18:09:45 -0500 Subject: [PATCH 071/535] fix(browser): check viewer capabilities before replacing a page --- TASKS.md | 4 +- .../browser/browser/browser-remote-display.js | 2 +- .../browser/browser/browser-runtime-api.js | 69 ++++++++++++++++- capsules/browser/browser/browser-status.js | 5 ++ .../browser-viewer-capabilities.test.mjs | 70 ++++++++++++++++++ capsules/browser/browser/browser.js | 11 +-- capsules/browser/browser/index.html | 2 +- docs/BROWSER_PROTOCOL.md | 9 +++ docs/BROWSER_SUPPORT.md | 8 ++ docs/audits/ElastOS-Home-Journey-Audit.xlsx | Bin 214272 -> 214337 bytes scripts/browser-entropy-check.mjs | 8 +- .../browser-window-close-handshake.test.mjs | 44 +++++++++++ state.md | 3 + 13 files changed, 222 insertions(+), 13 deletions(-) create mode 100644 capsules/browser/browser/browser-viewer-capabilities.test.mjs diff --git a/TASKS.md b/TASKS.md index d26e8ea17..02a60bdb4 100644 --- a/TASKS.md +++ b/TASKS.md @@ -30,7 +30,7 @@ selects an Engine that meets the requested display and isolation requirement. The role matrix and current protocol are in [BROWSER_SUPPORT.md](docs/BROWSER_SUPPORT.md) and [BROWSER_PROTOCOL.md](docs/BROWSER_PROTOCOL.md). Remaining B01 acceptance needs -installed artifact admission, viewer capability checks, approved remote-service +installed artifact admission, approved remote-service selection, and hash-bound device-role proof. The inspected Mac's existing VM image failed its manifest hash check; repair or rebuild and verify the image or reconcile its artifact provenance before qualification. Direct file and @@ -39,6 +39,8 @@ are still needed. Preserve the existing journey verdicts until those runs pass. Browser settings now retain an explicit Engine or Exit selection when its offer is absent. Installed verification must cover a service disappearing and returning, followed by an explicit operator switch to another service. +The web Runtime adapter now checks viewer eligibility before page replacement +or Engine dispatch. Installed viewer and negotiated-codec proof remains open. - [ ] [B01: One Browser contract and an explicit support matrix](docs/BROWSER_ACCEPTANCE.md#b01). - [ ] [B02: A fresh installation opens a usable Browser](docs/BROWSER_ACCEPTANCE.md#b02). diff --git a/capsules/browser/browser/browser-remote-display.js b/capsules/browser/browser/browser-remote-display.js index 964a1519e..90e19e49b 100644 --- a/capsules/browser/browser/browser-remote-display.js +++ b/capsules/browser/browser/browser-remote-display.js @@ -1,4 +1,4 @@ -import { collectWebrtcStats } from "./browser-status.js?v=browser-20260907a"; +import { collectWebrtcStats } from "./browser-status.js?v=browser-20260907b"; import { iceCandidateType, normalizeDisplayIceServers, diff --git a/capsules/browser/browser/browser-runtime-api.js b/capsules/browser/browser/browser-runtime-api.js index 0f2ef5382..2e88dd69e 100644 --- a/capsules/browser/browser/browser-runtime-api.js +++ b/capsules/browser/browser/browser-runtime-api.js @@ -53,6 +53,73 @@ export function visibleAddressForUrl(value) { return parsed.toString(); } +// The web Runtime adapter reports viewer features, independently of the Engine +// host's OS, virtualization or placement. A capability report is not media proof. +export function browserViewerCapabilities(host = globalThis) { + const peer = host.RTCPeerConnection?.prototype; + const report = { + schema: "elastos.browser.viewer-capabilities/v1", + display_mode: "webrtc_remote_display", + peer_connection: typeof host.RTCPeerConnection === "function", + transceivers: typeof peer?.addTransceiver === "function", + data_channel: typeof peer?.createDataChannel === "function", + video_codecs: null, + audio_codecs: null, + eligible: false, + reason: null, + }; + if (!report.peer_connection || !report.transceivers || !report.data_channel) { + report.reason = "webrtc_unavailable"; + return report; + } + try { + if (typeof host.RTCRtpReceiver?.getCapabilities === "function") { + for (const kind of ["video", "audio"]) { + const codecs = host.RTCRtpReceiver.getCapabilities(kind)?.codecs; + if (!Array.isArray(codecs)) { + report.reason = "codec_capabilities_unavailable"; + return report; + } + report[`${kind}_codecs`] = [...new Set(codecs + .map((codec) => String(codec.mimeType || "").toLowerCase()) + .filter((mime) => mime.startsWith(`${kind}/`) && mime.length <= 64))].sort(); + if (!report[`${kind}_codecs`].length) { + report.reason = `${kind}_decoder_unavailable`; + return report; + } + } + } + } catch { + report.reason = "codec_capabilities_unavailable"; + return report; + } + // Older viewers without a codec query proceed to normal SDP negotiation. + // Null means unreported, not that any particular codec was proven to work. + report.eligible = true; + return report; +} + +export function requireBrowserViewer(displayMode, host = globalThis) { + const capabilities = browserViewerCapabilities(host); + if (displayMode === "webrtc_remote_display" && capabilities.eligible) { + return capabilities; + } + const error = new Error("Browser viewer is incompatible with the requested display."); + error.status = 400; + error.payload = { + code: displayMode === "webrtc_remote_display" + ? "viewer_unavailable" : "unsupported_viewer_display_mode", + stage: "viewer_compatibility", + capabilities, + outcome: { + schema: "elastos.browser.open-outcome/v1", + state: "terminal_pre_effect_failure", + effects: { page_acquired: false, vm_acquired: false, stream_acquired: false }, + }, + }; + throw error; +} + export function createRuntimeApi({ launchToken }) { function homeHeaders(hasBody = false) { const headers = {}; @@ -97,5 +164,5 @@ export function createRuntimeApi({ launchToken }) { return payload; } - return { fetchJson, homeHeaders }; + return { fetchJson, homeHeaders, requireViewer: requireBrowserViewer }; } diff --git a/capsules/browser/browser/browser-status.js b/capsules/browser/browser/browser-status.js index db1f5e6c0..89867d3a9 100644 --- a/capsules/browser/browser/browser-status.js +++ b/capsules/browser/browser/browser-status.js @@ -63,6 +63,11 @@ export function friendlyOpenError(error) { return "Browser session expired. Reopening from Home..."; } const outcome = runtimeOpenOutcome(error); + if (outcome?.state === "terminal_pre_effect_failure" && error?.payload?.stage === "viewer_compatibility") { + return error.payload.code === "unsupported_viewer_display_mode" + ? "This Browser view needs a streamed display. Open Browser from Home with its default display." + : "This browser cannot show the Browser session. Use a supported browser or enable WebRTC."; + } if (outcome?.state === "terminal_pre_effect_failure" && error?.payload?.stage === "engine_compatibility") { switch (error?.payload?.code) { case "incompatible_engine_protocol": diff --git a/capsules/browser/browser/browser-viewer-capabilities.test.mjs b/capsules/browser/browser/browser-viewer-capabilities.test.mjs new file mode 100644 index 000000000..72fc5e299 --- /dev/null +++ b/capsules/browser/browser/browser-viewer-capabilities.test.mjs @@ -0,0 +1,70 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { browserViewerCapabilities, requireBrowserViewer } from "./browser-runtime-api.js"; +import { friendlyOpenError } from "./browser-status.js"; + +function hostFixture() { + class Peer { + constructor() { throw new Error("Capability inspection must not allocate a peer"); } + addTransceiver() {} + createDataChannel() {} + } + return { + RTCPeerConnection: Peer, + RTCRtpReceiver: { getCapabilities: (kind) => ({ + codecs: [{ mimeType: kind === "video" ? "video/H264" : "audio/opus" }], + }) }, + fetch() { throw new Error("Capability inspection must not dispatch requests"); }, + navigator: { mediaDevices: { + getUserMedia() { throw new Error("Receive-only inspection must not request device access"); }, + } }, + }; +} + +test("viewer eligibility uses capabilities without allocating media or requesting devices", () => { + const report = requireBrowserViewer("webrtc_remote_display", hostFixture()); + assert.equal(report.eligible, true); + assert.deepEqual(report.video_codecs, ["video/h264"]); + assert.deepEqual(report.audio_codecs, ["audio/opus"]); + assert.equal(report.schema, "elastos.browser.viewer-capabilities/v1"); +}); + +test("missing transport or receive codecs produces a typed pre-effect denial", () => { + const cases = [ + (host) => { delete host.RTCPeerConnection; }, + (host) => { delete host.RTCPeerConnection.prototype.addTransceiver; }, + (host) => { delete host.RTCPeerConnection.prototype.createDataChannel; }, + (host) => { host.RTCRtpReceiver.getCapabilities = () => ({ codecs: [] }); }, + (host) => { host.RTCRtpReceiver.getCapabilities = () => { throw new Error("disabled"); }; }, + ]; + for (const change of cases) { + const host = hostFixture(); + change(host); + assert.throws(() => requireBrowserViewer("webrtc_remote_display", host), (error) => { + assert.equal(error.payload.code, "viewer_unavailable"); + assert.equal(error.payload.stage, "viewer_compatibility"); + assert.deepEqual(error.payload.outcome.effects, { + page_acquired: false, vm_acquired: false, stream_acquired: false, + }); + assert.match(friendlyOpenError(error), /supported browser/); + return true; + }); + } +}); + +test("an unavailable codec query stays unknown until actual negotiation", () => { + const host = hostFixture(); + delete host.RTCRtpReceiver; + const report = browserViewerCapabilities(host); + assert.equal(report.eligible, true); + assert.equal(report.video_codecs, null); + assert.equal(report.audio_codecs, null); +}); + +test("a native Engine capability does not imply the web viewer can display it", () => { + assert.throws(() => requireBrowserViewer("native_surface", hostFixture()), (error) => { + assert.equal(error.payload.code, "unsupported_viewer_display_mode"); + assert.match(friendlyOpenError(error), /streamed display/); + return true; + }); +}); diff --git a/capsules/browser/browser/browser.js b/capsules/browser/browser/browser.js index c9a6c73d9..cce588fe3 100644 --- a/capsules/browser/browser/browser.js +++ b/capsules/browser/browser/browser.js @@ -6,7 +6,7 @@ import { normalizeUrl, sameBrowserStreamTarget, visibleAddressForUrl, -} from "./browser-runtime-api.js?v=browser-20260627b"; +} from "./browser-runtime-api.js?v=browser-20260907b"; import { createBrowserClipboardBridge, } from "./browser-clipboard.js?v=browser-20260725b"; @@ -26,9 +26,9 @@ import { isAuthoritySessionError, isMissingRuntimePageError, requestedDisplayMode, -} from "./browser-status.js?v=browser-20260907a"; -import { createBrowserRemoteDisplay } from "./browser-remote-display.js?v=browser-20260907a"; -import { renderServiceSelection } from "./browser-service-selection.js?v=browser-20260907a"; +} from "./browser-status.js?v=browser-20260907b"; +import { createBrowserRemoteDisplay } from "./browser-remote-display.js?v=browser-20260907b"; +import { renderServiceSelection } from "./browser-service-selection.js?v=browser-20260907b"; const STATUS_TTL_MS = 4200; const PAGE_STATUS_INTERVAL_MS = 2_500; @@ -70,7 +70,7 @@ const launchToken = new URLSearchParams(window.location.hash.replace(/^#/, "")). const homeParentOrigin = params.get("home_origin") || ""; const debugMetrics = params.get("debug") === "1" || params.get("metrics") === "1"; -const { fetchJson } = createRuntimeApi({ launchToken }); +const { fetchJson, requireViewer } = createRuntimeApi({ launchToken }); const homeClipboard = createHomeClipboardClient({ targetId: "browser", homeOrigin: homeParentOrigin, @@ -1835,6 +1835,7 @@ async function requestRuntimeOpen(value, { history = "push" } = {}) { runtimeOpenInFlight += 1; try { const { displayMode, guaranteeLevel } = await launchContractForOpen(); + requireViewer(displayMode); const previousPage = currentPage; const previousGeneration = currentPageGeneration; const previousOwner = runtimePageOwner(previousPage, previousGeneration); diff --git a/capsules/browser/browser/index.html b/capsules/browser/browser/index.html index a45fb790a..ae1771a4d 100644 --- a/capsules/browser/browser/index.html +++ b/capsules/browser/browser/index.html @@ -88,6 +88,6 @@ - + diff --git a/docs/BROWSER_PROTOCOL.md b/docs/BROWSER_PROTOCOL.md index d4c0a45b3..25b373a5b 100644 --- a/docs/BROWSER_PROTOCOL.md +++ b/docs/BROWSER_PROTOCOL.md @@ -111,6 +111,15 @@ Diagnostic and image-polling display modes are rejected. ## Errors, events, deadlines and cancellation +The web Runtime adapter reports `elastos.browser.viewer-capabilities/v1` with +WebRTC transport features, reported receive-codec MIME types and an eligibility +result. Its `viewer_unavailable` and `unsupported_viewer_display_mode` errors +carry `stage: viewer_compatibility` and a terminal pre-effect open outcome. This +check runs before closing the previous page or dispatching a new open. Codec +queries can be unreported; actual SDP negotiation and media proof determine +interoperability. Headless and native operators retain their own display +capability requirements behind the same Runtime page contract. + Compatibility errors are stable typed codes: `incompatible_engine_protocol`, `invalid_engine_status`, `engine_unavailable`, `engine_not_found`, `incompatible_engine_capabilities`, and diff --git a/docs/BROWSER_SUPPORT.md b/docs/BROWSER_SUPPORT.md index f917bad96..715dec6f1 100644 --- a/docs/BROWSER_SUPPORT.md +++ b/docs/BROWSER_SUPPORT.md @@ -51,6 +51,14 @@ before admitting a local VM. The kernel defines API version requirements. A successful query is an eligibility check, not a completed VM launch. [Linux KVM API](https://docs.kernel.org/virt/kvm/api.html). +The web Runtime adapter checks WebRTC peer connections, transceivers, +datachannels and reported receive codecs before it closes an existing page or +requests a new Engine page. It reports typed viewer incompatibility without +allocating a peer connection or requesting camera or microphone access. A +viewer whose browser lacks a codec query proceeds to actual SDP negotiation; +its unreported codecs remain unknown. This eligibility check does not establish +codec interoperability or installed media support. + Viewer adaptation uses WebRTC capability negotiation and actual media results. Qualify decoding, audio playback policy, datachannels, focus and input on the selected browser. Browser names and user-agent strings alone are insufficient diff --git a/docs/audits/ElastOS-Home-Journey-Audit.xlsx b/docs/audits/ElastOS-Home-Journey-Audit.xlsx index dea330c6ae087be59d0534ce5a1d799633247be9..274b619a36be016c7d31743a334e5305302f2c3a 100644 GIT binary patch delta 13780 zcmYLw1yEaE*EIo(ySo*NTZhw{nN0+vT^m2cRU@TTEi>x_(l`$5G`RLgi@1Be8(^k zSM*eti6R@;HGaKdM?E~fWy2nh$e8P!-fOk_vXxWdQvQTf!gY4x`o(8CCH;{^CuMAv zF?o%6AdeR+T``klTWMsEZ02v6((oCCsKSw(1J#FPoo^UQ>7hRe`|=_EY@ondDp|32 zBo6DDq2xUGV93?Pow?4WG}4*%{@Fz?Zm2Be`K$AJaNntIlwAdwB&Ke*oc~Z{DTK~KB zr$dI%XCG&5r?$++D?SCWnqLg|A+QS`9#;}vfV+W$I#ESd{!(_^xGUe;G?C?RMxb0(570@kI%~E55LByiWU# z0weMX*BrOy_eD1=_>FsAh2|P0)A%!1?kMSxyD4YnOXGu+78mZ-Ly>JDDR-^EzC7)T z8|a8^+BN(B8v6G)t-HRdIdrS&t>mV-7Dh?J7k{9ypWR<;gk;xF$&X$`ahq9kxD9EV zm3bZus<&@;kRK=0k4lv#A8Gi!QJY(OfvGFK7KNxeXG}X&GROKlU6y=>vCk=28(WiO ztdr)g;au$KtzIxPu`G!jpTf=S4Gb3Z3#2l`Zzxp7eK6*`DqG*Tq(-E(pCT+jnfePL zCv|Z`%`By|P)(*ph3xwv!db|kglsTYtm(lo3!UjGK$2*Yc4`nRS`ADjxuKw!0Ee;3 zFBub@D`u>i&taY0b}Ih(KjfpA&6z2q!^q^g>EylFewGm7B9 zX%P@x;_iN`R&l-ehhqT@>8zRYF*ZAsoDD z2=={{s)|&uBSd~F%X&ofzuP5s?2$_L#cD1}pPkJ+F*KNRwo3VBys~)aZl?y?3zN5B zYX7L%*}<^VswTH18~7P$u_;s#Q!*@j5El=?4>#?s#Q&3K9;DaT#M9h!4F_FBeOBiuHqL{|^9u>()(+)U^Ahw@yC zhC-ujwzVbQq~;p5LH1s3Lj+EQYDs987S2NCZ-8@?Y`e;5;5*-lu~C1#plFY7?m#Pq ze$Jy{ukVkhmzdQkEv?mze~f)RP21(tp;;r9aN;qSfg*l+xv4;sV+=BU$dyXwT37*7 zg`@+N64GR;W}#(S=Huh6(T22`OKd=s zT1)C+;`ybB#+3^3$Zn!9w4QxQ6qfT;C)b16B>(to<$pt(NSH?2rBiJvn8AwQsm^F@ zg%(o;NK5DACaRFeg#0|dZm{xo00?kLcrUtycGw*0hq#RWb`=W$o_YkX3eA_j4eFpdUR{;@n?tMku?;Tg`mjpA>1$98~sF>D4 z4aNx_Zswv*E8d#5k=n-EZx#l!?O^WU_vE36Gj{d#+HrUQJye=q5wEb14S+fIH3WSw zJbV(0>JzTV3Q}|p`h6$izFt|1TLwE>%-BRW@Cf6jg-g^Xg=3p4*w8F_4Hah4eYQL$ zh|fG6qCciX;;R3;e#FD3}vZNeD3Uc)v!5s)T;&7INp*g8n4xNZmb>?9Seg zP+Z%8b1Vnd0$)>h7E6WOHN9%EH{Lq7lKv%s1Edx%4;_#sc`7A$kx9Nt$=e`Q@k znH6A?9}NhK{kgE;&=Ryx?3zh;4E9eFQUb)^C$A4W*k1SSy z`Vd38I-1>;^&>P`Va#$U-IeYNJj+H4WWXC#WAJz~Q5stXx1Ocnyxa15=|C2{jOHen zlSOw>3Axzf#i-1@C8z*5=I>D+FNI70cge{VNkyU4EdD>y7}>|M6(oDOG0MRZ!LB5x zeTwu?V6CcDp_qiDh6Fiti-Q&O)R!6WAuyY8!0B9I-NTyYY%c~JX_0~LJly*UFCavR zZI<=K9txhqbQv=19U^HrhzB(wR(b;Ig`KsT4HV@6_^*SBzU@EklD>%o%>BK> z_pFx(__ufOMpd#t^jct`{(Wm;R0CmM-$CKA>Q-dIbNDvl=+5U0HH0Ca6esDw)i$6x z_-|1G{v_cn-0^dH+~gAw(9kaRWWi-ce$;+pgMY7(mJ&;})y$nUdeshhR4#QX_cFC2 zDyfYAHF?q%gk^w2S{;JKqMi#6htzcw{^OpiG}WKsz%)MQ?Arf4!0;~KuH-dSqs|pE zM=Qje8QEx9QoFinFO(zM9zicylSQRPUM272pXyV*z!H6z6`(hX&J!r4Pp<5bdI0W;_Q*2diM?9M58Y~3YHtW z+T_!SM4#OYrI$A{49{ijz%co4F2e~_IF18`2NZzxjitqy^q@0i6$coim(;-Tx5k2U zdvE36r{On5a~wTr-riNRJ=obYtx9WDFXdY31yvBHH9_vmoLQjYSH{e#pgV|Gx$J8H z&PM9^K1j|>O|dKl2XJ4+N;jlB7Jdtum; zoMIV7`l;3H$|@6#85{=8QM*-ja6moAx1Fn;bMIbw1!?d>&4n!dpw4YOr?_kPiPm)_ z>oE&~*Vjxg5?>NfenNs4rMS2vR=eu^tn z)N`;j=emftXZ0>fB>#no>pMTi%1_WH6xu$7vqpAYsq?POk^nU_W7JvT%wBV<_TR+<`MW}R=JPE zYV^k^P!xX`NMCN2kNaQzDQe-$-QySEV=*JCe|hbqwo1L0b#Qe!V`xDvjSKRAL^vE0 zz6lF=xa-qi4l^M4@ZJ=s!5|dQZ%e#+dM}1E?H$|p86X=g-ElMCytDADJUuOCU^x$3 zEde{X1Z~95aa#8}Wiqa#wOlZ>qqpe%+hbkEye*9%yZK{We&{9YKNA9Ziz3>Cesu#u%Q9}ZjB-o?@>j*i^Xn(Utj0?$WEj(( zU)o*8A#cF^nL$`nQV#WrRE79o% zo6&Vc@{~6k%ZClrP1&8_g@&G)zf#|h+hl>rJ_6 zYXsUp!ycQQDMtkgaO!UN=ExCBHM)9kZC9p;+}S9s>WQ(y+F*KtyvwwfJn)a!@mr+? zNT*DayE3`H^=go}>9AOfJBCr>-Dg=!sA0a8Nq<=4=pN=URn-ERuszs&*VY*!E{x{! z$xYN30GVu^rg*Yig5#y7ES7y$A?RwSS)vDj)cR|VN7@qn&hfz`0ZUfm&?^mK%@dhl z%dZ45&UR}Qj3Up{m@ulN;P~w=)Z@~<5_SzZy=7Zd-;NNx{!8M&Y_km+1>T}-2?!*bVf@!J^db^pNhC)AwAQCR%D7kYaRNP%onSGyhLRaAqaNtPm`bQH z0*9ak^fv(54Hk?_qa?#Js)Z0FW*o19tE4SU@fN6ZhAWi;d1Xkg3eR;f5wGjrg?*Ad z(zsv?Z^xieB0bI)nXy~=Sy%GY%b2zz^;He3RWl=i#E~3svc#=VC?Zu z0pYRYl@7gAwr%g^G1fuS4JjrMino9p%o%h|hs_aT^4LG_yJ-% ztTyUR_|zhEv1`JMJ1ihf2IgApUzuAx=RJ2%gy(_9cpY#;V>@Q7K0aV9Tl$Pv_s*l9 zAnnyc|0n|Mm;M%h+Cp3$6{q=awE938Ae?x~&#KX+Pisq?wsOFUeKKybQ#_niB0{I+ zBv%LV%_8))pF_I8+nj;Ju+Ez}a@8^GR61xr&NY{>4vf zaVj>BnqW9m=N4j(y3Yx*{K6mySXVQ5 ze*ryQdN4h;o4SeV@}*T}I{F=nk}dvX6aym$OLd%+E}MQDk3udHZo&6*1S2UM$X%^+ zs#7ek!_SKLknigT;(2f%_)Ge0`y+GPiU{F1Q}2Sf8LwQN99TzR}Tg(-4cz@GOTPBTfU#xl->$`BNn4*YX&9ymg46Xe-lTSP-JT8?-V zxzv?tR2G7bZ{RI$_6~p$osRRAOot@_c|VuJAGG zIdaU6CK~6e%)T1qQ95Qf3#V7wXY#tU_A(mQl!Q#I&$cv|_Ndml>x`Yk8`qZ@J-7W( z4$ZSsTc`HcJqwUfSdbDl!yse<7IX{9-h3D0pVGzuS*oHYqq`)QEbG~=`18v(;o5wa z5VldO;>kBuv^{$)Z)>VDZ~{T;uo>Q*P}=0Ro3>t+T#B6Ht{!g${6pShjmInw{M@IA zct#_WU&we0mOsm^wPevS+Nw3|srT0u8ZZUeKMfDa*DY{AzN%|)yz7{PYIBqVo)XRD z^&(}wv8a$MRbC(FW*5Qym0!D<3a`?uc!mF-*X)%C9cx{;OPy*5c1df489u)mH&9GQ zDtzP3xCj$&q+$9C@&AnbZ~taogo$f}H{!u)b_x2p^;=c)ECGt$R zyeAUvM%C2WokI_3E9eP3X=75biDSOoNxY*ZCdxp14Q(%oS;%;EK=D`T6O9M^J#U7Y z8J~s~*h$+_oN)@dFBH*VJrNCpG!!h#=Qmy2}ijDt%tqt%zw)be!uIm1k;VB{vvG ztBV9F+U#pV|CZxMe&V^M6E19fHz4ShSt*h0q!#}Ak%%CNaG#^5p~Bbe>?;FoUz;r@ zUK~tr(_L@7rDvqmdLh!hML&0_QwvECUtz_`@1gwo7HV#8THOl$%iGJeeEaq!OS#5c z>zLQ;F$x!lo1q_gWgnWRTxMa6Uv)zy0-raV@Xl^`CM30y|E?};wI*;x`~{p%a@P$< zn8qLD1ismhga``vz~>I-YiTBg(U^dd&kI)cs5f8QB7AU;!N&qb4_;kRN^s&xQJc=p zkdj_EdWco#P0N(@rBWq~HB+anV6`{&-%ri^0y=53uO6@GmY!_f)D42J4?CB8N85Kl zYM)uGvh-S=>y;ZPgSNYuHUZg7x%O)O*wvopv6K(LYBIVAWps5^YRXbsDh1oUdh3H; z$Nw~{SM(9Lj8|k^??7! z)j@7=Wf_G|k;<8X9+MX(s>P4j=l0%+?HQ4K29hS{WQV;tfpm$P^{{8l$ z`C`P48J_Xb7m^V}-rR2wj{F+8T0$cFVq87^c~;%m7Lx?G_#%w$W;uK*VsZz~h|bwL zuETnFRgE+_&2fp(jWTCWYzZN9#fImpeYDqhfyV|H8mxnv2y8DowCz&IWBE2u-mLr1zG?~FDZ`l^4rwF$}5reYUDfK9tVdA7EOWbj_2?YD#o&{z2zFo^<$#RD_z>6 zP0pUqwVTS=>of4uJ>KRwiT}qZC#5u5;N=;3cDU``UVEx->2w1Toy@Z({rS@kjPi;< zZVV-3RO8kij;258qa`qA(CH4-gzxxHLKicl>6%}vUw7FW#9MhMDxglrf6PBs98&)! zZ=aHnZ{nTvHUEen)_Kl{(BBAcAu1QY`3VY6xt~PjrfTzs8)@-YZQjY7mtzcbvNpqCb9rq%CL#q#0Hq1)V z9|^M~XN0I4zVg?3AGK&u7}947v)0X%J8kKt65)I!?caO}m&xnTXfe!FnAQhehcJDUj!Z`W^c2P9^-*1J}QI&iEMx~&wm^pnfYxh{|C4(JaBPIbcNEGF6q=1ACGzc_8os3%XV zWA&&~sHIKobN5oE?tJFR@vcb7u;P16m;NdH8ln?@0U!bwUy3A;WHu^H(wW}O_nfJKI)8I=WJI}6y`L*%FZm2nu~9S85K4!`H#PYB0ek{Kukvl zUspg|RO_bScKROsb+gWd`BHC+*2i5|q{?|E=;PU{Ls+l_Jij!)a|B5RGGt|=TYaeX z?>yYw0<>F|6}27}2Xx6(kXueXW8oguf0^A=LOsEkIWWBPlF=a}B?OM{MQh!oYi^8)m~T#=RHp&okp3R3U*s@(N5Z-o~XQ2}AZqYt4Q0Op6XL8vYgaE~yI%i}>)< z#Xkh=GA@IAgU8_CJ=nNYTU(7p3^wh)7eGY6?}gL=74irC*q)2H$CEqVDj=2ad0}i; zL=BB$&Rd*iCXn}ju4QZw*+hV)tfp+s%Yp)!o@Wq}_Gdq8HQG&<_5c`vHfk3+LmkDQ z)8-+MzE*eI9N7`K_MJWB2CL*UQ=~W1NT~)hZ|JXlt#|4mwrF;E$E3t@qhg<{saV$IIB|o|B_NB${U69H$jNnIrfuGPpdvjEI>U-ej6OKIcMmXjNh<~ibsAuo^Vb?Uw1V8UQ6fnV3Qu6%q zl@s60s)?P7UGCZhmKtt~>N$ZW5gg@9VvpxZR$yx(uZlPFqEkYM7&1TB|=Oy2_fRsb=I_=n$a$1(})c7 zX3wQ5^sK-K3>rA*OSw5)0RrX-AGOx5D7U_2$UL1Dg_GYt;$rZYWUz zR!Y^yq79De&7COx6n!jfbG^V9f6XLD(yMH+j-ZdkSMd^)-BL3x~(fftL z1`_a8x0|=AiD=X;Y7m{_->z)6<`Dm;9l@8~qjlt=N!Fu<7eTYmtL}Wp50o@yEaAHR zS__W*FnABunM`EW1O#blX~e;U)o`VFE_bM)C;dSz5&YGfq#=(uP}2h++g&uls>?Lc z&RF3pdX;SJHGZ}>J}fm4aZ-z-vpNEn{){a0jGb}fmxD4?dq>hk9gMS!0g0|%3$+us zWjN^9rUNp3>*cCX$xaCqYGuP_Z_ArJ_G(MOQ57Px{~kF1DPthvDVn)Gc*}RQ?L*ow zLx4$(xVbKxOv)oek;K96`!!1(*0F(mp?qpViUR>uRPw$EmBIM^WPy{03=~&SG%Dt1 zW`5M=&&FTNGIx>`9?#v#an;CiHNIUe!lG^tt5NNIZF6ymQB9FGMnDn~Pt~gHSkWhV z_uqrXl*>S)i^KyyhymfMsglP@drexwhmm83=Pv>gZZfB)_fR*?E$M+g8-=)Z#hYdz zQCI8bT1AEOf11Hk@^MzaX(ke`So)^f+wvy=7e>D{i{CUmhok(b7{->A!sk<;z__m8 z?0?tu`|Fhd&`i|^+nTUj(CY}r0ozQSk~gZLtd@-Wq>V+Cx^iTU!ftMHCyzTKtf~23 zLOH4!9F`&2Nv10%4HNswuh3=pX6`Uczqk0eGIClsrX{85kMNA2YT3E4+z;dy3tLy% z$B^GE%Kk#6ZsBw&mTb^DcgED3er&nRRKLdxx3_Z;$XG5gAX;jLn4m}-^_vZ*c5lRW`Apwf=E*S;e!L4p*H3}C!1aP{+pEp5;TkbRK?y3n%HZ`Z6G{ zSA>RUpMuUT-Bfc=Gt8mn0daqQlkaz)uLv5Q0dehZgc4Uy@e}1We*q|@gi5w@OI0K@zgNGBw)kHFc0&^QvXdCV-x)S+>MlWet zEJHVg2PbV|O?$m4D01?2K6%4H`R>@h(w6$wpV1A-pCfxK+YxIc0)rBs9286(6fGPS zO1Wx*a>ZD9&dqS8x6~ouYLEkUs}W2qE?TuW9}vuc?>cCIAtx*vNM5h}XIgq_bw%kf zX;(_R1PFI0-PJV`=rif-odZ@+PpQ4B6Gy-f2M9QOvSAv;RymHn&CDjAor2FLirh_GN^ zy*XVUuVDY5T$*(aroi<-AH~hQ`3RQYKdBe^qBOEmn=ybb9Jw9N@%f**TY{$yzPtbLpCZ`LDs*pbb0Q#IzXxmbIyO@<7bQG{1O zM&&c+4o&t?+zt!9bkZmkv><$mna~xhs!8DpUy_(E{64`XOZSlMMxWp za0fD?s-$h&k2g|QV0Yg4)fj%-mE?Pe0&|!JqH}`_>fgweju1hHAqsHQw?o)tSw&pmL3A;Kr7op{-S%}gUCYK(QF1mSz^;m7HgL*aigq}|G6;!4#* z%4I$Fc%M!Zf1SHycZQdItsOT)ktj&7#$-)sZ>2rt>Lv?Us#Ywu3I8`$neAH*aL5TKvECGI5nOQecTPas4MRz&{HxX+seu58w_xVCcg(B@i;-uBH8S zv8d30e|TMJ>*B2Cg{dU8ZY(WlTuH4qMg8iRwb@S6z4;;r{s^t(96F6vuc-<>!2~XN z&XpK~?4B4wagatNzRLre2;5N)+5U|EF=H(`?db#Q61MyvGFOYz36uFl%--HpvE{}$ zsC_`7&NeR@q9v#17C$g*O}R-JKtm{2Q-OXo8=f!$MS>Y-q8cYR=4|8b!{01WghF*4 zkNaI*vZq29b_489{L4BOa{zi2%**Hl+&($V3PXtFp^d|r4 zD<&XQpKsx!%p;FMow&3p0aCDIi^uSCR6-H$pbX3QhQoa5J7wjy5)(UGK_ja7XbldB zy)Yve->y204u|zN_~srdsH?vKyAYpK4dpN1w|ncPs0TfiO;!JeCt51FH(*l%b1!P% zzhuAnMF8edusI$$%9_=?MCqwVutlv*rK5@e6|~z#8Kde=83An>`gfGQpTIfKDZNB6 zJPMO6HnZ9fq8c5Toi?@Q3|iRt9sH7YrcF|RplByCbXsu5sh3R%_7}5;-oqN}dXWvQi1oB&Lq=;6r%h0flxT{Y|ZfEqL2#h&+ zA|Yl!sWwS~j^Fwg#|0#Y#Wp=`96~EM3YlME)uK~%oxi@r_}xGu2`#cynfd`_ef(9< z-i=Ba%>#QaPR#hi9KA?u$+f;OoPk)NKol*WjgSIeOfGhKXq5O02L6j~s zBN)vCu-F=Lpfp_DfrfI5PvU|Sr|*m#=u@CIMvXTCJURB?r9=NPn8tSA_wE0ZlppWXqu0L1!esJ zf)Y;>;D^3Vw)C;RG$?dw%7wspTUxmzi5+P%b30eK6c?_i|ARaC;A*+ z2c%(SSz&`;G-=}f%3)iyLL^c&u9Xyln62MDm9_(17FE8J`SF$T{N6I-1M&6I&xk0D zaQ8CC1D&ac-m&)l1EOx_=Oy@(Kgg(7*^Fboqj?*c<>$FEFel{ms{Fug6lUQ5 zQ@8A2?m!AYpz=db2n8Q{jVk|4)Ge6ee<2(A7P5$x;pFJ&i;3`_$3@@85vXdrTFPC6 zDNZFLuDHBgdw%Qpl0u7}lM$gSKz-Z*_baq@MI`V_F3Uq>MP+L2)0%IO9#1SIS(|V2 z@>moA>1V_=@VwNHW^w=%j6VrM5K?vB!2@!~*NFtL4jm*{s=e!}v+vj`B%*oY=;nN! zQ81`{-DrQa4P%)z^ya%e%aqi_|jGN*ASdjaams|3dLolhUp|*lx z6sfEKoBUsyCjDQTcHjS3rr}xrR}nQ232_6w+~}EZ{uIC3uT{V`#BzZlz8zz&c*<8J@bn$pkAWh+gDe>VXEOeOe3)Aip9!T&!8zs$R|QfmsBy-AX_YlDjdm^VK6p z#>(&?T2Z^~X_RL(Q?+yER*u(P!Y`4&zBP49B&vP+l$0LWGCx1hHpl;tUzIj4KHZ$+ zd$o4uhl>rTM85nE=w6xB@qT~OBQ>S6Ri8wkR7BCzXlXb;GmK)_`Hw<;CNXynKzj!J&04@f8W7t)y`ErG z08uuH5zak2IB%=BK~e;D0->GuqElcs+xT8?-w$C61Ga$VzC{QTQpYwqweF#v+SyNW z3g}sfT|THkh`gyt>_gY!v-iY4TKS7+;3lkz9@mU_l5}kN3c9e(8R5l<4{1Eo>_Q=vZ@%2N!bXE;FIfg+M+2#Z@d`eHFFr4W|F$ZQNTGYIDoz5(SWvqsZk z0*3Yq&8KLJ(2?!S?gLh|28dLByfZ*U9W{PEGpd4Q>&mR5>yz__c$ z;+FK6pg||%>dP+cfULALCZ?A8p=+=Y{k$E=P(7}`O8v(8+}w`TjiGRP6PPU+?pC<~ zfmwSUBdJ8(Jv@aYQ>NHor9q0LloG>1+x~GIovq}TCoraDuc~VF$) zK0HzRi%jWV@1!t(-N)*~oRNWva>r3*<_w#c3hm92Ng^yQUFpSRVecEF&xKErn#IO< zuo4v5_y7y@5sopgG+wTRj|?Bn@z&UJePiP)g0u)mlDfMc;U@wf>|fs_t&s3I5x%E! z@RgWfLhBjJv=J@3Lzh{4uLojRRkN-b0T^Ly`$n<5-(TV}9SAcn%f1z{>6s0XW5Q8O zsN<=P9eE*4&pTyrg}pk$YCqGkv9l)Te>+YYzyd~2G$f^Qqm-N$kJD=O({MlQKcB)9 zLGm}SnqMbPQR(?^3`4?h#}q|1?`a)gtVg^L(IvhAl5x93Opb|yZnIW3!fB)8lB?;$ z4LG!5YTWZK1X2gfahA~HCo@i?k6abD35$*zYImFqDDD1yP-gSATYvZ0t7+czsC1D! z;}cLlW|_uZyo<9actPrze}+i=K$PiN#NbD%c@c;$YEY!N_nD|TytIz)rjI0PHy1fY z=fFG$mv{|E6Msc)sbI+9iV#JGv|*dHs&5#eof=PKI45e#&{PNZ4Pf<}%$E zqN>w|Cf`9Pf+!*%igkUW2nmjd9^NshbO5vwtsF`2a%O zFNpr9KI5T10u;GOBC~i-esZ8`X+A6B+Cz&n1uUwK@GfGhmaZ;G;8_Ic;c*Z%5ZFM9 z!LT0U$**4b`+Ok3lhZ@Ga#kut@J2N@!ddzkIpG{grr%6FfVnI&49{XOSR}EIbM0lv z_nwr;NraM6*~JrP87%friBI*m>XWC%(8ft2bS8$H^^_;71A40e*GKm}bz2+L93^`ZRIBL!Apm8P_XDik#SAq(L|I+ zc)L3+iAJLOYH4td;`TfpG}SERjX`uSi7xuQV23;EtM+pgQKc3!SH$us01j31Y)nvM z-siicw-rH{Qp;0gr3f$Is%n%vIi*A7pZY_tb@GRD<4X>%u0xB`rdb8p=v)-}QCK;}LrP`MOXM2CNoah_cvE9GGXa zm0>}KSJ=}(B1+G_{{~V(&6V(H6}#t`$SP5O$|8)LUR?p z#{Kgr0fX|h-WI(XGbCy?lYh*@(QCXEH$IkGVWi4@bl~6L7%2U-xI)qQ)T~b}bInj? zis6}z!#yk~{Zey{h-QEwYQ>K#DidU}_n$5opOBOTR}KF52G99v4u(r2Y4^62r{S0*4WXk_4t(nlwuUK-y^)Ti2I-8 zlJQ`qKMS(hLmjasw(7)w@81NqYFu`gGYtZ1(au>KyMDufe;_}c3@B%;UbGeQ4kp_h zyn{`k+d13dd3`JLm!FgP9I|A4b?nOJo01t*4d4IkUgUhxx|Jm-V;)Z1gq#b)h`_ASff6y1 za3In`Fsz^mh^G*Y2uKyu_O`%$sQMp-B@Dv~je!gi6NV9nnL~kw)S^HWLUM#*7(i)| z5#hI!LW`~^g9Tx$QzIkQ5AUu$cO(2pbx(R)&GDV$nqNinud^w!tlUQeS(3! ze1su}l#0V(K$19MFi_sK`0rH^FvtOk7kk4xIbe|fci@*O3=woU=l{e=VlW)gEnNS5 zsN(v6XWYbKh(LpoI58MyP&Z`rZGmp#{of%)@wZRZ^8N22=j{Ph2ALIy`2Z&%0Q2^n F{|7I~LCOFC delta 13708 zcmYLw1x%dH_w_FBZpGalio3hJ6)09*i?e94Qk=!zy|{aeySo&3w<7=T`}>k_lg-TB z^E}yP?wxba++=&BPy?e-YZoaH5uTRYzQF(i*!5c!UK5~`xt8G&wdLfg86w}1zM`&Z zOmzv{l&YFY8sl8AQH{lR zA%6Ei9GK8G96K$xb2ldsv%Vn8@ln|I3TWn!n=fXMtH((8c_H0Hf_zk4UBx6 ze8Oi~qDDZOF&+ch3IpW-O zr^kJGVH`V!rSS0@b0M|C_TxQ0cQSL|IoLr{`}qA@$Vo6C0Z?=6*^r)-*{wXqFSLE%vRE~~-0yC?buDm&^!<$2h$6kT;TTKWG|d`_31uwkxFxYB@l{Zz^h0f|*YeOK+&L`<4cTwyg3n z2K+|G+lw%uSr5k&H*Y9FVL#!`&Q{YU0*M?;KNMkzjVzi_`Q6A?w<#pZW4>q$K9Iaz zzJA;aL_d1evQb_tEt6dBY-8S?VJ_)y`MC*NDTbOrPg~RW{1xZpt=%%4k)o3%rS|3w z=F0e%-Jx~Y8Vto2m*Qi{_M2-Dx;fM%Y*d=Wxnpe5^6GDq?(@_TXtu9w|}T|glQp%R1DyfWPLP2P7KfL*@8 z-k1+8XpMRWN)L060rWxXu0w`}k7jwf!FtuMDx?t5P^?0lHi6pIE)G9zSxfmix_)@Oh6@QyZm}Nv3vCoV^(sdH7)M&vBH)w z4|zd2t_0P0!9)|y;{jk@1PgOikyIvzodJ{mge2>F;kdZlU<5T#O{GS5xY0keVAub? z6q}eP=l1D4%JAdo-s!)(+}}Bg|GxgLXvIp!45rdos1}R*Aj20!(q#P!sz+E%8%mY` z_`?1K6mUj-*zhTLW7lEEOF0h1NW`g0K|YtTO%zaPmI0)sSHpRw!MKHC;S=9ZID}@6 z{4aCCeKgaM#KD9qdP5TT5BV=~2nzot4m;^z;&KP%^UGn ziK*fh9DeLUe3bA0yhVEo3O2Q?5wYoRzGlA`+=Ox+%_+3E&QcLpsH|#Z!>b-^j-XYN zZk%JhWWKZcWh?-V+qSt-?=xbAbDfdZI>$YIIKBFtsnz&`^^9}g2FlQm>{?PIot)U=CQQP-vWUlDq zCG=nq6%YRcwr}i>a5E{vMY&UKC+Y!4P~Xusn&hdk{)7cwsntGPvIFl38@Dhf8_VF0 zGSXAw*DC+bAvFORlF~kr?zGZ=1aY9Z^1nr?m;5`;l771QO$b+c7dC($s>UDiW~lvy zJ0Zj`AMIr*ND&J6?{QiHsFVKvWau<%*3#aBn#Sh7fHTKGZ)wUagU{6w%hCaw#d`(c z9?xF4IbW-LpO2?s5=6N@y)oyYU3R#P1EugK2kr8Lk#P@$f`V&=&0V@ca*V|}3T=9Y ze|$SFeRG>_Er^aqt?$T<`wq18Oh)^3>o6CPH(Du_i-`JCT$2?!ST(gu#~x3_uiMzi z*JVF8_TC!IYY0^B{pmUAR$?)Uqm(P3~zT6v6wrqDl!J*}AQ;AB2E?ax)(Fevg2^fsgC~H0eJ(w@s z&w(q*_`v2IapVc<7L4maWQzX`L$ou6h4}^t{&rvI7~7)L_jH#{0(+Bp$My)lMv79oivT;`L7r}l&5rI%FZ?*#-%Yo zXsXe<55#^~)}#gZg+#D{v7;5YlH}(By2+;01e1Y&ezXA2HZ;5=x$kFfpQ(%r?ZyM# z7-9T4{}vjwE*qg-sBI5N^&puGgv%gj^l3lxwY8porR*h7W42a*N_oudnnF=1B)~@> z-5=Dqk&Ty|biQr4XE7Ihoz6@w({O@Tt~ZCXd>|kd0{VNo{IQYu7r;GL%tpbM)7ah( zT!lNp_O?1&UMyxq{L=FQQpZAp(Kl!jDCT4EO0@k=Fe0Q8$aNVPEZeKQA% zKVNV`DJ-6SOqSS(eVpWnNvzogzNIJrAr9OojI#fA9%MK~T3yC3vw z%Y3}nSP3^K^Yq!aAjS9~Hq&q?aKTFsw`y&en5sS&TgR}ZtHr9d9NQpt@Gk~ zsN;Vg&78$e`mJyVgPO`iGkr%tw0VNgi5y*HD223)L#@ks*)u9Ig~{Gk8vSlxM6w!h zu#ln6_h$9HjWfw}Sd>H&T7v+`@V$#XO@C1GW;9K82`vgOW@$6Zd;d1D5x|EyfO5%_Qo zT0b_VU=tp~h~>se;Qz>7c4gmwR#j@$ftv+$)b>@S2E>zL&_8-T!*uqM6uKNUKv9Ra zsxDWu#hZH^f!AqZLFSx42FsTn*iF%$--U*rX}?8%isbc?=bzgDq@oUo zwMhXY6%0niNl$waM4x7#g4fCUc*+vX4{n;dr~=v0f&oI2fkjdZ>1Zv5M-*|~(&O}s zz~_9|F4pO-Yzd`>s8r~gx}S9n1W)6$SJZE6R$TK3o~t1Dj7QU_?@$qc*XIkKUV2~h zFDBQ+6j}#9G0koex>D#x4%Di0>-!4i_A~nPI}n6aY`fJP%|&dXNitsx zPDd42KWoP;)qM#_{4!8jou@~8ZXVp(t;>3#Ge*heV7C1(lw?t*!B9fIy+55qy=DY^ zf4A3M)PRpRYHN|>nYO%ldi0loRV*0|NCRS|NJqO%Y3Rm1-ejEFgFQ~D#btzx>bf~u zMoM}|`Lxpl@q>oj*E!Iv@ACirU{l|oC`3W*!$x%=1@{m6&s8k_&sEeX|KqQ!fng9D z)t+d$(c2Bi5Gdd3dsQG$&TK|`NnK)cRRPvMy(BpdVGbqmwBInl_26|-8lH&Ja z<*!a+y3`F?(RT5ZyE>(!w58F#^qJ45F~kCX-n(w<>l94{vog#e2( z(kz1N&+zd!llh>ri}PJMOSZC?%@YJI@}rroj!IUISN?9+=TqQ3AeQx>lir3@@uS#{ zYAcR(-137(W1k;Xg9kqzOP48lnzda+(2~d`Y@z$R*?Hi`X-TJM6F{KXep^2Tw+YbZ zo1zy9BRZW=G~;Yv@fUawAk;X=u%C4fr&xTY-Usp`dxFUKH61rtJPBMo752e@&@QP` z1Ti2~_Gj|5fO7C8d5WO>S2N!=ipLmK=Wx08Ni#jg^Ml12(`~#pYVVuc%&1+`f0v01 zv#wewT@m;V$7?`1U?+!f0{G2;FIm{%nA3MqylKl3kLu?AyPdGQdAi~YSEVM+#z}-pRQX&ARl+;UFrjnzWUR-n!e?+09n2G`hU{Td zGRk=3uh4A?f2Xigh{5|YM253#t-kc{GVd#O&i)af8=0+RoEEaTZnR;A>NeATnH`8L z!u8e@<`#1$le!p?E(`77^5m^Q_RcItq^Yj~@^yG3L7vL#{=B`3TR}84Gwd$yePW1z z-IHzd0ZNHnXeaDqDo!usV#mZt47RUxC#qB;+%_e~@h59&>95v!r4UH*aV zy8qg+#SHnmwT0#%HRk0+8$zfNGH^mF_14Wr(VW!P&2K4pmiiGTKd&2djzix-d@iGA zg)MvRl{3Xtu@Z2#!vcr!sIW>MNa#I(V?MI<>X|6l>W~#WaV;kiGzHHkCE8xJ5T}MzcnAI~NOf^B-K2{1ToW zgh3@U-AI6|yue+e5|elfpMtM@A86Cl{=|J4e-J)?$#T-4g`4{Ydn~a5f(<;?BE@na zj3Yskkgz~+ju87=jl{oO9{2AiE)AK>j|s|rt+pC^xgMI(rueVDLGJsXS)uZUxW$N5 zm-4v&0P4C2j=B&D(w1QX3v}L&?BZD0&`z}VzeELs&T}tDwq>Xl!C>3%3R{yg#`^BB zC4)ZJGx=W?%PBQRO}&q-nbJJ%)G8iLj$ol`a>4;!WN=>XMJ70&o*EY$;87T$DwIJl zm>3rL04g?c^wnAH$0?C<K)hHS_No-=UtPJg-a(!%%lg^7y{0Uosw{=8 zhDG(Ubu9mx?xwX1&2H3V*mZc=kBRO|OjL{H{zv*wt`9AT>mHN6Pj5crxSsZz!Z1+c zp>ZSzX1zUPIn(){B)BD$I2tyc#0u0!)q!V=3vhc{oWP zsK?je4EifxCzSXUVVaN@o5^XNMg>Aqz`m#E%iV)#&RqJk64$-UN-JtzU#V2Wh{`@b z7@ci2pCozWV({_ym$~yKnav4F)GeCh!y*zZmc*2@6DBvvbu_*fSXyzov57}_eXk zVRfeaDFT&p@_yr|uQY*oKhwFyvEo?<)DI*xg^LMs5MBi3XA73+So|q-G^J5Cxjr1I z-++)`XNq-!K>NB?AE?UlvU~5#OtJ8NZont#KU~mRa49LtY!_KdVPbk8Hx_tze_tv(+3qbsM03sojO9?qqceWho&CY$ ze4Mm(Nn(Bg&*z5^;Pg{Ptz50>+QQCj*V;0I9YvSnUq;;oS`dc`E6VMvVp@?mSzUO% za_ba4!Xp4j|8_*-x|SNn8JL~q8%@3TiPt4Im?W-KR#J2le``2U&WyvTdJpSk3+o; z*AcJ$-aGDv7bIx(CE&}$-Of)Uy^Q4<3&O9nyj1c`Oh`JovqkgPylh@yFE=HzJ0!3z zP!K94gE>=krG32t&^uW1lB}5jIS9A`8cK+RK#5Q!g7ES`Np!OfBaOM7U5WrvZ zQfO2(l;LKW&1}{eBJaq>qT4cUd2paiy0sm6K6JHc>Q?maJYEa_?CD({lh*1zQ|YV= z*V(H-UzbqT;!)~|k|l@8PCt5wr$@!I1Q+aRyt6cYF*n#mR3Pa&>ipaEe~Ob zR9^|}ZoLmZk$Ai>YBxZxqg-g%5K!_A`$1s7H_LUWgzpMUmCro+2K7GoBnd$FWu#8mq*OU}( zzMNQzi0B0wRx+_TU5A-?&+TM>qm|1K)hZ`u`_KxM$QVZ*&nO50ah{UVVEdlk2vu~k zMm=@EPrf(nH(xxN`(4a(Hr5Sxzt_B+veAhnOjFpvrQi!*SMur zx=2)V7Y-Av_F?P47!}vgI@Pd~1+>j6qhWK$icmpaM>n+?epE0?&gWrQ5GmnhWwS~m zKF|4)@Tu;|NG0BJ9MlKT#_%2`zSe-U5i-&pll>_Y^E+d@BbMm)a;HrB^sl73p z8K#c)-KLLIImeco8GIr0V}LQvK!DwuCJZ$agB2+mawHaRMDt%Z^>6g^<1og^xIgo; z=%D{jPA#0*h4sUS-{*jJJa-1({=ALy@Cnzw>xD&Wt}d6o?t;n!t}lN@zS0piKLI|W z_IrytXiV*F2w5`dA=9IS_laG9D1Pks-Bq0zZ*O2$k<>Lr&QjCf zh@%&zL8czd&MI8Ex2PaNDM3~VE(_;kvs52dXtf7-aCTT*dBN=PizRaPxRv|1au;Yx zeSK1STLlxPfhntwc*n~UD_rNttg=aHgrz$Ng^^v3#=OBT=~UkwnbC~W%dgWEp!AR| z7`f6;cx+SMpGvaJ~G!4I=p2|tW>U>%)1{c6D*)j~6U#eAc7H~SJD zB|r{*Il!U!3h0`&FLT=y%jxdGZanO>8uf{yr?X95AhsYe#LBLl1c_cE>R{hJ*e%S4 z;b9J+zS~`__^is*-Tr+;qSQ0>1Wq@w@F!5-0w#XVrnwWV4zAZbAsgf~f2B@P)NpAB zy)Lxk$S-^RP$omNz##8FU``?XQ zQw1!=IGXalT zI7$O7sKhWv->HNFsuAAQ&br41yxIh@Pd)%0ECkk!o3#;W0ji2Z^G{?u2psQr@yNdT zdfWW4BkIm!o3kXNUA^}{POL}%Q}Vgu4%)JO+cxK7Vd9`mU5$M&OnqkDZTXEg6}p$0euJDs&_VB9cr0Ao_U(SnmP-EaUaALnLV)CT_wN{-zmo@SZs! zd~wZ0pYjEj#eWsl^Uy~12JmUSgmQhVqP#rC-%1#|v7J|MQ3Ya*GuI1R!1&u#l;;LzQoDAkgpRdK#0c}rANXb+`_(91 z>9NUV7T$C+{@Ns4sVl$OrhMOeAg^D18#UD;+rPP1tp%=;73LO2ydUXuEwv7q%k z(AAg7uV+T_Z_#7eKAaO_;Z8pr?VLT&rh|7J#%*G6TRG(D| z&nCAx{HpZgG;O9|dIfi=C(%@!S`SL9B$IPZ*&z2V3shMf*H>`1hg+V9jo=g_$Zrw! zgGaO>y7^9nDT{<}5p0d;lTyyjE6oW#|02*GcIx1Qc{M(1bZJvikmSL{{pYqRjRqEIY`-Ih9$<{$Oj~toj(?x zI&O-yCPk$tXBl#$4K?gCvM}Y2nf-(E#@hT1MRjWu<4uVwmnBTJq`1{V!=-Pn*mH?i zIwsF_!2uoz+RLH9ZKkiMFM4`Zw^tjHF@|uCE$Ie5)g+LaE(?C&DiuT_xk>qI+cN{k zLEIi!6nI8~*qC0=1_1`O2hg}1fR<9Ux>nt<0~A`mAvR>Z{+;r1FppM0g2?Hac-Eyux4ume$YtLw#4VTF!ro2UoXI5u%qcyc3OKwy32a!djf~;_2 zGTJ-_IsG%=_OK#Rke(|k!jQGi!uJgT;1mDqyR_}*sUQ*u_De9SHfR-Ar_TkGG!_9l z27xtZ2>bB+)=XMx*jknNv9LS^)u{w|pnLi5+fnt5xP(AXwT93_G$!JaUz@@euZ)D` zE8ON}3Da9>eB4Ay1rAwY=rf80Sr9MQF;gEGCB>rKiI_yTUz;ZH6Ad{Xm_-q&ADs6fyU-vl;0Kdy}KrgqyoO0qN~qXnmCRP zRDLNR*|o!}VUm*W9YF{`?Z&WJNVYKTs!QP2zc=v1WVsJCFf&1U;>zW zGL5MdSr4O0^rf66w{AeYG40F3!DT_=5kf;jU2+i@G^06Vv)WhZr}n=HE~7`am}sN6 zW7R@q$Z?S4K;=r17SQ}%g;j!zTULZsib2vVCi_s^QsI>FQOa%asZFKV?dg^Hb%)I* zwRU>UDit{74kFTdqi9_FiIK(vXBb~hyieGb>)pU)s?&aF7E%;fvJ)~#H6<<@%FI=P zphfs1ZHsaLBT!rGhiP6bnhyVoL<-lzhj)2f8EA%cLPoi1hL92g%i8~vRCt)w3rP|j zgTQZU zkZg-G*wW>94BWJQd$3om{~X_|e%UtGP0(YSr z4OwVN@N%2RWvCxTa~yV16QM*o`S{tfkgJD1cjH5lhBdbU5^F>fK?8d(=FpChAr!9# z-2qfPn|80UhJ-?4Xc&HAl&?c4H|r~iAnMM+Vc!P0FIP`tHRcw2^F^eGHE8RG@EQ-B z=x3mg6xtuvhWxV^SafqJqZ_E$mkUnCODDr9M>Z4w={Sw6$T%2-Q{K810k%OV2V6)Ki z5;KCV58mZmR80RkmE-%w_euz-q7-mFLpT*OK;qb)j{uZe>wlceq4bYa`yV-|RE5*x zOB@L&nlGv?G7{TIPQ2ds@FWeO>}b+LXrDyui8uEiQ6IWt=&9mp4hUF&9#Ye6jX}w+ zpc{mmJ8Zncip?g1`1CPs!_5YB(YPikP0)j%NtuarlY`o0?0I5Rl~cL9Fzb=WXeVZ9 zek%;-2LI$?>@Zj+t2w)EFKkV$d}Zbk5+BqkV{F9^;GZncuITVj;O zRubTHLhau%0^sM`Q+4{klYjGK5Tq;R;76M9T>BOvgBtq_B*XHhLAn`^kWfmU&VsE? zUgv6&n+=`^KCB-`oHH%PIAv-M4;wKa{$Y%Kz|i&Zixoy5+v!_WB2PoZ#Ck4=FIX`` zj{MWTxYT+1Muc4M;xC($!}?zy4$dS`Pe@y`)b?)#_zg6Hrwp8e-)6m}2#5(bQe1u> zpN6NCo{GBWg3jHDT1DhK8z%um(LD=Ok&zl${KB@5_&fAP-~{?WD}wG!_U{zv=7@Jbnyz(A+W{FnUzP$(p#LO|ZTdC7X;SyrR@qpwN4|{4Z z{Z>>uo%2U*9owZ8#Gt~ydEe>puM{I@(^Tp0j)l|EdFkX=mHr+FE#Ui|K|sN@OGmXU z2oLdKtN)(Xnr`osb{x<@_2RtR>=J~jglc{~6b}}&pA8~Fgh}=^PlqY}31(@XjRf$o zDSJtx1r28ir*7=uoWW6*+uz0^@k{kt1QH85g^;tPXtzMGI3J&M9Kn>Z9Wz$~1lC@& zqM5owyu884fB|WZ6)23NTMaB^-mE~-N2zby!!{T;NI0I-szZ4Lca;2?bG5tHyJ=_eAB`(*V--PcV*y78F~c8m7`G!ZL7V# zQpNf1p1d?65?V=4s$Zs2iC<~_rZ;s<2!Se6Cjg*lz%bmq;#iyB2lX%uN?%d3s(IFl z+ot*?7g|>IQ~3YlWT_B%Or02Ai!h1PMX*lWfr&JwDeZCRs+YR!z@u1cweh?0MH6mA zP%W-hp!}-?>s*62DpUi?$jKxf^mH*;jc}MSdWx5(GDP>fO-PTVvNrsSO`wBvo!ArM z%h7pD`sx)CcC52K&HOz|SgOsehjg6K&Bid*q8S5oQgPic#x@=!WZ#sh$!xCG&gI!kAg6!6q#$Fd*O8zfy7-u{gaX8H+KBJd8X`A;l@4_~?!t~`Uu z1wd7_Ibq}i=nxTsh(#dz|HLBrpIA^tkfB4g0^aMmxIhYl@@H2IWUtB3Wg>5ReCqpF zje1F-Brk!4=*j@jt-yZe^mUnGEGh>pV5IZ}<1IGVwMT%H11xI&B~~WaO!wGBRYC|c z7O16;CIu*p_FX(ocbWSy5?fld#RsQFsH`DruG{d#P?~Tx`kj!p$?=`QrB5-;EOLiv z=7voI@jo`93VoqCH}-J}-Zrm)nzp!qpdh4D&4{oJm|~7=n98;A_~!fJWmx?g^YM5! zu99Bh8^)0rQ1X&c3iX7Q3A?f$yDDs*sul#lGQ9a&$mQX0Bps7`CGBq;1`LVq_1DrA z6NQ=F=U{(Hm{aoWqo}1`<9=+x!;Uu1EL~j&(vx2iBnyUNTStE!^3&f5rW9msZq?FCjN$5|J+bI%u3t^oLZq|Vo9CP8 zd(vaaXc}yyeSa%2IIeaMkz#e=^|Q%w)ICgIl4PBFg8(qt5K_n?$U+# zxn>vch(;@w3ggg6pyT{3Js6}ZR%2F^fx*3nV`y=SnlrqO%CVY*{k)v}4T|kaHZldS zeXgAYGXijz(TdjNC4p8ff#S-w6+ZNPBbojyZf;d57yJ5N5GK-?bUj+HgJAnTse_Se zpzk7~w;I<>AR2M12hmn7q0X)(QZQ!=(`*~uc~bXH>55@nT4m`;wJoSP@Gt0}LE;A^5{=U=-9KOd@b}=vk@8rdi(K z-a_6S!+*HW`@G_`C>rhvx>+=*p>B&k4D`aLEKvBN$Tq*m;P+BG2paBWMXc%3(1eaIp+tHDbwn5O(hO z8G>iqRA9-p#l^zWkCJa&*15>bsk(QYauj6R=bjl zE4cEq4MV7#zlGKOF8Yg5!~HY_DfS|*GqmzrHQ>!tlSd824&mmOpMOz4ZxVD|oOVeF z8HQH+NsFK6w-N?6A}3$;a&8n3eU0$jaCcfO{19)QT0OgvG8+G_A!b#qcPq;uf8b!* z;7?VFIVi(^qJy)*vMD`7)AV7;vnMi-E|b94xi78!woz{wSp~r(%`xl3CX=#wB!3Cn zO)WiyRqr4BwEXZek#L}wK_y{LODC1H6_Z3t?k zfK~dXk?|EAjs)1R@?mITI6f$FHwzkL$skdKL586nW;BJp3>oqvTOq74952&hQ-Qo~ z7@k+Ri&Mr3` zEYD5$6Lzy7j(Q@Ik%j|T`iZ9Ch#9?R;@fyfx3eh1o56(65=v#_e^53^*8;)5AgGEeReDtE&`jG=8#s8IDyux{*h5%q@wjyWKpDuOa;AYCZl zo3;d@urD1dfgy@oXP})2del2-5KiO0#P7gZKv`S7)Lg#2(Wx>ph}~+KgL>Kz*Zw-{ zkkh3SD-L7#Gx;+y%<=K}aVC(tc9_f`yP=<`JzW6EAr)FM~DIkp%mWa>K* zwR?MBL8LMFWUHhWnnFx9Oro>Uy%h5bpI7T@IH8IOS~! z-@Y@*JPWsg{k7f@*9HrMo@`4YrR#rIBjh(QswJxgb%j{zU(Zmk@vkc z8_>wV@>GAu@JdW-9ww4|tGz))Ge!{g_HgM_eeTbn)y0>zlU#Oy=|O)65&k|w4*d90 zUFVH$!B;RKYy$PMf3L=n=l$*0+_Aj7#snr>6nud%+R5HiOs_e{j+|RO-ST$ZXLNs+ zuCvk5WjqM-b+O#Pfo1&w3JLkW@Nnlgb9UgaDvCk#cOXEe{T1cm$3b4=Ahs{bS6`B| zqxrDX?4^s+)r&Oi?k%iG(7@Y7+iU0A>?V!Wsw3CW3c01esi8h0o^I$QQ%_w#b~6t$ z{JKIoqy;vk_MCQ+FEE5XJfuCOL_!+?=*43cNN*+3Y6t3x8%5<1?E(hAktw^Rr)03C z1|#$xgHrSv37r^W{EYzI9O$t)JX1M%HT?+Ns(sUa($ND-i*Zc^E$;gd2Lubj$s^{B z)l0M@K0;^v1nyxI7}U=-c#S`ZKQVqJx?KEaetm7v5s;Rt>?HH2elL1!a$KOoByB;* z!?+_V-?od>t3v>a+i@hNrDkmKnr8$&>YlAfnR0mh_*le$JRC;+<9dX&p^EPjGynaH z`+fBtsHO-Fg98A7(M6&0!6oHD^za*6Amj_xd_xPpj{pDvT?Aehfz}0vfjLB>#i29k zfr*$%IN(T8Xf~i6xbw+30-6tcn-dx=$^}gXK9+*U0MBznW1s;3-={F>8#g#e5`w*PLnHm~0USwaLa2a` z{|hThLURK>!GV&{YQS&cX~+N-!1q5$L<*W4%9HK8 diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index 755c5f5be..f4ed2aafd 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -1850,7 +1850,7 @@ assert( ) && browserStyle.includes('.browser-status[data-visible="true"][data-copyable="true"]') && browserStyle.includes(".browser-status-copy") && - browser.includes("browser.js?v=browser-20260907a") && + browser.includes("browser.js?v=browser-20260907b") && !browser.includes("browser.js?v=browser-20260731a") && !browser.includes("browser.js?v=browser-20260730a") && !browser.includes("browser.js?v=browser-20260728a") && @@ -1896,8 +1896,8 @@ assert( ); assert( - browserJs.includes("browser-status.js?v=browser-20260907a") && - browserRemoteDisplay.includes("browser-status.js?v=browser-20260907a") && + browserJs.includes("browser-status.js?v=browser-20260907b") && + browserRemoteDisplay.includes("browser-status.js?v=browser-20260907b") && !browserJs.includes("browser-status.js?v=browser-20260711c") && !browserRemoteDisplay.includes("browser-status.js?v=browser-20260711c") && !browserJs.includes("browser-status.js?v=browser-20260626e") && @@ -1912,7 +1912,7 @@ assert( ); assert( - browserJs.includes("browser-remote-display.js?v=browser-20260907a") && + browserJs.includes("browser-remote-display.js?v=browser-20260907b") && !browserJs.includes("browser-remote-display.js?v=browser-20260730b") && !browserJs.includes("browser-remote-display.js?v=browser-20260728a") && !browserJs.includes("browser-remote-display.js?v=browser-20260727a") && diff --git a/scripts/browser-window-close-handshake.test.mjs b/scripts/browser-window-close-handshake.test.mjs index 8e5cb428a..e930ac732 100644 --- a/scripts/browser-window-close-handshake.test.mjs +++ b/scripts/browser-window-close-handshake.test.mjs @@ -4,6 +4,7 @@ import assert from "node:assert/strict"; import fs from "node:fs"; import test from "node:test"; import vm from "node:vm"; +import { requireBrowserViewer } from "../capsules/browser/browser/browser-runtime-api.js"; const browserSource = fs.readFileSync( new URL("../capsules/browser/browser/browser.js", import.meta.url), @@ -1127,6 +1128,49 @@ test("pending Home close blocks creation of a replacement Runtime owner", async ); }); +test("viewer denial preserves an existing page before close or open dispatch", async () => { + const page = { page_id: "page-kept" }; + const loading = []; + const context = vm.createContext({ + runtimeOpenInFlight: 0, + homeWindowCloseInFlight: false, + pendingHomeWindowCloseDelivery: null, + homeWindowTerminalCloseConfirmed: false, + currentPage: page, + currentPageGeneration: 7, + currentRemoteExitId: "exit-kept", + currentBrowserEngineId: "engine-kept", + selectedRemoteExitId: "exit-next", + selectedBrowserEngineId: "engine-next", + runtimeOwnershipTerminallyAbsent: false, + normalizeRuntimeOpenUrl: (value) => value, + visibleAddressForUrl: (value) => value, + browserEngineLabel: (value) => value, + browserExitLabel: (value) => value, + setLoading: (value) => loading.push(value), + showStatus() {}, + launchContractForOpen: async () => ({ displayMode: "webrtc_remote_display" }), + requireViewer: (mode) => requireBrowserViewer(mode, {}), + closeRuntimePage: () => assert.fail("Viewer rejection must preserve the old page"), + fetchJson: () => assert.fail("Viewer rejection must precede Engine dispatch"), + isAuthoritySessionError: () => false, + friendlyOpenError: (error) => error.message, + }); + vm.runInContext( + `${extractFunction(browserSource, "proveRuntimeOwnershipAbsentBeforeDispatch")}\n${extractFunction(browserSource, "requestRuntimeOpen")}\nthis.requestRuntimeOpen = requestRuntimeOpen;`, + context, + ); + await assert.rejects( + context.requestRuntimeOpen("https://example.com/"), + (error) => error.payload?.code === "viewer_unavailable", + ); + assert.equal(context.currentPage, page); + assert.equal(context.currentPageGeneration, 7); + assert.equal(context.runtimeOwnershipTerminallyAbsent, false); + assert.equal(context.runtimeOpenInFlight, 0); + assert.deepEqual(loading, [true, false]); +}); + test("ownership-changing open returns pending without closing or claiming terminal", async () => { const harness = createHarness({ openInFlight: 1, diff --git a/state.md b/state.md index 74ad6920c..2d2a8720f 100644 --- a/state.md +++ b/state.md @@ -78,6 +78,9 @@ validates declared Engine capabilities before launch, and preserves explicit operator selection. Installed acceptance of this slice remains open. Browser settings retain a missing selected Engine or Exit as unavailable until the operator changes it. Automatic Engine selection remains with Runtime. +The web Runtime adapter checks WebRTC viewer eligibility before page replacement +and Engine dispatch. Tests prove that viewer denial preserves an existing page +and acquires no new page or stream. Installed viewer proof remains open. Mac and Linux host adapters now query their virtualization APIs for eligibility. The Mac unit check ran locally; the Linux-specific KVM query still needs a Linux test run. From 717efd3ea20f336c31e9cac5741a51176aa571d2 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 18:16:09 -0500 Subject: [PATCH 072/535] test(browser): use the selected viewer for installed Home proof --- scripts/home-passkey-virtual-auth-smoke.mjs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/scripts/home-passkey-virtual-auth-smoke.mjs b/scripts/home-passkey-virtual-auth-smoke.mjs index 811225d2e..cdba22117 100644 --- a/scripts/home-passkey-virtual-auth-smoke.mjs +++ b/scripts/home-passkey-virtual-auth-smoke.mjs @@ -2603,13 +2603,13 @@ async function signBackIn(page) { state.shell === "resolving" && state.gui === "dormant" && state.activeShellRootHidden && - !state.activeShellFrameSrc && + ["", "about:blank"].includes(state.activeShellFrameSrc) && !state.hostGuiDomPresent, "A Home shell remained mounted behind the passkey prompt", state, ); const clickTokenPromise = captureNextPasskeyToken(page).catch(() => null); - await page.locator("#home-unlock-primary").click(); + await page.getByRole("button", { name: /^Use passkey/ }).click(); await waitForSignedHome(page); const token = await settleTokenWithin(clickTokenPromise, 1_000) || await settleTokenWithin(tokenPromise, 1_000); @@ -3586,6 +3586,7 @@ async function main() { const context = await chromium.launchPersistentContext(PROFILE_DIR, { headless: HEADLESS, + executablePath: process.env.ELASTOS_BROWSER_EXECUTABLE || undefined, ignoreHTTPSErrors: true, viewport: { width: 1280, height: 900 }, }); @@ -3646,6 +3647,11 @@ async function main() { const report = { schema: "elastos.home.passkey-virtual-auth-smoke/v1", ok: true, + viewer: { + version: context.browser()?.version() || null, + executable_override: Boolean(process.env.ELASTOS_BROWSER_EXECUTABLE), + headed: !HEADLESS, + }, home_url: HOME_URL, profile_dir: PROFILE_DIR, created_mode: created.mode, From c2ab2d84b126cb7fc907bd304d1fc1d57b727d5a Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 18:22:04 -0500 Subject: [PATCH 073/535] fix(home): expose the passkey control to accessible operators --- capsules/home/browser/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/capsules/home/browser/index.html b/capsules/home/browser/index.html index e4f59d6f5..d5b5babc7 100644 --- a/capsules/home/browser/index.html +++ b/capsules/home/browser/index.html @@ -61,7 +61,7 @@

Home didn't open

- + diff --git a/docs/BROWSER_PROTOCOL.md b/docs/BROWSER_PROTOCOL.md index 25b373a5b..41fc77d2e 100644 --- a/docs/BROWSER_PROTOCOL.md +++ b/docs/BROWSER_PROTOCOL.md @@ -20,7 +20,7 @@ meanings and must be checked at the boundary they identify. | --- | --- | --- | | Capsule package | `elastos.capsule/v1`, `elastos.runtime-projection/v1`; interface `elastos.browser.page` version `0.5.0` | Runtime admits the package and its declared affordances. These identifiers describe the current package interface, not a release readiness claim. | | Runtime web projection | `elastos.browser.runtime/v1` summary and operation-specific result schemas | A verified Browser launch grant binds each request to principal, session and launched Browser instance. The HTTP routes are a Runtime adapter for those resources. | -| Runtime to Engine | Provider `browser-engine-adapter`, protocol `2.0` | Runtime validates the provider inventory before profile preparation, lifecycle reservation, streams or launch. Engine results and cleanup records retain exact provider and protocol bindings. | +| Runtime to Engine | Provider `browser-engine-adapter`, protocol `2.1` | Runtime validates the provider inventory before profile preparation, lifecycle reservation, streams or launch. Engine results and cleanup records retain exact provider and protocol bindings. | The Engine version policy is exact compatibility with the declared protocol. Missing, older, newer, or malformed versions produce @@ -92,9 +92,29 @@ public typed inventory. `configured` means a provider has configuration. Installed artifact readiness, available capacity, permission to use a service, and installed-product -certification are separate facts. B02 supplies authoritative readiness; B08 -supplies normal remote Engine service admission. Neither is inferred from a -configuration label. +certification are separate facts. Before preparing a profile or reserving launch +effects, Runtime requests `readiness` for a compatible Engine. The result uses +`elastos.browser.engine-readiness/v1`, binds the selected adapter ID, and carries +`ready` or `unavailable` with a typed reason. An explicit selection retains its +identity; automatic selection tries the compatible default, then other compatible +entries in the admitted inventory. A failed adapter prewarm retains the inventory +so Runtime can evaluate the other entries. + +The VM control service exposes the same readiness report through its private +`GET /readiness` operation. Its host check verifies the rootfs receipt, kernel and +initrd identities, actual host architecture, and usable KVM or entitled VZ support. +It caches a successful result only while file identity, size and nanosecond change +timestamps match. Changed files require a fresh check. Host inspection is bounded +at eight seconds, and Runtime bounds candidate readiness requests at twelve +seconds in total. Installation and repair still need atomic admitted artifact +sets; service discovery and installed qualification retain their own gates. + +Protocol 2.1 requires this readiness operation. Runtime rejects older Engine +protocols before launch effects. Close active 2.0 sessions before installing the +new Runtime/Engine pair; durable cleanup bindings keep their exact protocol +version. Normal remote Runtime service admission remains B08 work. Legacy +operator tunnel launchers report `readiness_unsupported` until they implement a +verified remote-host result. An explicit Engine selection either satisfies the requested display and isolation requirement or returns a typed rejection. Automatic selection first @@ -123,7 +143,10 @@ capability requirements behind the same Runtime page contract. Compatibility errors are stable typed codes: `incompatible_engine_protocol`, `invalid_engine_status`, `engine_unavailable`, `engine_not_found`, `incompatible_engine_capabilities`, and -`no_compatible_engine`. Open failures carry `stage: engine_compatibility` and +`no_compatible_engine`. Readiness rejection uses `engine_not_ready`, a typed +`reason`, and `stage: engine_readiness`; it retains the same terminal pre-effect +outcome. Browser UI maps those reasons to preparation, repair, update, connection, +or host-selection instructions. Compatibility failures carry `stage: engine_compatibility` and an `elastos.browser.open-outcome/v1` stating that page, VM and stream effects were not acquired. Browser UI explains version and capability remedies; agents can use the same code and outcome. A cleanup-pending outcome retains priority diff --git a/elastos/crates/elastos-common/src/browser_protocol.rs b/elastos/crates/elastos-common/src/browser_protocol.rs index 7ae6c4d45..9b3a31247 100644 --- a/elastos/crates/elastos-common/src/browser_protocol.rs +++ b/elastos/crates/elastos-common/src/browser_protocol.rs @@ -8,7 +8,7 @@ use serde_json::Value; use std::collections::BTreeSet; pub const BROWSER_ENGINE_PROVIDER_ID: &str = "browser-engine-adapter"; -pub const BROWSER_ENGINE_PROTOCOL_VERSION: &str = "2.0"; +pub const BROWSER_ENGINE_PROTOCOL_VERSION: &str = "2.1"; pub const BROWSER_ENGINE_CLEANUP_BINDING_SCHEMA: &str = "elastos.browser.engine-cleanup-binding/v2"; pub const BROWSER_ENGINE_CLEANUP_RESULT_SCHEMA: &str = "elastos.browser.engine-cleanup-result/v2"; pub const MAX_BROWSER_ENGINE_ADAPTERS: usize = 64; @@ -202,6 +202,34 @@ pub struct BrowserEngineInventory { pub wallet_injection: bool, } +pub const BROWSER_ENGINE_READINESS_SCHEMA: &str = "elastos.browser.engine-readiness/v1"; + +/// A host adapter's current preparation result. Capacity and permission are +/// checked separately. Private host paths stay behind the adapter boundary. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +#[serde(tag = "state", rename_all = "snake_case", deny_unknown_fields)] +pub enum BrowserEngineReadiness { + Ready {}, + Unavailable { + reason: BrowserEngineReadinessReason, + }, +} + +#[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq, thiserror::Error)] +#[serde(rename_all = "snake_case")] +pub enum BrowserEngineReadinessReason { + #[error("the Engine artifacts need preparation")] + PreparationRequired, + #[error("the Engine artifact receipt does not match the installed files")] + ArtifactInvalid, + #[error("this host cannot run the selected Engine")] + HostUnsupported, + #[error("the Engine control service is unavailable")] + ControlUnavailable, + #[error("the Engine needs an update to report readiness")] + ReadinessUnsupported, +} + #[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq, thiserror::Error)] #[serde(tag = "code", rename_all = "snake_case")] pub enum BrowserCompatibilityError { @@ -213,6 +241,10 @@ pub enum BrowserCompatibilityError { EngineUnavailable, #[error("The selected Browser Engine is unavailable. Choose an available approved Engine.")] EngineNotFound, + #[error("Browser Engine is not ready: {reason}. Prepare this Engine or choose another approved Engine.")] + EngineNotReady { + reason: BrowserEngineReadinessReason, + }, #[error( "The selected Browser Engine does not support this display and isolation requirement." )] diff --git a/elastos/crates/elastos-common/src/browser_protocol/tests.rs b/elastos/crates/elastos-common/src/browser_protocol/tests.rs index 39e981298..6c94b8c35 100644 --- a/elastos/crates/elastos-common/src/browser_protocol/tests.rs +++ b/elastos/crates/elastos-common/src/browser_protocol/tests.rs @@ -119,7 +119,8 @@ fn incompatible_versions_are_classified_before_version_specific_fields() { for version in [ Value::Null, json!("1.0"), - json!("2.1"), + json!("2.0"), + json!("2.2"), json!("3.0"), json!(2), ] { @@ -244,3 +245,19 @@ fn operator_open_intent_cannot_supply_runtime_or_profile_authority() { ); } } + +#[test] +fn readiness_accepts_only_the_declared_state_and_reason_fields() { + assert_eq!( + serde_json::from_value::(json!({"state":"ready"})).unwrap(), + BrowserEngineReadiness::Ready {} + ); + for value in [ + json!({"state":"ready", "unexpected_authority":true}), + json!({"state":"unavailable", "reason":"future_reason"}), + json!({"state":"unavailable"}), + json!({"state":"future_state"}), + ] { + assert!(serde_json::from_value::(value).is_err()); + } +} diff --git a/elastos/crates/elastos-server/src/api/gateway_browser.rs b/elastos/crates/elastos-server/src/api/gateway_browser.rs index ec8fab3f0..4e98b8f50 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser.rs @@ -585,7 +585,14 @@ async fn execute_browser_open( let mut body = serde_json::to_value(&error).expect("Browser compatibility error serializes"); body["message"] = serde_json::json!(error.to_string()); - body["stage"] = serde_json::json!("engine_compatibility"); + body["stage"] = serde_json::json!(if matches!( + error, + BrowserCompatibilityError::EngineNotReady { .. } + ) { + "engine_readiness" + } else { + "engine_compatibility" + }); return Err(BrowserOpenFailure::json(StatusCode::BAD_REQUEST, body)); } }; diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs b/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs index de60ed5fc..722a2bf07 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs @@ -1,6 +1,9 @@ //! Browser engine and Net/Exit summary helpers. use super::*; +use elastos_common::browser_protocol::{ + BrowserEngineReadiness, BrowserEngineReadinessReason, BROWSER_ENGINE_READINESS_SCHEMA, +}; pub(in crate::api::gateway) async fn resolve_browser_engine_adapter( registry: &ProviderRegistry, @@ -25,9 +28,54 @@ pub(in crate::api::gateway) async fn resolve_browser_engine_adapter( let data = provider_response_data(&response).ok_or(BrowserCompatibilityError::InvalidEngineStatus)?; let inventory = BrowserEngineInventory::from_status(&data)?; - inventory - .select(requested_adapter, display_mode, guarantee_level) - .map(|adapter| adapter.id.clone()) + // Validate explicit choice and capability errors before probing a host. + inventory.select(requested_adapter, display_mode, guarantee_level)?; + let mut candidates = inventory + .adapters + .iter() + .filter(|adapter| adapter.supports(display_mode, guarantee_level)) + .filter(|adapter| requested_adapter.is_none_or(|id| adapter.id == id)) + .collect::>(); + candidates.sort_by_key(|adapter| !adapter.default); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(12); + let mut unavailable_reason = BrowserEngineReadinessReason::ReadinessUnsupported; + for adapter in candidates { + let request = serde_json::json!({ + "op": "readiness", "principal_id": principal_id, "adapter_id": adapter.id, + }); + let result = + match tokio::time::timeout_at(deadline, registry.send_raw("browser-engine", &request)) + .await + { + Ok(result) => result, + Err(_) => { + return Err(BrowserCompatibilityError::EngineNotReady { + reason: BrowserEngineReadinessReason::ControlUnavailable, + }) + } + }; + let readiness = result.ok().and_then(|response| { + if response.get("status").and_then(serde_json::Value::as_str) != Some("ok") { + return None; + } + let data = provider_response_data(&response)?; + if data.get("schema").and_then(serde_json::Value::as_str) + != Some(BROWSER_ENGINE_READINESS_SCHEMA) + || data.get("adapter_id").and_then(serde_json::Value::as_str) != Some(&adapter.id) + { + return None; + } + serde_json::from_value::(data["readiness"].clone()).ok() + }); + match readiness { + Some(BrowserEngineReadiness::Ready {}) => return Ok(adapter.id.clone()), + Some(BrowserEngineReadiness::Unavailable { reason }) => unavailable_reason = reason, + None => unavailable_reason = BrowserEngineReadinessReason::ReadinessUnsupported, + } + } + Err(BrowserCompatibilityError::EngineNotReady { + reason: unavailable_reason, + }) } pub(in crate::api::gateway) async fn browser_engine_summary( diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs b/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs index 591b9467d..48f84d69a 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs @@ -460,6 +460,7 @@ async fn test_browser_open_rejects_invalid_adapter_inventory_before_reservation( } struct BrowserCompatibilityFixture { + readiness_by_adapter: HashMap, inventory: serde_json::Value, calls: Arc>>, } @@ -487,6 +488,14 @@ impl Provider for BrowserCompatibilityFixture { if request["op"] == "status" && request.get("lifecycle_generation").is_none() { return Ok(json!({"status": "ok", "data": self.inventory})); } + if request["op"] == "readiness" { + if let Some(response) = request["adapter_id"] + .as_str() + .and_then(|id| self.readiness_by_adapter.get(id)) + { + return Ok(response.clone()); + } + } MockBrowserEngineProvider.send_raw(request).await } } @@ -525,7 +534,7 @@ async fn test_browser_compatibility_denial_precedes_every_launch_effect() { ), ( "/protocol_version", - json!("2.0"), + json!(BROWSER_ENGINE_PROTOCOL_VERSION), Some("missing-engine"), "engine_not_found", ), @@ -544,6 +553,7 @@ async fn test_browser_compatibility_denial_precedes_every_launch_effect() { .register_sub_provider( "browser-engine", Arc::new(BrowserCompatibilityFixture { + readiness_by_adapter: HashMap::new(), inventory, calls: calls.clone(), }), @@ -629,6 +639,103 @@ async fn test_browser_compatibility_auto_selects_a_compatible_engine() { .register_sub_provider( "browser-engine", Arc::new(BrowserCompatibilityFixture { + readiness_by_adapter: HashMap::new(), + inventory, + calls: calls.clone(), + }), + ) + .await + .unwrap(); + let app = gateway_router(state); + let response = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri("/api/apps/browser/open") + .header("x-elastos-home-token", token.clone()) + .header(CONTENT_TYPE, "application/json") + .body(Body::from( + json!({"url":"https://ela.city/", + "display_mode":"webrtc_remote_display", "guarantee_level":"operator_rbi", + "viewport":{"width":900,"height":520}}) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + let status = response.status(); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let opened: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!(status, StatusCode::OK, "{opened}"); + assert_eq!(opened["engine_page"]["adapter"], "mock-jetson-engine"); + let launches: Vec<_> = calls + .lock() + .await + .iter() + .filter(|call| call["op"] == "launch") + .cloned() + .collect(); + assert_eq!(launches.len(), 1); + assert_eq!(launches[0]["adapter_id"], "mock-jetson-engine"); + assert_eq!(launches[0]["guarantee_level"], "operator_rbi"); + let page_id = opened["engine_page"]["page_id"].as_str().unwrap(); + let response = app + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri(format!("/api/apps/browser/pages/{page_id}/close")) + .header("x-elastos-home-token", token) + .header(CONTENT_TYPE, "application/json") + .body(browser_close_body(browser_cleanup_id(&opened))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert_eq!(browser_page_session_count(dir.path()).await, 0); + assert_eq!(browser_engine_cleanup_obligation_count(dir.path()).await, 0); + assert_eq!(browser_stream_cleanup_obligation_count(dir.path()).await, 0); +} + +#[tokio::test] +async fn test_browser_readiness_auto_selects_an_approved_ready_remote_engine() { + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let state = browser_engine_attached_test_state(dir.path()).await; + state + .provider_registry + .as_ref() + .unwrap() + .unregister_sub_provider("browser-engine") + .await + .unwrap(); + let mut inventory = MockBrowserEngineProvider + .send_raw(&json!({"op": "status", "principal_id": "person:compatibility-fixture"})) + .await + .unwrap()["data"] + .clone(); + inventory["adapters"][1]["backing_substrate"] = json!("remote_operator_vm"); + let readiness_by_adapter = HashMap::from([( + "mock-browser-engine".to_string(), + json!({ + "status":"ok", "data": {"schema":"elastos.browser.engine-readiness/v1", + "adapter_id":"mock-browser-engine", "readiness":{"state":"unavailable","reason":"artifact_invalid"}} + }), + )]); + let calls = Arc::new(TokioMutex::new(Vec::new())); + state + .provider_registry + .as_ref() + .unwrap() + .register_sub_provider( + "browser-engine", + Arc::new(BrowserCompatibilityFixture { + readiness_by_adapter, inventory, calls: calls.clone(), }), @@ -703,6 +810,7 @@ async fn test_browser_compatibility_summary_rejects_future_protocol_inventory() .register_sub_provider( "browser-engine", Arc::new(BrowserCompatibilityFixture { + readiness_by_adapter: HashMap::new(), inventory: json!({"provider": "browser-engine-adapter", "protocol_version": "3.0", "direct_network": false, "wallet_injection": false, "adapters": "future format"}), calls: Arc::new(TokioMutex::new(Vec::new())), @@ -7331,3 +7439,94 @@ async fn test_browser_account_access_status_returns_only_exact_origin_bound_gran .unwrap(); assert_eq!(missing_context.status(), StatusCode::BAD_REQUEST); } + +#[tokio::test] +async fn test_browser_readiness_denial_precedes_profile_and_launch_effects() { + for (readiness_data, reason) in [ + ( + json!({"schema":"elastos.browser.engine-readiness/v1","adapter_id":"mock-browser-engine", + "readiness":{"state":"unavailable","reason":"artifact_invalid"}}), + "artifact_invalid", + ), + ( + json!({"schema":"elastos.browser.engine-readiness/v1","adapter_id":"foreign-engine", + "readiness":{"state":"ready"}}), + "readiness_unsupported", + ), + ( + json!({"schema":"elastos.browser.engine-readiness/v9","adapter_id":"mock-browser-engine", + "readiness":{"state":"ready"}}), + "readiness_unsupported", + ), + ( + json!({"schema":"elastos.browser.engine-readiness/v1","adapter_id":"mock-browser-engine", + "readiness":{"state":"ready", "unexpected_authority":true}}), + "readiness_unsupported", + ), + ] { + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let state = net_exit_test_state(dir.path()).await; + let inventory = MockBrowserEngineProvider + .send_raw(&json!({"op":"status","principal_id":"person:readiness"})) + .await + .unwrap()["data"] + .clone(); + let calls = Arc::new(TokioMutex::new(Vec::new())); + state + .provider_registry + .as_ref() + .unwrap() + .register_sub_provider( + "browser-engine", + Arc::new(BrowserCompatibilityFixture { + inventory, + calls: calls.clone(), + readiness_by_adapter: HashMap::from([( + "mock-browser-engine".to_string(), + json!({"status":"ok","data":readiness_data}), + )]), + }), + ) + .await + .unwrap(); + let response = gateway_router(state) + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri("/api/apps/browser/open") + .header("x-elastos-home-token", token) + .header(CONTENT_TYPE, "application/json") + .body(Body::from( + json!({ + "url":"https://readiness.invalid/", "adapter_id":"mock-browser-engine", + "display_mode":"webrtc_remote_display", "guarantee_level":"operator_rbi" + }) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let result: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!(result["code"], "engine_not_ready"); + assert_eq!(result["reason"], reason); + assert_eq!(result["stage"], "engine_readiness"); + assert_eq!(result["outcome"]["state"], "terminal_pre_effect_failure"); + let calls = calls.lock().await; + assert_eq!(calls.len(), 2); + assert_eq!(calls[0]["op"], "status"); + assert_eq!(calls[1]["op"], "readiness"); + assert!(calls.iter().all(|call| call.get("profile").is_none())); + assert_eq!(browser_page_session_count(dir.path()).await, 0); + assert_eq!(browser_engine_cleanup_obligation_count(dir.path()).await, 0); + assert_eq!(browser_stream_cleanup_obligation_count(dir.path()).await, 0); + assert!(!dir.path().join("browser-lifecycle").exists()); + assert!(!dir.path().join("browser-streams").exists()); + } +} diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_transport.rs b/elastos/crates/elastos-server/src/api/gateway_browser_transport.rs index 7337bcd26..438130bcf 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_transport.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_transport.rs @@ -1210,7 +1210,7 @@ mod tests { serde_json::json!({ "schema": "elastos.browser.engine.page/v1", "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "page_id": authority["page_id"], "adapter": "browser-vm-product", "engine": "chromium_microvm", diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/browser_profile.rs b/elastos/crates/elastos-server/src/api/gateway_tests/browser_profile.rs index 79e6765ef..25987fc05 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/browser_profile.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/browser_profile.rs @@ -156,7 +156,7 @@ async fn browser_profile_reset_refuses_live_principal_session() { BrowserLaunchEffect { page_id: "profile-reset-live-page".to_string(), engine_provider: "browser-engine-adapter".to_string(), - engine_protocol_version: "2.0".to_string(), + engine_protocol_version: "2.1".to_string(), engine_adapter: "mock-adapter".to_string(), engine: "mock-engine".to_string(), provider_cleanup: serde_json::json!({ diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs b/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs index a1454bea3..45bb4294b 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs @@ -3330,6 +3330,12 @@ impl Provider for MockBrowserEngineProvider { .get("principal_id") .and_then(|value| value.as_str()) .is_some()); + if request["op"] == "readiness" { + return Ok(json!({"status": "ok", "data": { + "schema": "elastos.browser.engine-readiness/v1", + "adapter_id": request["adapter_id"], "readiness": {"state": "ready"} + }})); + } if request.get("op").and_then(|value| value.as_str()) == Some("status") && request.get("lifecycle_generation").is_some() { @@ -3530,7 +3536,7 @@ impl Provider for MockBrowserEngineProvider { "data": { "schema": "elastos.browser.engine.page/v1", "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "page_id": page_id, "adapter": adapter, "engine": "selkies_gstreamer", @@ -3731,7 +3737,7 @@ impl Provider for MockBrowserEngineProvider { "status": "ok", "data": { "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "status": "configured", "adapter_count": 2, "adapters": [ @@ -3913,7 +3919,7 @@ impl Provider for MockReconciliatingBrowserEngineProvider { "data": { "schema": "elastos.browser.engine.page/v1", "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "page_id": "unsafe page id", } })), @@ -4133,7 +4139,7 @@ impl Provider for MockReconciliatingBrowserEngineProvider { "stream_id": stream_id, "effect": { "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "page_id": page_id, "adapter": "mock-browser-engine", "engine": "selkies_gstreamer", @@ -4335,6 +4341,7 @@ impl Provider for MockRejectingBrowserEngineProvider { request: &serde_json::Value, ) -> Result { match request.get("op").and_then(|value| value.as_str()) { + Some("readiness") => MockBrowserEngineProvider.send_raw(request).await, Some("launch") => Ok(json!({ "status": "error", "code": "display_session_unavailable", @@ -4357,7 +4364,7 @@ impl Provider for MockRejectingBrowserEngineProvider { "status": "ok", "data": { "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "status": "configured", "adapter_count": 1, "adapters": [{ @@ -4418,7 +4425,7 @@ impl Provider for MockMalformedBrowserEngineProvider { "status": "ok", "data": { "provider": "browser-engine-adapter", - "protocol_version": "2.0", + "protocol_version": "2.1", "status": "configured", "adapter_count": 1, "required_byte_transport": "adapter_ipc", diff --git a/elastos/crates/elastos-server/src/server_infra.rs b/elastos/crates/elastos-server/src/server_infra.rs index 0dc8340f3..39c58b5a3 100644 --- a/elastos/crates/elastos-server/src/server_infra.rs +++ b/elastos/crates/elastos-server/src/server_infra.rs @@ -2258,7 +2258,8 @@ mod tests { async fn browser_engine_startup_reaps_old_or_mixed_version_before_launch() { for status in [ provider_status(BROWSER_ENGINE_PROVIDER_ID, "1.0"), - provider_status(BROWSER_ENGINE_PROVIDER_ID, "2.1"), + provider_status(BROWSER_ENGINE_PROVIDER_ID, "2.0"), + provider_status(BROWSER_ENGINE_PROVIDER_ID, "2.2"), provider_status("other-provider", BROWSER_ENGINE_PROTOCOL_VERSION), ] { let registry = provider::ProviderRegistry::new(); diff --git a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs index 73f4b4da3..4f85d17bf 100644 --- a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs +++ b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs @@ -79,6 +79,22 @@ const VZ_AUTHORITY_BOOT_ARG_PREFIXES: [&str; 4] = [ #[tokio::main] async fn main() { + if std::env::args().nth(1).as_deref() == Some("--host-capabilities") { + let supported = elastos_vz::is_supported(); + let entitled = std::env::current_exe() + .ok() + .is_some_and(|path| verify_virtualization_entitlement(&path).is_ok()); + println!( + "{}", + json!({ + "schema": "elastos.browser.vm-host-capabilities/v1", + "available": supported && entitled, + "reason": if !supported { Some("host_unsupported") } + else if !entitled { Some("preparation_required") } else { None }, + }) + ); + return; + } init_tracing(); if let Err(error) = run().await { eprintln!("{error}"); diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index f4ed2aafd..abffc3e80 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -522,7 +522,7 @@ assert( assert( browserEngineAdapter.includes("elastos.browser.engine.page/v1") && browserEngineAdapter.includes( - 'const BROWSER_ENGINE_PROTOCOL_VERSION: &str = "2.0"', + 'const BROWSER_ENGINE_PROTOCOL_VERSION: &str = "2.1"', ) && browserEngineAdapter.includes( "elastos.browser.engine-cleanup-binding/v2", @@ -1850,7 +1850,7 @@ assert( ) && browserStyle.includes('.browser-status[data-visible="true"][data-copyable="true"]') && browserStyle.includes(".browser-status-copy") && - browser.includes("browser.js?v=browser-20260907b") && + browser.includes("browser.js?v=browser-20260907c") && !browser.includes("browser.js?v=browser-20260731a") && !browser.includes("browser.js?v=browser-20260730a") && !browser.includes("browser.js?v=browser-20260728a") && @@ -1896,8 +1896,8 @@ assert( ); assert( - browserJs.includes("browser-status.js?v=browser-20260907b") && - browserRemoteDisplay.includes("browser-status.js?v=browser-20260907b") && + browserJs.includes("browser-status.js?v=browser-20260907c") && + browserRemoteDisplay.includes("browser-status.js?v=browser-20260907c") && !browserJs.includes("browser-status.js?v=browser-20260711c") && !browserRemoteDisplay.includes("browser-status.js?v=browser-20260711c") && !browserJs.includes("browser-status.js?v=browser-20260626e") && @@ -1912,7 +1912,7 @@ assert( ); assert( - browserJs.includes("browser-remote-display.js?v=browser-20260907b") && + browserJs.includes("browser-remote-display.js?v=browser-20260907c") && !browserJs.includes("browser-remote-display.js?v=browser-20260730b") && !browserJs.includes("browser-remote-display.js?v=browser-20260728a") && !browserJs.includes("browser-remote-display.js?v=browser-20260727a") && diff --git a/scripts/browser-vm-artifact-preflight-smoke.sh b/scripts/browser-vm-artifact-preflight-smoke.sh index 250f3d95f..ab596a4da 100755 --- a/scripts/browser-vm-artifact-preflight-smoke.sh +++ b/scripts/browser-vm-artifact-preflight-smoke.sh @@ -238,6 +238,110 @@ if debugfs and mke2fs: assert contract["source_kind"] == "ext4_image" and contract["inspectable"] is True assert contract["verified_sidecar"] is True and contract["audio_default_ready"] is True check("same real ext4 without debugfs", valid, True) + # The VZ host fixture is platform-specific; ext4 integrity runs on both hosts. + import platform as host_platform + if host_platform.system() == "Darwin" and host_platform.machine() == "arm64": + # Exercise the read-only host operation with a complete, known fixture set. + import http.client + import socket + import time + (data / "bin/initrd").write_bytes(b"fixture-initrd") + for name in ("kernel", "initrd"): + artifact = data / "bin" / ("vmlinux" if name == "kernel" else name) + valid[name] = {"size": artifact.stat().st_size, + "sha256": hashlib.sha256(artifact.read_bytes()).hexdigest()} + host_helper = data / "bin/browser-vz-engine-supervisor" + host_helper.write_text("#!/bin/sh\nprintf '%s\\n' '{\"schema\":\"elastos.browser.vm-host-capabilities/v1\",\"available\":true}'\n") + sidecar.write_text(json.dumps(valid)) + + def host_check(name, expected): + proc = subprocess.run([str(repo / "scripts/browser-vm-artifact-preflight.sh"), "--host-readiness"], + env={**env, "ELASTOS_DEBUGFS_BIN": debugfs}, + capture_output=True, text=True, timeout=15, check=True) + result = json.loads(proc.stdout) + assert result == {"schema":"elastos.browser.engine-readiness/v1", "readiness":expected}, (name, result) + checks.append(name) + + host_check("complete fixture host readiness", {"state":"ready"}) + original_kernel = (data / "bin/vmlinux").read_bytes() + (data / "bin/vmlinux").write_bytes(b"changed kernel") + host_check("host rejects changed kernel", {"state":"unavailable","reason":"artifact_invalid"}) + (data / "bin/vmlinux").write_bytes(original_kernel) + (data / "bin/initrd").unlink() + host_check("host rejects missing initrd", {"state":"unavailable","reason":"artifact_invalid"}) + (data / "bin/initrd").write_bytes(b"fixture-initrd") + host_helper.write_text(host_helper.read_text().replace("true", "false")) + host_check("host rejects unavailable virtualization", {"state":"unavailable","reason":"host_unsupported"}) + host_helper.write_text(host_helper.read_text().replace("false", "true")) + + scripts = data / "scripts" + scripts.mkdir() + shutil.copy2(repo / "scripts/browser-vm-artifact-preflight.sh", scripts) + import shlex + probe_count = scratch / "readiness-probe-count" + installed_probe = scripts / "browser-vm-artifact-preflight.sh" + installed_probe.write_text(installed_probe.read_text().replace("set -euo pipefail", + "set -euo pipefail\nprintf x >> " + shlex.quote(str(probe_count)), 1)) + control_socket = str(scratch / "readiness.sock") + service_env = {**env, "ELASTOS_DEBUGFS_BIN":debugfs, + "ELASTOS_BROWSER_VM_CONTROL_SERVICE_CONFIG":json.dumps({ + "schema":"elastos.browser.vm-control-service.config/v1", + "control_socket_path":control_socket, "launcher_program":str(host_helper), + "network_mode":"runtime_net_only", "direct_network":False, + })} + service = subprocess.Popen([shutil.which("node"), str(repo / "scripts/browser-vm-control-service.mjs")], + env=service_env, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True) + def get(path): + client = http.client.HTTPConnection("browser-vm", timeout=12) + client.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + client.sock.settimeout(12) + client.sock.connect(control_socket) + try: + client.request("GET", path) + response = client.getresponse() + assert response.status == 200 + return json.loads(response.read()) + finally: + client.close() + try: + deadline = time.monotonic() + 5 + while not pathlib.Path(control_socket).exists() and time.monotonic() < deadline and service.poll() is None: + time.sleep(0.02) + assert pathlib.Path(control_socket).exists(), "control service did not start" + assert get("/readiness")["readiness"] == {"state":"ready"} + checks.append("control service reports host readiness") + assert get("/readiness")["readiness"] == {"state":"ready"} + assert probe_count.read_text() == "x" + checks.append("unchanged admitted artifacts reuse verified readiness") + (data / "bin/vmlinux").write_bytes(b"changed kernel") + assert get("/readiness")["readiness"] == {"state":"unavailable","reason":"artifact_invalid"} + checks.append("control service notices artifact change") + assert probe_count.read_text() == "xx" + (data / "bin/vmlinux").write_bytes(original_kernel) + status = get("/status") + assert status["active_pages"] == status["active_vms"] == status["pending_launches"] == 0 + checks.append("readiness creates no page or VM") + sleeper_pid = scratch / "readiness-sleeper.pid" + installed_probe.write_text("#!/bin/sh\necho $$ > " + shlex.quote(str(sleeper_pid)) + "\nexec sleep 30\n") + started = time.monotonic() + assert get("/readiness")["readiness"] == {"state":"unavailable","reason":"preparation_required"} + assert time.monotonic() - started < 10 + try: + os.kill(int(sleeper_pid.read_text()), 0) + except ProcessLookupError: + pass + else: + raise AssertionError("timed-out readiness probe remained alive") + checks.append("slow readiness is bounded and its process is reaped") + finally: + service.terminate() + try: + _, stderr = service.communicate(timeout=5) + except subprocess.TimeoutExpired: + service.kill() + _, stderr = service.communicate(timeout=5) + if service.returncode not in (0, -15): + raise AssertionError(stderr) check("real ext4 missing receipt with debugfs", None, False, debugfs, "sidecar missing") check("real ext4 wrong architecture with debugfs", {**valid, "target_platform": "linux-amd64"}, False, debugfs, "target_platform") diff --git a/scripts/browser-vm-artifact-preflight.sh b/scripts/browser-vm-artifact-preflight.sh index 515ec25cf..6b6d3aca5 100755 --- a/scripts/browser-vm-artifact-preflight.sh +++ b/scripts/browser-vm-artifact-preflight.sh @@ -1,6 +1,12 @@ #!/usr/bin/env bash set -euo pipefail +mode="${1:-artifact}" +if [[ "$#" -gt 1 || ( "$mode" != "artifact" && "$mode" != "--host-readiness" ) ]]; then + echo "usage: browser-vm-artifact-preflight.sh [--host-readiness]" >&2 + exit 2 +fi + platform="${ELASTOS_BROWSER_VM_PLATFORM:-}" if [[ -z "$platform" ]]; then case "$(uname -s)-$(uname -m)" in @@ -33,7 +39,7 @@ engine_supervisor="${ELASTOS_BROWSER_VM_ENGINE_SUPERVISOR:-${data_dir}/bin/brows target_preflight="${ELASTOS_BROWSER_VM_TARGET_PREFLIGHT:-${repo_root}/scripts/browser-vm-target-preflight.sh}" debugfs_bin="${ELASTOS_DEBUGFS_BIN:-$(command -v debugfs 2>/dev/null || true)}" -python3 - \ +exec python3 - \ "$platform" \ "$data_dir" \ "$control_socket" \ @@ -46,7 +52,7 @@ python3 - \ "$control_service" \ "$engine_supervisor" \ "$target_preflight" \ - "$debugfs_bin" <<'PY' + "$debugfs_bin" "$mode" <<'PY' import json import hashlib import os @@ -69,6 +75,7 @@ import sys engine_supervisor, target_preflight, debugfs_bin, + mode, ) = sys.argv[1:] REQUIRED_ROOTFS_FILES = { @@ -484,6 +491,55 @@ missing_for_local_substrate = [ local_substrate_artifacts_ready = not missing_for_local_substrate and rootfs_contract["ok"] launch_ready = bool(control["control_socket"]["ok"]) +if mode == "--host-readiness": + # This operation is read-only. It admits the local host's immutable image + # set; control-socket presence is independent of artifact identity. + reason = None + import platform as host_platform + host_id = {("Darwin", "arm64"): "darwin-arm64", ("Linux", "x86_64"): "linux-amd64", + ("Linux", "aarch64"): "linux-arm64", ("Linux", "arm64"): "linux-arm64"}.get( + (host_platform.system(), host_platform.machine())) + if platform != host_id: + reason = "host_unsupported" + elif not rootfs_contract.get("verified_sidecar"): + reason = "artifact_invalid" if pathlib.Path(rootfs).is_file() else "preparation_required" + elif not local_substrate_artifacts_ready: + reason = "preparation_required" + else: + try: + receipt = json.loads(pathlib.Path(rootfs_manifest).read_text()) + initrd = os.environ.get("ELASTOS_BROWSER_VM_INITRAMFS") or str(pathlib.Path(data_dir) / "bin/initrd") + for name, artifact in [("kernel", kernel), ("initrd", initrd)]: + entry = receipt.get(name, {}) + actual = pathlib.Path(artifact) + if (not actual.is_file() or entry.get("size") != actual.stat().st_size + or entry.get("sha256") != sha256_file(artifact)): + reason = "artifact_invalid" + break + if reason is None and platform == "darwin-arm64": + probe = subprocess.run([vz_supervisor, "--host-capabilities"], + capture_output=True, text=True, timeout=2, check=True) + host = json.loads(probe.stdout) + if host.get("schema") != "elastos.browser.vm-host-capabilities/v1": + reason = "readiness_unsupported" + elif host.get("available") is not True: + reason = ("preparation_required" if host.get("reason") == "preparation_required" + else "host_unsupported") + elif reason is None and platform in {"linux-arm64", "linux-amd64"}: + import fcntl + with open("/dev/kvm", "r+b", buffering=0) as kvm: + if fcntl.ioctl(kvm.fileno(), 0xAE00, 0) != 12: + reason = "host_unsupported" + elif reason is None: + reason = "host_unsupported" + except (ValueError, subprocess.SubprocessError, AttributeError): + reason = "readiness_unsupported" + except OSError: + reason = "host_unsupported" + readiness = {"state": "unavailable", "reason": reason} if reason else {"state": "ready"} + print(json.dumps({"schema": "elastos.browser.engine-readiness/v1", "readiness": readiness})) + sys.exit(0) + if launch_ready: reason = "Browser VM control socket is available; Runtime can delegate Browser launches." elif local_substrate_artifacts_ready: diff --git a/scripts/browser-vm-control-service.mjs b/scripts/browser-vm-control-service.mjs index b56b554dd..c4cc35c2d 100755 --- a/scripts/browser-vm-control-service.mjs +++ b/scripts/browser-vm-control-service.mjs @@ -5,10 +5,76 @@ import http from "node:http"; import net from "node:net"; import path from "node:path"; import process from "node:process"; -import { spawn } from "node:child_process"; +import { execFile, spawn } from "node:child_process"; import { isDeepStrictEqual } from "node:util"; const CONFIG_ENV = "ELASTOS_BROWSER_VM_CONTROL_SERVICE_CONFIG"; +let verifiedHostReadiness = null; + +function readinessArtifactIdentity(dataDir, launcher) { + const artifact = (key, fallback) => process.env[key] || path.join(dataDir, fallback); + const files = [ + launcher, + artifact("ELASTOS_BROWSER_VM_ROOTFS", "browser-vm/rootfs.ext4"), + artifact("ELASTOS_BROWSER_VM_ROOTFS_MANIFEST", "browser-vm/browser-vm-rootfs-manifest.json"), + artifact("ELASTOS_BROWSER_VM_KERNEL", "bin/vmlinux"), + artifact("ELASTOS_BROWSER_VM_INITRAMFS", "bin/initrd"), + path.join(dataDir, "scripts/browser-vm-artifact-preflight.sh"), + ]; + if (process.env.ELASTOS_BROWSER_VM_PLATFORM?.startsWith("linux-")) { + files.push(artifact("ELASTOS_BROWSER_VM_CROSVM_BIN", "bin/crosvm"), "/dev/kvm"); + } + try { + return JSON.stringify(files.map((file) => { + const info = fs.statSync(file, { bigint: true }); + return [file, info.dev, info.ino, info.size, info.mtimeNs, info.ctimeNs].map(String); + })); + } catch { + return null; + } +} + +async function engineReadiness(config) { + const unavailable = (reason) => ({ + schema: "elastos.browser.engine-readiness/v1", + readiness: { state: "unavailable", reason }, + }); + // Remote providers report their own host readiness through the same Engine + // contract. An operator tunnel's local files cannot certify its remote host. + if (path.basename(config.launcher_program).startsWith("browser-vm-remote-vz-launcher")) { + return unavailable("readiness_unsupported"); + } + const dataDir = process.env.ELASTOS_BROWSER_VM_DATA_DIR; + if (!dataDir || !path.isAbsolute(dataDir)) return unavailable("preparation_required"); + const script = path.join(dataDir, "scripts/browser-vm-artifact-preflight.sh"); + const identity = readinessArtifactIdentity(dataDir, config.launcher_program); + if (identity && verifiedHostReadiness?.identity === identity) return verifiedHostReadiness.result; + return new Promise((resolve) => { + execFile(script, ["--host-readiness"], { + timeout: 8000, maxBuffer: 64 * 1024, + env: { + ...process.env, ELASTOS_BROWSER_VM_STAGED_ROOTFS: "", + ...(process.env.ELASTOS_BROWSER_VM_PLATFORM === "darwin-arm64" + ? { ELASTOS_BROWSER_VM_VZ_SUPERVISOR: config.launcher_program } : {}), + }, + }, (error, stdout) => { + if (error) return resolve(unavailable("preparation_required")); + try { + const result = JSON.parse(stdout); + if (result.schema !== "elastos.browser.engine-readiness/v1") throw new Error("schema"); + if (result.readiness?.state === "ready") { + if (!identity || readinessArtifactIdentity(dataDir, config.launcher_program) !== identity) { + return resolve(unavailable("preparation_required")); + } + verifiedHostReadiness = { identity, result }; + } + resolve(result); + } catch { + resolve(unavailable("readiness_unsupported")); + } + }); + }); +} const OPEN_REQUEST_ENV = "ELASTOS_BROWSER_VM_OPEN_REQUEST"; const MAX_BROWSER_FILE_UPLOAD_BYTES = 16 * 1024 * 1024; const MAX_BROWSER_INPUT_BODY_BYTES = @@ -3492,6 +3558,10 @@ function main() { }; try { const url = new URL(req.url || "/", "http://browser-vm-control"); + if (req.method === "GET" && url.pathname === "/readiness") { + sendJson(200, await engineReadiness(config)); + return; + } if (req.method === "GET" && url.pathname === "/status") { sendJson(200, { schema: "elastos.browser.vm-control-service.status/v1", diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index 266b3b4b2..e9c766b47 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -8534,7 +8534,7 @@ assert( assert( browserEngineAdapter.includes("elastos.browser.engine.page/v1") && browserEngineAdapter.includes( - 'const BROWSER_ENGINE_PROTOCOL_VERSION: &str = "2.0"', + 'const BROWSER_ENGINE_PROTOCOL_VERSION: &str = "2.1"', ) && browserEngineAdapter.includes( "elastos.browser.engine-cleanup-binding/v2", diff --git a/state.md b/state.md index eff33d7ca..e8b80323c 100644 --- a/state.md +++ b/state.md @@ -94,6 +94,24 @@ Mac and Linux host adapters now query their virtualization APIs for eligibility. The Mac unit check ran locally; the Linux-specific KVM query still needs a Linux test run. +The current source adds Engine protocol 2.1 readiness before profile preparation +and launch effects. Runtime retains explicit Engine choice and can automatically +select another compatible, ready entry from the admitted inventory. VM host +readiness verifies the image set and host eligibility; successful checks are +reused while artifact identities remain unchanged. A failed prewarm retains the +other configured adapters. Browser maps typed readiness failures to repair and +selection instructions. Installed Runtime, control-service and helper parity, +normal remote Runtime service admission, and device/media proof remain open. +Source verification passed 10 shared-contract tests, 80 Engine tests, 225 Browser +Runtime library tests, 15 Browser startup tests, 43 Browser JavaScript tests and +26 artifact/control-service cases. The last group includes real ext4 inspection, +readiness cache reuse, invalidation after a kernel change, and zero page/VM +acquisition and bounded probe termination. A read-only run of the built VZ helper +reports preparation required before the official signing step and available +host support after signing the same development build. That check launched no VM +and changed no installed artifact. These are source and controlled fixture +results; independent review and installed acceptance remain open. + Read-only target observations on 2026-09-07 identify a macOS 26.5.2 ARM64 machine with 24 GiB RAM, 10 logical CPUs and hardware virtualization reported available; its Brave version is 152.1.94.119. A Linux AMD64 server reports From e4861c622bf66b9ff42ce21142b8b5ff40a2e2a6 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 20:31:27 -0500 Subject: [PATCH 079/535] fix(browser): preserve verified guest images during setup --- TASKS.md | 8 +- docs/BROWSER_VM_TARGET.md | 82 +- scripts/browser-entropy-check.mjs | 25 +- .../browser-vm-artifact-preflight-smoke.sh | 28 +- scripts/browser-vm-artifact-preflight.sh | 48 +- scripts/browser-vm-control-service.mjs | 8 +- scripts/browser-vm-target-refresh.sh | 143 +--- scripts/home-entropy-check.mjs | 11 +- ...tup-source-home-browser-artifacts-smoke.sh | 217 +++-- .../setup-source-home-browser-artifacts.sh | 269 +++--- ...ource-home-browser-rootfs-refresh-smoke.sh | 195 ----- ...-source-home-collaboration-config-smoke.sh | 20 +- scripts/setup-source-home.sh | 788 +----------------- state.md | 20 +- 14 files changed, 432 insertions(+), 1430 deletions(-) delete mode 100755 scripts/setup-source-home-browser-rootfs-refresh-smoke.sh diff --git a/TASKS.md b/TASKS.md index e379c067e..0ebacf731 100644 --- a/TASKS.md +++ b/TASKS.md @@ -34,9 +34,11 @@ installed artifact admission, approved remote-service selection, and hash-bound device-role proof. The inspected Mac's existing VM image failed its manifest hash check. Rootfs preflight now checks the receipt with and without direct guest inspection. Repair or rebuild the image with -verified provenance and a matching receipt before qualification. Setup refresh -currently changes guest files without renewing that receipt; add an atomic, -provenance-bound update before relying on it for preparation. Runtime still +verified provenance and a matching receipt before qualification. Source-home setup now verifies a complete image set and preserves its bytes; +managed projections carry the matching receipt. Host maintenance reports guest +drift before any mutation. Build guest changes through the image builder. Add +automatic artifact acquisition and an atomic, provenance-bound host/guest update +before relying on this developer setup for ordinary installation. Runtime still uses a typed readiness result before profile preparation and launch. Finish atomic artifact admission, project that result into service discovery, and prove the installed path. Verify cold and unchanged-artifact readiness latency on each diff --git a/docs/BROWSER_VM_TARGET.md b/docs/BROWSER_VM_TARGET.md index 5e4cd398d..b0555f063 100644 --- a/docs/BROWSER_VM_TARGET.md +++ b/docs/BROWSER_VM_TARGET.md @@ -357,21 +357,29 @@ the matching `bin/initrd` when initramfs boot is used, and `browser-vm/rootfs.ext4` with its matching `browser-vm/browser-vm-rootfs-manifest.json`. Linux crosvm readiness requires `/dev/kvm`, `bin/crosvm`, `bin/vmlinux`, `browser-vm/initrd`, and the same -rootfs contract. A no-KVM public gateway can still be launch-ready if -`ELASTOS_BROWSER_VM_CONTROL_SOCKET` points to a local Runtime-facing control -socket backed by an approved remote/operator VM provider. - -`scripts/setup-source-home.sh` refreshes the VM guest manifest and startup -scripts in the installed script mirror, `browser-vm/initrd` and `bin/initrd` -when present, and `browser-vm/rootfs.ext4` when present. Existing VM artifacts -are backed up before the refresh and verified by reading the updated files back -from the artifact, so source-home Browser diagnostics, control behavior, and -Selkies startup behavior do not drift from the checked-in source. This refresh -path cannot add guest OS packages, change Python site-packages, or replace the -complete guest image contract; those changes require a rebuilt rootfs artifact. -`ELASTOS_BROWSER_VM_INITRD` and -`ELASTOS_BROWSER_VM_INITRAMFS` can narrow the initrd refresh to explicit -artifact paths for crosvm and VZ target maintenance. +rootfs contract. A gateway without KVM can consume an approved remote Engine. +Protocol 2.1 requires that Engine to report readiness; a local socket alone +provides diagnostic connectivity evidence. + +`scripts/setup-source-home.sh` consumes a verified image set. Its artifact +helper checks rootfs, kernel and initrd against the same build manifest before +creating links, and carries that manifest with the image. An existing file +retains its owner; a mixed or changed set fails with a repair error. Managed +Runtime projections use read-only links to the selected artifact store. Setup +preserves guest bytes, including files reached through symlinks. + +Guest changes belong to `scripts/build/build-browser-vm-rootfs.sh`, which stages +the guest and produces a new image, boot files and manifest together. Source-home +setup builds the host Runtime and providers; it no longer cross-builds guest +helpers or patches installed rootfs/initrd files. A host setup alone therefore +does not establish that its guest was built from the same source. Bind the guest +build provenance and host installation receipt before product qualification. + +Use `scripts/browser-vm-artifact-preflight.sh --verify-image-set` for the +read-only image-set check, independent of host virtualization. It uses +`ELASTOS_BROWSER_VM_INITRD` or `browser-vm/initrd` for Linux crosvm, and +`ELASTOS_BROWSER_VM_INITRAMFS` or `bin/initrd` for Mac VZ, matching each launcher. +The control-service readiness cache watches those same files. ## Source-Home Install Truth @@ -395,16 +403,14 @@ generator/stager for: - `bin/browser-vm-remote-vz-launcher` - `bin/browser-vm-prepare-rootfs-pool` - `bin/browser-vz-engine-supervisor` on macOS after the VZ helper is built -- `browser-vm/rootfs.ext4`, `browser-vm/initrd`, and their refreshed guest - scripts when those artifacts already exist +- verified links to an existing rootfs, kernel, initrd and matching build manifest For already-provisioned targets, `scripts/browser-vm-target-refresh.sh` is the -canonical drift-repair path. Do not repair target hosts by hand-copying one -helper and editing `components.json`; that creates a second install truth. A -future release may promote Browser VM helpers and rootfs/initrd artifacts into -explicit component entries, but until that decision is made the renewable -source-home setup/refresh scripts plus artifact preflight are the reviewable -contract. +host-helper maintenance path. Guest drift requires a rebuilt image set. Runtime +package admission, automatic compatible artifact acquisition, and atomic updates +of a complete host/guest set remain B01/B02/B15 work. These developer projections +verify file integrity; publisher authorization and installed product proof retain +their separate gates. ## Target Refresh @@ -413,25 +419,17 @@ runtime from source. It requires the normal source toolchain because it can buil Rust helpers and WASM capsules. Use `scripts/browser-vm-target-refresh.sh` for already-provisioned Browser VM -targets when the reviewed source checkout is current but the installed Browser -VM helper scripts or guest script artifacts may have drifted. It does not build -Rust/WASM artifacts. It copies the Browser VM script helpers, refreshes the -guest target manifest, `browser-selkies-control-service.mjs`, and -`browser-vm-selkies-start` inside existing rootfs artifacts, refreshes the -initrd control service helper, preserves `browser-vm/initrd` and -`browser-vm/rootfs.ext4` symlinks, and creates timestamped backups before -changed writes. If the guest-control bridge binary has already been built for -the Linux guest architecture, pass `--guest-control-bridge-bin ` to -refresh `/opt/elastos/bin/browser-vm-guest-control-bridge` inside the rootfs -with the same backup and `debugfs` verification path. Refresh-only is not -sufficient for package/dependency changes, Chromium wrapper changes, Selkies -Python package patches, or other complete guest image changes; rebuild/restage -the Browser VM rootfs and run the artifact preflight before claiming parity. - -Run the target refresh in two phases: `--verify-only` first, then the write pass -only if drift is reported. This keeps target closeout reviewable because the -operator can see exactly whether source, installed helpers, or VM artifacts -changed before mutating the target. +host helpers. It verifies the image set and checks guest parity before changing +host helpers. A guest mismatch reports the required rebuild and preserves the +rootfs, initrd, build manifest and installed host helpers. The optional +`--guest-control-bridge-bin` compares the built bridge with the guest copy. +The script retains one backup set for changed host helpers under its existing +retention rule; guest artifacts remain build outputs. + +Run `--verify-only` first. If host helpers alone differ, the write pass installs +them. If guest files differ, build and install the complete image set before +continuing. The image builder supplies guest dependencies, Chromium wrappers, +Selkies patches and guest helpers through one build path. Typical Jetson source-home maintenance: diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index abffc3e80..a8845313d 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -1141,31 +1141,18 @@ assert( setupSourceHome.includes("browser-vm-artifact-preflight.sh") && setupSourceHome.includes("browser-vm-target-preflight.sh") && setupSourceHome.includes("setup-source-home-browser-artifacts.sh") && + !setupSourceHome.includes("refresh_browser_vm_rootfs_files") && + !setupSourceHome.includes("refresh_browser_vm_initrd_control_service") && setupSourceHomeBrowserArtifacts.includes("elastos.setup-source-home.browser-artifacts/v1") && setupSourceHomeBrowserArtifacts.includes("managed-runtimes") && setupSourceHomeBrowserArtifacts.includes("browser-vm/rootfs.ext4") && setupSourceHomeBrowserArtifacts.includes("bin/crosvm") && setupSourceHomeBrowserArtifacts.includes("browser-vm/initrd") && setupSourceHomeBrowserArtifacts.includes("bin/initrd") && + setupSourceHomeBrowserArtifacts.includes("--verify-image-set") && + setupSourceHomeBrowserArtifacts.includes("browser-vm/browser-vm-rootfs-manifest.json") && setupSourceHomeBrowserArtifactsSmoke.includes("elastos.setup-source-home.browser-artifacts-smoke/v1") && - setupSourceHomeBrowserArtifactsSmoke.includes("existing real kernel file must not be replaced") && - setupSourceHomeBrowserArtifactsSmoke.includes("Linux managed setup must not create the Mac VZ initrd path") && - setupSourceHomeBrowserArtifactsSmoke.includes("Mac managed setup must not create a crosvm link") && setupSourceHome.includes("browser-selkies-control-service.mjs") && - setupSourceHome.includes("browser-vm-selkies-start") && - setupSourceHome.includes("browser-vm-init") && - setupSourceHome.includes("extract_browser_vm_init") && - setupSourceHome.includes("extract_browser_vm_selkies_start") && - setupSourceHome.includes("write_browser_vm_target_manifest") && - setupSourceHome.includes('"guarantee_level": "mechanism_microvm"') && - setupSourceHome.includes("/etc/elastos/browser-vm-target.json") && - setupSourceHome.includes("resolve_browser_vm_native_proxy_source") && - setupSourceHome.includes("validate_linux_guest_binary") && - setupSourceHome.includes("/opt/elastos/bin/browser-native-proxy-engine") && - setupSourceHome.includes("refresh_browser_vm_initrd_control_service") && - setupSourceHome.includes("refresh_browser_vm_rootfs_files") && - setupSourceHome.includes("ELASTOS_DEBUGFS_BIN") && - setupSourceHome.includes("debugfs") && setupSourceHome.includes("ELASTOS_NODE_BIN") && setupSourceHome.includes("ELASTOS_BROWSER_VM_CONTROL_LAUNCHER") && setupSourceHome.includes("existing_remote_browser_vm_config") && @@ -1684,8 +1671,8 @@ assert( browserVmTargetDoc.includes("full bootable rootfs must also pass runtime dependency mode") && browserVmTargetDoc.includes("--target-dir /path/to/full-rootfs --require-runtime-deps") && browserVmTargetDoc.includes("PipeWire, PipeWire Pulse, WirePlumber") && - browserVmTargetDoc.includes("Refresh-only is not") && - browserVmTargetDoc.includes("sufficient for package/dependency changes") && + browserVmTargetDoc.includes("Guest drift requires a rebuilt image set") && + browserVmTargetDoc.includes("preserves guest bytes") && browserVmTargetDoc.includes("browser-vm-runtime-relay") && browserVmTargetDoc.includes("browser-vm-guest-control-bridge") && browserVmTargetDoc.includes("browser-vm-selkies-start") && diff --git a/scripts/browser-vm-artifact-preflight-smoke.sh b/scripts/browser-vm-artifact-preflight-smoke.sh index ab596a4da..bfc22794a 100755 --- a/scripts/browser-vm-artifact-preflight-smoke.sh +++ b/scripts/browser-vm-artifact-preflight-smoke.sh @@ -238,6 +238,32 @@ if debugfs and mke2fs: assert contract["source_kind"] == "ext4_image" and contract["inspectable"] is True assert contract["verified_sidecar"] is True and contract["audio_default_ready"] is True check("same real ext4 without debugfs", valid, True) + # Target maintenance must report guest drift before touching host helpers + # or rewriting the image/receipt. Use a real cpio initrd with an old helper. + import gzip + initrd_tree = scratch / "initrd-tree" + (initrd_tree / "bin").mkdir(parents=True) + (initrd_tree / "bin/browser-selkies-control-service.mjs").write_text("old control helper") + packed = subprocess.run(["cpio", "-o", "--format=newc"], cwd=initrd_tree, + input=b"bin/browser-selkies-control-service.mjs\n", capture_output=True, check=True).stdout + for rel in ["bin/initrd", "browser-vm/initrd"]: + (data / rel).write_bytes(gzip.compress(packed)) + maintenance_receipt = copy.deepcopy(valid) + for name, rel in [("kernel", "bin/vmlinux"), ("initrd", "bin/initrd")]: + artifact = data / rel + maintenance_receipt[name] = {"size": artifact.stat().st_size, + "sha256": hashlib.sha256(artifact.read_bytes()).hexdigest()} + sidecar.write_text(json.dumps(maintenance_receipt)) + preserved = {p: hashlib.sha256(p.read_bytes()).hexdigest() + for p in [image, sidecar, data / "bin/initrd", data / "browser-vm/initrd"]} + refresh = subprocess.run([str(repo / "scripts/browser-vm-target-refresh.sh"), + "--source-dir", str(repo), "--data-dir", str(data)], + env={**env, "ELASTOS_DEBUGFS_BIN":debugfs}, capture_output=True, text=True, timeout=30) + assert refresh.returncode == 1 and "Rebuild and install the complete image set" in refresh.stderr, (refresh.stdout, refresh.stderr) + assert all(hashlib.sha256(p.read_bytes()).hexdigest() == digest for p,digest in preserved.items()) + assert not (data / "scripts").exists() and not (data / "backups").exists() + checks.append("guest drift preserves real ext4, initrd, receipt and host helpers") + sidecar.write_text(json.dumps(valid)) # The VZ host fixture is platform-specific; ext4 integrity runs on both hosts. import platform as host_platform if host_platform.system() == "Darwin" and host_platform.machine() == "arm64": @@ -268,7 +294,7 @@ if debugfs and mke2fs: host_check("host rejects changed kernel", {"state":"unavailable","reason":"artifact_invalid"}) (data / "bin/vmlinux").write_bytes(original_kernel) (data / "bin/initrd").unlink() - host_check("host rejects missing initrd", {"state":"unavailable","reason":"artifact_invalid"}) + host_check("host rejects missing initrd", {"state":"unavailable","reason":"preparation_required"}) (data / "bin/initrd").write_bytes(b"fixture-initrd") host_helper.write_text(host_helper.read_text().replace("true", "false")) host_check("host rejects unavailable virtualization", {"state":"unavailable","reason":"host_unsupported"}) diff --git a/scripts/browser-vm-artifact-preflight.sh b/scripts/browser-vm-artifact-preflight.sh index 6b6d3aca5..97d903ac0 100755 --- a/scripts/browser-vm-artifact-preflight.sh +++ b/scripts/browser-vm-artifact-preflight.sh @@ -2,8 +2,8 @@ set -euo pipefail mode="${1:-artifact}" -if [[ "$#" -gt 1 || ( "$mode" != "artifact" && "$mode" != "--host-readiness" ) ]]; then - echo "usage: browser-vm-artifact-preflight.sh [--host-readiness]" >&2 +if [[ "$#" -gt 1 || ( "$mode" != "artifact" && "$mode" != "--host-readiness" && "$mode" != "--verify-image-set" ) ]]; then + echo "usage: browser-vm-artifact-preflight.sh [--host-readiness|--verify-image-set]" >&2 exit 2 fi @@ -365,6 +365,7 @@ def inspect_ext4_sidecar_manifest(image): result["missing"] = preflight.get("missing") if isinstance(preflight.get("missing"), list) else [] result["manifest"] = preflight.get("manifest") if isinstance(preflight.get("manifest"), dict) else {} result["preflight"] = preflight + result["build_receipt"] = manifest for name in REQUIRED_ROOTFS_FILES: entry = result["required"].get(name) @@ -455,6 +456,36 @@ def inspect_rootfs(): rootfs_contract = inspect_rootfs() + +def verify_image_set(): + if not rootfs_contract.get("verified_sidecar") or not rootfs_contract.get("ok"): + return "artifact_invalid" if pathlib.Path(rootfs).is_file() else "preparation_required" + receipt = rootfs_contract["build_receipt"] + default_initrd = "bin/initrd" if platform == "darwin-arm64" else "browser-vm/initrd" + initrd_key = "ELASTOS_BROWSER_VM_INITRAMFS" if platform == "darwin-arm64" else "ELASTOS_BROWSER_VM_INITRD" + initrd = os.environ.get(initrd_key) or str(pathlib.Path(data_dir) / default_initrd) + try: + for name, artifact in [("kernel", kernel), ("initrd", initrd)]: + entry = receipt.get(name) + actual = pathlib.Path(artifact) + if not actual.is_file(): + return "preparation_required" + if (not isinstance(entry, dict) or type(entry.get("size")) is not int + or entry["size"] != actual.stat().st_size + or entry.get("sha256") != sha256_file(artifact)): + return "artifact_invalid" + except OSError: + return "artifact_invalid" + return None + + +if mode == "--verify-image-set": + reason = verify_image_set() + print(json.dumps({"schema": "elastos.browser.vm-image-set/v1", + "ok": reason is None, "reason": reason, + "rootfs_contract": rootfs_contract})) + sys.exit(1 if reason else 0) + control = { "control_socket": path_stat(control_socket, socket=True), "control_service": path_stat(control_service, executable=True), @@ -501,21 +532,12 @@ if mode == "--host-readiness": (host_platform.system(), host_platform.machine())) if platform != host_id: reason = "host_unsupported" - elif not rootfs_contract.get("verified_sidecar"): - reason = "artifact_invalid" if pathlib.Path(rootfs).is_file() else "preparation_required" + elif image_reason := verify_image_set(): + reason = image_reason elif not local_substrate_artifacts_ready: reason = "preparation_required" else: try: - receipt = json.loads(pathlib.Path(rootfs_manifest).read_text()) - initrd = os.environ.get("ELASTOS_BROWSER_VM_INITRAMFS") or str(pathlib.Path(data_dir) / "bin/initrd") - for name, artifact in [("kernel", kernel), ("initrd", initrd)]: - entry = receipt.get(name, {}) - actual = pathlib.Path(artifact) - if (not actual.is_file() or entry.get("size") != actual.stat().st_size - or entry.get("sha256") != sha256_file(artifact)): - reason = "artifact_invalid" - break if reason is None and platform == "darwin-arm64": probe = subprocess.run([vz_supervisor, "--host-capabilities"], capture_output=True, text=True, timeout=2, check=True) diff --git a/scripts/browser-vm-control-service.mjs b/scripts/browser-vm-control-service.mjs index c4cc35c2d..116109527 100755 --- a/scripts/browser-vm-control-service.mjs +++ b/scripts/browser-vm-control-service.mjs @@ -13,15 +13,17 @@ let verifiedHostReadiness = null; function readinessArtifactIdentity(dataDir, launcher) { const artifact = (key, fallback) => process.env[key] || path.join(dataDir, fallback); + const linux = process.env.ELASTOS_BROWSER_VM_PLATFORM?.startsWith("linux-") ?? process.platform === "linux"; const files = [ launcher, artifact("ELASTOS_BROWSER_VM_ROOTFS", "browser-vm/rootfs.ext4"), artifact("ELASTOS_BROWSER_VM_ROOTFS_MANIFEST", "browser-vm/browser-vm-rootfs-manifest.json"), artifact("ELASTOS_BROWSER_VM_KERNEL", "bin/vmlinux"), - artifact("ELASTOS_BROWSER_VM_INITRAMFS", "bin/initrd"), + linux ? artifact("ELASTOS_BROWSER_VM_INITRD", "browser-vm/initrd") + : artifact("ELASTOS_BROWSER_VM_INITRAMFS", "bin/initrd"), path.join(dataDir, "scripts/browser-vm-artifact-preflight.sh"), ]; - if (process.env.ELASTOS_BROWSER_VM_PLATFORM?.startsWith("linux-")) { + if (linux) { files.push(artifact("ELASTOS_BROWSER_VM_CROSVM_BIN", "bin/crosvm"), "/dev/kvm"); } try { @@ -54,7 +56,7 @@ async function engineReadiness(config) { timeout: 8000, maxBuffer: 64 * 1024, env: { ...process.env, ELASTOS_BROWSER_VM_STAGED_ROOTFS: "", - ...(process.env.ELASTOS_BROWSER_VM_PLATFORM === "darwin-arm64" + ...((process.env.ELASTOS_BROWSER_VM_PLATFORM?.startsWith("darwin-") ?? process.platform === "darwin") ? { ELASTOS_BROWSER_VM_VZ_SUPERVISOR: config.launcher_program } : {}), }, }, (error, stdout) => { diff --git a/scripts/browser-vm-target-refresh.sh b/scripts/browser-vm-target-refresh.sh index 273db4eae..4cb127d76 100755 --- a/scripts/browser-vm-target-refresh.sh +++ b/scripts/browser-vm-target-refresh.sh @@ -8,25 +8,25 @@ usage() { Usage: scripts/browser-vm-target-refresh.sh [options] -Refresh deployed Browser VM helper scripts and guest helper artifacts from a +Refresh deployed Browser VM host helper scripts after verifying guest parity with a reviewed source checkout without running a full Rust/WASM source-home setup. Options: --source-dir PATH Source checkout. Default: this repository. --data-dir PATH ElastOS data dir. Default: platform data dir. - --initrd PATH Initrd to refresh. May be passed more than once. + --initrd PATH Initrd to verify. May be passed more than once. Default: existing browser-vm/initrd and bin/initrd. - --rootfs PATH Rootfs ext4 to refresh. Default: browser-vm/rootfs.ext4. + --rootfs PATH Rootfs ext4 to verify. Default: browser-vm/rootfs.ext4. --guest-control-bridge-bin PATH Optional prebuilt Linux guest-control bridge binary to - refresh inside the rootfs. + compare with the rootfs. --backup-dir PATH Backup directory. Default: data-dir/backups/browser-vm-target-refresh--. --node-bin PATH Node executable for refreshed wrapper scripts. Default: auto-detect. --verify-only Report whether target files match; do not write. --help, -h Show this help. -The script preserves initrd/rootfs symlinks by updating their resolved targets. -It creates timestamped backups before changed installed helpers or VM artifacts +The script preserves initrd/rootfs bytes and their build receipt. Guest changes +require a rebuilt image set. It creates backups before changed installed host helpers and retains one default backup set. ELASTOS_BROWSER_VM_BACKUP_RETENTION must be 1. Finish target closeout by running: @@ -160,13 +160,6 @@ prune_default_backup_dirs() { done } -safe_label() { - local label="${1#/}" - label="${label//\//_}" - label="${label// /_}" - printf '%s\n' "$label" -} - install_with_backup() { local source="$1" local target="$2" @@ -222,21 +215,14 @@ initrd_helper_matches() { return "$status" } -refresh_initrd() { +check_initrd() { local requested_initrd="$1" local source="$2" local initrd - local work_dir - local verify_dir - local gzip_level="${ELASTOS_BROWSER_VM_INITRD_GZIP_LEVEL:-1}" if [[ ! -f "$requested_initrd" ]]; then return fi - if [[ ! "$gzip_level" =~ ^[1-9]$ ]]; then - echo "ELASTOS_BROWSER_VM_INITRD_GZIP_LEVEL must be 1-9" >&2 - exit 1 - fi command -v gzip >/dev/null 2>&1 || { echo "gzip not found" >&2; exit 1; } command -v cpio >/dev/null 2>&1 || { echo "cpio not found" >&2; exit 1; } @@ -245,27 +231,8 @@ refresh_initrd() { echo "[browser-vm-target-refresh] unchanged initrd helper: $requested_initrd" return fi - if [[ "$VERIFY_ONLY" == "1" ]]; then - echo "[browser-vm-target-refresh] drift initrd helper: $requested_initrd" - DRIFT=1 - return - fi - - backup_file "$initrd" "$(safe_label "$requested_initrd").initrd" >/dev/null - work_dir="$(mktemp -d)" - verify_dir="$(mktemp -d)" - gzip -dc "$initrd" | (cd "$work_dir" && cpio -id --quiet) - mkdir -p "$work_dir/bin" - install -m 644 "$source" "$work_dir/bin/browser-selkies-control-service.mjs" - (cd "$work_dir" && find . -print0 | cpio --null -o --format=newc 2>/dev/null | gzip "-${gzip_level}") > "${initrd}.new" - mv "${initrd}.new" "$initrd" - gzip -dc "$initrd" | (cd "$verify_dir" && cpio -id --quiet bin/browser-selkies-control-service.mjs) - if ! cmp -s "$source" "$verify_dir/bin/browser-selkies-control-service.mjs"; then - echo "initrd helper refresh did not verify: $requested_initrd" >&2 - exit 1 - fi - rm -rf "$work_dir" "$verify_dir" - echo "[browser-vm-target-refresh] refreshed initrd helper: $requested_initrd -> $initrd" + echo "[browser-vm-target-refresh] guest initrd requires a rebuilt image set: $requested_initrd" + DRIFT=1 } rootfs_helper_matches() { @@ -316,7 +283,7 @@ verify_rootfs_guest_control_bridge_contract() { fi done if [[ "$missing" == "1" && "$VERIFY_ONLY" != "1" ]]; then - echo "rootfs guest-control bridge is stale; pass --guest-control-bridge-bin with the current Linux guest bridge binary" >&2 + echo "rootfs guest-control bridge is stale; rebuild the image set with the current Linux guest bridge binary" >&2 exit 1 fi } @@ -383,50 +350,21 @@ target.write_text(text[start:end]) PY } -refresh_rootfs_file() { +check_rootfs_file() { local rootfs="$1" local source="$2" local guest_path="$3" - local mode="$4" - local label="$5" - local debugfs="$6" - local staged_source - local verify_file - local commands_file + local debugfs="$4" if rootfs_helper_matches "$rootfs" "$source" "$guest_path" "$debugfs"; then echo "[browser-vm-target-refresh] unchanged rootfs helper: $guest_path" return fi - if [[ "$VERIFY_ONLY" == "1" ]]; then - echo "[browser-vm-target-refresh] drift rootfs helper: $guest_path" - DRIFT=1 - return - fi - - if [[ -z "$ROOTFS_BACKUP_PATH" ]]; then - ROOTFS_BACKUP_PATH="$(backup_file "$rootfs" "$(safe_label "$rootfs").${label}.rootfs")" - fi - staged_source="$(mktemp)" - verify_file="$(mktemp)" - commands_file="$(mktemp)" - cp "$source" "$staged_source" - cat > "$commands_file" </dev/null 2>&1 - "$debugfs" -R "cat ${guest_path}" "$rootfs" > "$verify_file" 2>/dev/null - if ! cmp -s "$source" "$verify_file"; then - echo "rootfs helper refresh did not verify: $guest_path in $rootfs; backup kept at $ROOTFS_BACKUP_PATH" >&2 - exit 1 - fi - rm -f "$staged_source" "$verify_file" "$commands_file" - echo "[browser-vm-target-refresh] refreshed rootfs helper: $guest_path -> $rootfs" + echo "[browser-vm-target-refresh] guest file requires a rebuilt image set: $guest_path" + DRIFT=1 } -refresh_rootfs() { +check_rootfs_guest_files() { local requested_rootfs="$1" local source="$2" local rootfs @@ -445,7 +383,6 @@ refresh_rootfs() { fi rootfs="$(resolve_existing_symlink_target "$requested_rootfs")" - ROOTFS_BACKUP_PATH="" init_source="$(mktemp)" selkies_start_source="$(mktemp)" manifest_source="$(mktemp)" @@ -453,22 +390,17 @@ refresh_rootfs() { extract_browser_vm_selkies_start "$SOURCE_DIR" "$selkies_start_source" write_browser_vm_target_manifest "$manifest_source" - refresh_rootfs_file "$rootfs" "$manifest_source" \ - "/etc/elastos/browser-vm-target.json" "0100644" \ - "target-manifest" "$debugfs" - refresh_rootfs_file "$rootfs" "$source" \ - "/opt/elastos/bin/browser-selkies-control-service.mjs" "0100644" \ - "control-service" "$debugfs" - refresh_rootfs_file "$rootfs" "$init_source" \ - "/opt/elastos/bin/browser-vm-init" "0100755" \ - "vm-init" "$debugfs" - refresh_rootfs_file "$rootfs" "$selkies_start_source" \ - "/opt/elastos/bin/browser-vm-selkies-start" "0100755" \ - "selkies-start" "$debugfs" + check_rootfs_file "$rootfs" "$manifest_source" \ + "/etc/elastos/browser-vm-target.json" "$debugfs" + check_rootfs_file "$rootfs" "$source" \ + "/opt/elastos/bin/browser-selkies-control-service.mjs" "$debugfs" + check_rootfs_file "$rootfs" "$init_source" \ + "/opt/elastos/bin/browser-vm-init" "$debugfs" + check_rootfs_file "$rootfs" "$selkies_start_source" \ + "/opt/elastos/bin/browser-vm-selkies-start" "$debugfs" if [[ -n "$GUEST_CONTROL_BRIDGE_BIN" ]]; then - refresh_rootfs_file "$rootfs" "$GUEST_CONTROL_BRIDGE_BIN" \ - "/opt/elastos/bin/browser-vm-guest-control-bridge" "0100755" \ - "guest-control-bridge" "$debugfs" + check_rootfs_file "$rootfs" "$GUEST_CONTROL_BRIDGE_BIN" \ + "/opt/elastos/bin/browser-vm-guest-control-bridge" "$debugfs" fi verify_rootfs_guest_control_bridge_contract "$rootfs" "$debugfs" rm -f "$init_source" "$selkies_start_source" "$manifest_source" @@ -592,6 +524,24 @@ echo "[browser-vm-target-refresh] data: $DATA_DIR" echo "[browser-vm-target-refresh] backup: $BACKUP_DIR" echo "[browser-vm-target-refresh] mode: $([[ "$VERIFY_ONLY" == "1" ]] && echo verify-only || echo write)" +if [[ -f "$ROOTFS" ]]; then + ELASTOS_BROWSER_VM_DATA_DIR="$DATA_DIR" \ + ELASTOS_BROWSER_VM_ROOTFS="$ROOTFS" \ + "$SOURCE_DIR/scripts/browser-vm-artifact-preflight.sh" --verify-image-set +fi + +if [[ "${#INITRDS[@]}" -gt 0 ]]; then + for initrd in "${INITRDS[@]}"; do + check_initrd "$initrd" "$selkies_source" + done +fi +check_rootfs_guest_files "$ROOTFS" "$selkies_source" + +if [[ "$DRIFT" == "1" ]]; then + echo "Browser guest files differ from source. Rebuild and install the complete image set before refreshing host helpers." >&2 + exit 1 +fi + install_with_backup "$selkies_source" \ "$DATA_DIR/scripts/browser-selkies-control-service.mjs" 644 \ "browser-selkies-control-service.mjs" @@ -642,13 +592,6 @@ write_node_wrapper "$DATA_DIR/bin/browser-vm-prepare-rootfs-pool" \ "$DATA_DIR/scripts/browser-vm-prepare-rootfs-pool.mjs" \ "browser-vm-prepare-rootfs-pool" -if [[ "${#INITRDS[@]}" -gt 0 ]]; then - for initrd in "${INITRDS[@]}"; do - refresh_initrd "$initrd" "$selkies_source" - done -fi -refresh_rootfs "$ROOTFS" "$selkies_source" - if [[ "$VERIFY_ONLY" == "1" && "$DRIFT" == "1" ]]; then echo "[browser-vm-target-refresh] verify-only found drift" exit 1 diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index e9c766b47..190d88557 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -9907,22 +9907,15 @@ assert( ) && browserSelkiesRuntimeExitSmoke.includes("--cleanup-after-verify") && setupSourceHome.includes("install_browser_runtime_helpers") && + !setupSourceHome.includes("refresh_browser_vm_rootfs_files") && + !setupSourceHome.includes("refresh_browser_vm_initrd_control_service") && !setupSourceHome.includes("browser-per-launch-selkies-supervisor.mjs") && !setupSourceHome.includes("browser-selkies-runtime-exit-target.sh") && !setupSourceHome.includes("browser-hosted-product-operator-config.mjs") && !setupSourceHome.includes("browser-hosted-product-supervisor.mjs") && setupSourceHome.includes("browser-selkies-control-service.mjs") && - setupSourceHome.includes("browser-vm-selkies-start") && setupSourceHome.includes("build Browser VZ engine supervisor") && setupSourceHome.includes("-p elastos-vz --bin browser-vz-engine-supervisor") && - setupSourceHome.includes("extract_browser_vm_selkies_start") && - setupSourceHome.includes("resolve_browser_vm_native_proxy_source") && - setupSourceHome.includes("validate_linux_guest_binary") && - setupSourceHome.includes("/opt/elastos/bin/browser-native-proxy-engine") && - setupSourceHome.includes("refresh_browser_vm_initrd_control_service") && - setupSourceHome.includes("refresh_browser_vm_rootfs_files") && - setupSourceHome.includes("ELASTOS_DEBUGFS_BIN") && - setupSourceHome.includes("debugfs") && read("scripts/browser-hosted-product-target-preflight.sh").includes( "browser-hosted-product-display-smoke.sh", ) && diff --git a/scripts/setup-source-home-browser-artifacts-smoke.sh b/scripts/setup-source-home-browser-artifacts-smoke.sh index 4ecc75b1d..69c60f3fa 100755 --- a/scripts/setup-source-home-browser-artifacts-smoke.sh +++ b/scripts/setup-source-home-browser-artifacts-smoke.sh @@ -1,101 +1,146 @@ #!/usr/bin/env bash set -euo pipefail - repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -tmp_dir="$(mktemp -d)" -trap 'rm -rf "$tmp_dir"' EXIT +exec python3 - "$repo_root" <<'PY' +import copy +import hashlib +import json +import os +from pathlib import Path +import re +import shutil +import subprocess +import sys +import tempfile + +repo = Path(sys.argv[1]) +checks = [] +# This is an artifact ownership fixture. Real ext4 inspection is covered by +# browser-vm-artifact-preflight-smoke.sh; these bytes are never launched. +contract = {"schema": "elastos.browser.vm-target/v1", "engine": "chromium_microvm", + "network_mode": "runtime_net_only", "direct_network": False, "wallet_injection": False, + "media_transport": "runtime_relay", "display_mode": "webrtc_remote_display", + "guarantee_level": "mechanism_microvm", "display_backend": "vm_selkies_gstreamer_webrtc", + "runtime_exit_transport": "vsock_relay", "control_transport": "vsock_relay", "control_port": 19092} +required = "manifest init native_proxy runtime_relay guest_control_bridge control_service selkies_start node chromium xvfb python3 gst_inspect".split() +audio = "pipewire pipewire_pulse wireplumber pw_cli".split() +preflight = {"ok": True, "required": {n: {"ok": True} for n in required}, + "optional_audio": {n: {"ok": True} for n in audio}, "audio_default_ready": True, + "missing": [], "manifest_errors": [], "script_errors": [], "manifest": contract} -parent="$tmp_dir/xdg/elastos" -linux_managed="$parent/managed-runtimes/home/home/.local/share/elastos" -mac_managed="$parent/managed-runtimes/mac/home/.local/share/elastos" -explicit_source="$tmp_dir/explicit-artifacts" -mkdir -p \ - "$parent/bin" \ - "$parent/browser-vm" \ - "$linux_managed/bin" \ - "$linux_managed/browser-vm" \ - "$mac_managed/bin" \ - "$mac_managed/browser-vm" \ - "$explicit_source/bin" \ - "$explicit_source/browser-vm" +def sha(p): + return hashlib.sha256(p.read_bytes()).hexdigest() -printf 'parent-crosvm\n' > "$parent/bin/crosvm" -printf 'parent-kernel\n' > "$parent/bin/vmlinux" -printf 'parent-linux-initrd\n' > "$parent/browser-vm/initrd" -printf 'parent-rootfs\n' > "$parent/browser-vm/rootfs.ext4" -printf 'parent-vz-initrd\n' > "$parent/bin/initrd" -printf 'keep-existing-kernel\n' > "$linux_managed/bin/vmlinux" +def make_set(source, target="linux-arm64"): + (source / "browser-vm").mkdir(parents=True) + (source / "bin").mkdir() + for rel, contents in [("browser-vm/rootfs.ext4", b"fixture-image"), ("bin/vmlinux", b"fixture-kernel"), + ("bin/initrd", b"fixture-initrd"), ("browser-vm/initrd", b"fixture-initrd"), + ("bin/crosvm", b"fixture-crosvm")]: + (source / rel).write_bytes(contents) + image = source / "browser-vm/rootfs.ext4" + manifest = {"schema": "elastos.browser.vm-rootfs-build/v1", "ok": True, + "target_platform": target, "size": image.stat().st_size, "sha256": sha(image), + "preflight": copy.deepcopy(preflight)} + for name, rel in [("kernel", "bin/vmlinux"), ("initrd", "bin/initrd")]: + p = source / rel + manifest[name] = {"size": p.stat().st_size, "sha256": sha(p)} + (source / "browser-vm/browser-vm-rootfs-manifest.json").write_text(json.dumps(manifest)) + return manifest -linux_output="$("$repo_root/scripts/setup-source-home-browser-artifacts.sh" \ - --data-dir "$linux_managed" \ - --platform linux-arm64)" -OUTPUT="$linux_output" \ -PARENT="$parent" \ -MANAGED="$linux_managed" \ -node - <<'NODE' -const fs = require("node:fs"); -const path = require("node:path"); +with tempfile.TemporaryDirectory(prefix="browser-image-ownership-") as temp: + root = Path(temp) + env = {k:v for k,v in os.environ.items() if not k.startswith("ELASTOS_BROWSER_VM_")} + env["ELASTOS_DEBUGFS_BIN"] = str(root / "absent-debugfs") + script = repo / "scripts/setup-source-home-browser-artifacts.sh" -const output = process.env.OUTPUT.trim().split(/\n/).at(-1); -const receipt = JSON.parse(output); -const parent = process.env.PARENT; -const managed = process.env.MANAGED; -if (receipt.schema !== "elastos.setup-source-home.browser-artifacts/v1") throw new Error("wrong schema"); -if (receipt.linked !== 3) throw new Error(`expected 3 Linux links, got ${receipt.linked}`); -if (fs.readFileSync(path.join(managed, "bin/vmlinux"), "utf8") !== "keep-existing-kernel\n") { - throw new Error("existing real kernel file must not be replaced"); -} -for (const [dest, source] of [ - ["bin/crosvm", "bin/crosvm"], - ["browser-vm/initrd", "browser-vm/initrd"], - ["browser-vm/rootfs.ext4", "browser-vm/rootfs.ext4"], -]) { - const actual = fs.readlinkSync(path.join(managed, dest)); - const expected = path.join(parent, source); - if (actual !== expected) throw new Error(`${dest} link mismatch: ${actual} !== ${expected}`); -} -if (fs.existsSync(path.join(managed, "bin/initrd"))) { - throw new Error("Linux managed setup must not create the Mac VZ initrd path"); -} -NODE + def run(name, data, source=None, platform="darwin-arm64", ok=True): + command = [str(script), "--data-dir", str(data), "--platform", platform] + if source is not None: + command += ["--artifact-data-dir", str(source)] + p = subprocess.run(command, env=env, capture_output=True, text=True, timeout=20) + result = json.loads(p.stdout) + assert (p.returncode == 0) is ok, (name, result, p.stderr) + checks.append(name) + return result -printf 'explicit-kernel\n' > "$explicit_source/bin/vmlinux" -printf 'explicit-rootfs\n' > "$explicit_source/browser-vm/rootfs.ext4" -printf 'explicit-vz-initrd\n' > "$explicit_source/bin/initrd" + for platform in ["darwin-arm64", "linux-arm64", "linux-amd64"]: + source = root / platform + make_set(source, "linux-amd64" if platform == "linux-amd64" else "linux-arm64") + before = {str(p.relative_to(source)): sha(p) for p in source.rglob("*") if p.is_file()} + managed = source / "managed-runtimes/test/home/data" + result = run(platform + " complete set", managed, platform=platform) + assert result["image_set_verified"] and result["linked"] == (4 if platform == "darwin-arm64" else 5), result + for rel in ["browser-vm/rootfs.ext4", "browser-vm/browser-vm-rootfs-manifest.json", "bin/vmlinux", + "bin/initrd" if platform == "darwin-arm64" else "browser-vm/initrd"]: + assert (managed / rel).is_symlink() and (managed / rel).resolve() == (source / rel).resolve(), (platform, rel) + assert (managed / "bin/crosvm").exists() is platform.startswith("linux-") + if platform.startswith("linux-"): + assert not (managed / "bin/initrd").exists(), "Linux setup uses its own initrd path" + assert run(platform + " repeated setup", managed, platform=platform)["linked"] == 0 + initrd_key = "ELASTOS_BROWSER_VM_INITRAMFS" if platform == "darwin-arm64" else "ELASTOS_BROWSER_VM_INITRD" + foreign_key = "ELASTOS_BROWSER_VM_INITRD" if platform == "darwin-arm64" else "ELASTOS_BROWSER_VM_INITRAMFS" + checked = subprocess.run([str(repo / "scripts/browser-vm-artifact-preflight.sh"), "--verify-image-set"], + env={**env, "ELASTOS_BROWSER_VM_DATA_DIR": str(managed), "ELASTOS_BROWSER_VM_PLATFORM": platform, + foreign_key: str(root / "wrong-host-initrd")}, capture_output=True, text=True, timeout=15) + assert checked.returncode == 0, (platform, checked.stdout, checked.stderr) + checks.append(platform + " launcher initrd path") -mac_output="$("$repo_root/scripts/setup-source-home-browser-artifacts.sh" \ - --data-dir "$mac_managed" \ - --platform darwin-arm64 \ - --artifact-data-dir "$explicit_source")" + assert all(sha(source / rel) == digest for rel,digest in before.items()), "shared source changed" -OUTPUT="$mac_output" \ -SOURCE="$explicit_source" \ -MANAGED="$mac_managed" \ -node - <<'NODE' -const fs = require("node:fs"); -const path = require("node:path"); + source = root / "darwin-arm64" + manifest = source / "browser-vm/browser-vm-rootfs-manifest.json" + original_manifest = manifest.read_bytes() + partial = root / "partial" + manifest.unlink() + run("missing manifest creates zero links", partial, source, ok=False) + assert not partial.exists() + manifest.write_bytes(original_manifest) + mixed = root / "mixed" + (mixed / "bin").mkdir(parents=True) + (mixed / "bin/vmlinux").write_bytes(b"other-kernel") + run("mismatched existing kernel is preserved", mixed, source, ok=False) + assert (mixed / "bin/vmlinux").read_bytes() == b"other-kernel" + assert not (mixed / "browser-vm").exists() + image = source / "browser-vm/rootfs.ext4" + old = image.read_bytes() + image.write_bytes(b"changed-image") + run("changed image rejected", root / "changed", source, ok=False) + image.write_bytes(old) + run("wrong architecture rejected", root / "wrong-arch", source, platform="linux-amd64", ok=False) + broken = root / "broken" + (broken / "browser-vm").mkdir(parents=True) + (broken / "browser-vm/browser-vm-rootfs-manifest.json").symlink_to(root / "absent") + run("broken destination preserves all files", broken, source, ok=False) + assert not (broken / "bin").exists() + assert not (broken / "browser-vm/rootfs.ext4").exists() + run("missing explicit artifact store is actionable", root / "missing-explicit", root / "absent-store", ok=False) + unavailable = run("no local image permits a viewer Runtime", root / "viewer") + assert unavailable["missing"] == 4 and not unavailable["image_set_verified"] -const output = process.env.OUTPUT.trim().split(/\n/).at(-1); -const receipt = JSON.parse(output); -const source = process.env.SOURCE; -const managed = process.env.MANAGED; -if (receipt.schema !== "elastos.setup-source-home.browser-artifacts/v1") throw new Error("wrong schema"); -if (receipt.linked !== 3) throw new Error(`expected 3 Mac links, got ${receipt.linked}`); -for (const [dest, sourceRel] of [ - ["bin/vmlinux", "bin/vmlinux"], - ["browser-vm/rootfs.ext4", "browser-vm/rootfs.ext4"], - ["bin/initrd", "bin/initrd"], -]) { - const actual = fs.readlinkSync(path.join(managed, dest)); - const expected = path.join(source, sourceRel); - if (actual !== expected) throw new Error(`${dest} link mismatch: ${actual} !== ${expected}`); -} -if (fs.existsSync(path.join(managed, "bin/crosvm"))) { - throw new Error("Mac managed setup must not create a crosvm link"); -} -NODE + # Exercise the actual source-home helper installation. Older setup modified + # rootfs/initrd here, including a shared store reached through symlinks. + text = (repo / "scripts/setup-source-home.sh").read_text() + functions = text.split('echo "[setup-source-home] repo:')[0] + functions = re.sub(r'^SOURCE_HOME_BINARY_NAMES_JSON=.*\n', '', functions, flags=re.M) + definitions = root / "setup-functions.sh" + definitions.write_text(functions) + installed = source / "managed-runtimes/full-helper-install/data" + before = {rel: sha(source / rel) for rel in ["browser-vm/rootfs.ext4", "browser-vm/initrd", + "bin/vmlinux", "browser-vm/browser-vm-rootfs-manifest.json"]} + shell = 'source "$DEFINITIONS"\nROOT="$SOURCE"\nDATA_DIR="$DEST"\nPLATFORM=linux-arm64\nNODE_BIN="$NODE"\ninstall_browser_runtime_helpers\n' + p = subprocess.run(["bash", "-c", shell], env={**env, "DEFINITIONS": str(definitions), + "SOURCE": str(repo), "DEST": str(installed), "NODE": shutil.which("node")}, + capture_output=True, text=True, timeout=30) + assert p.returncode == 0, (p.stdout, p.stderr) + assert all(sha(source / rel) == digest for rel,digest in before.items()) + assert (installed / "browser-vm/browser-vm-rootfs-manifest.json").is_file() + assert not list(source.glob("**/*.before-*")) + checks.append("actual helper setup preserves shared image, initrd and receipt") -printf '%s\n' '{"schema":"elastos.setup-source-home.browser-artifacts-smoke/v1","ok":true}' +print(json.dumps({"schema": "elastos.setup-source-home.browser-artifacts-smoke/v1", "ok": True, + "checks": checks, "count": len(checks)})) +PY diff --git a/scripts/setup-source-home-browser-artifacts.sh b/scripts/setup-source-home-browser-artifacts.sh index 2b18f3cd2..de9a0f4e6 100755 --- a/scripts/setup-source-home-browser-artifacts.sh +++ b/scripts/setup-source-home-browser-artifacts.sh @@ -1,167 +1,116 @@ #!/usr/bin/env bash set -euo pipefail - -usage() { - cat >&2 <<'USAGE' -Usage: - scripts/setup-source-home-browser-artifacts.sh --data-dir PATH --platform PLATFORM - -Options: - --artifact-data-dir PATH Source ElastOS data dir containing Browser VM artifacts. - Defaults to ELASTOS_BROWSER_VM_ARTIFACT_DATA_DIR - and, for managed runtimes, the parent data dir. - -Links missing Browser VM substrate artifacts into a source-home data dir without -copying large VM images or replacing real files. This keeps managed runtimes -renewable while preserving the operator-owned top-level artifact store. -USAGE -} - -data_dir="" -platform="" -artifact_data_dir="${ELASTOS_BROWSER_VM_ARTIFACT_DATA_DIR:-}" - -while [[ $# -gt 0 ]]; do - case "$1" in - --data-dir) - data_dir="${2:-}" - shift 2 - ;; - --platform) - platform="${2:-}" - shift 2 - ;; - --artifact-data-dir) - artifact_data_dir="${2:-}" - shift 2 - ;; - --help|-h) - usage - exit 0 - ;; - *) - echo "unknown argument: $1" >&2 - usage - exit 2 - ;; - esac -done - -if [[ -z "$data_dir" || "$data_dir" != /* ]]; then - echo "--data-dir must be an absolute path" >&2 - exit 2 -fi -if [[ -z "$platform" ]]; then - echo "--platform is required" >&2 - exit 2 -fi -if [[ -n "$artifact_data_dir" && "$artifact_data_dir" != /* ]]; then - echo "--artifact-data-dir must be an absolute path" >&2 - exit 2 -fi - -candidate_dirs=() -candidate_dir_count=0 -add_candidate_dir() { - local candidate="$1" - if [[ -z "$candidate" || "$candidate" == "$data_dir" || ! -d "$candidate" ]]; then - return - fi - if (( candidate_dir_count > 0 )); then - local existing - for existing in "${candidate_dirs[@]}"; do - if [[ "$existing" == "$candidate" ]]; then - return - fi - done - fi - candidate_dirs+=("$candidate") - candidate_dir_count=$((candidate_dir_count + 1)) -} - -add_candidate_dir "$artifact_data_dir" -case "$data_dir" in - */managed-runtimes/*) - add_candidate_dir "${data_dir%%/managed-runtimes/*}" - ;; -esac - -linked=0 -skipped_existing=0 -missing=0 - -link_artifact() { - local dest_rel="$1" - shift - local dest="${data_dir}/${dest_rel}" - local source_dir source_rel source="" - - if [[ -e "$dest" && ! -L "$dest" ]]; then - skipped_existing=$((skipped_existing + 1)) - return - fi - if [[ -L "$dest" && -e "$dest" ]]; then - skipped_existing=$((skipped_existing + 1)) - return - fi - - if (( candidate_dir_count > 0 )); then - for source_dir in "${candidate_dirs[@]}"; do - for source_rel in "$@"; do - if [[ -e "${source_dir}/${source_rel}" ]]; then - source="${source_dir}/${source_rel}" - break 2 - fi - done - done - fi - - if [[ -z "$source" ]]; then - missing=$((missing + 1)) - return - fi - - mkdir -p "$(dirname "$dest")" - ln -sfn "$source" "$dest" - linked=$((linked + 1)) - echo "[setup-source-home] linked Browser VM artifact: ${dest} -> ${source}" -} - -link_artifact "bin/vmlinux" "bin/vmlinux" -link_artifact "browser-vm/rootfs.ext4" "browser-vm/rootfs.ext4" - -case "$platform" in - linux-*) - link_artifact "bin/crosvm" "bin/crosvm" - link_artifact "browser-vm/initrd" "browser-vm/initrd" - ;; - darwin-arm64) - link_artifact "bin/initrd" "bin/initrd" - ;; -esac - -candidate_dirs_text="" -if (( candidate_dir_count > 0 )); then - printf -v candidate_dirs_text '%s\n' "${candidate_dirs[@]}" -fi - -CANDIDATE_DIRS="${candidate_dirs_text}" python3 - "$platform" "$data_dir" "$linked" "$skipped_existing" "$missing" <<'PY' +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +exec python3 - "$repo_root" "$@" <<'PY' +import argparse import json import os +from pathlib import Path +import subprocess import sys -platform, data_dir, linked, skipped_existing, missing = sys.argv[1:] -candidate_dirs = [ - entry for entry in os.environ.get("CANDIDATE_DIRS", "").splitlines() if entry -] -print(json.dumps({ - "schema": "elastos.setup-source-home.browser-artifacts/v1", - "ok": True, - "platform": platform, - "data_dir": data_dir, - "candidate_dirs": candidate_dirs, - "linked": int(linked), - "skipped_existing": int(skipped_existing), - "missing": int(missing), -}, separators=(",", ":"))) +repo_root = Path(sys.argv.pop(1)) +parser = argparse.ArgumentParser(description="Link one verified Browser VM image set; preserve the source artifacts.") +parser.add_argument("--data-dir", type=Path, required=True) +parser.add_argument("--platform", choices=["darwin-arm64", "linux-arm64", "linux-amd64"], required=True) +parser.add_argument("--artifact-data-dir", type=Path, + default=os.environ.get("ELASTOS_BROWSER_VM_ARTIFACT_DATA_DIR")) +args = parser.parse_args() +for value in (args.data_dir, args.artifact_data_dir): + if value is not None and not value.is_absolute(): + parser.error("artifact and data directories must be absolute") +data = args.data_dir +candidates = [] +if args.artifact_data_dir is not None: + candidates.append(args.artifact_data_dir) +elif "managed-runtimes" in data.parts: + candidates.append(Path(*data.parts[:data.parts.index("managed-runtimes")])) + +rootfs_rel = "browser-vm/rootfs.ext4" +manifest_rel = "browser-vm/browser-vm-rootfs-manifest.json" +initrd_rel = "bin/initrd" if args.platform == "darwin-arm64" else "browser-vm/initrd" +result = {"schema": "elastos.setup-source-home.browser-artifacts/v1", "ok": True, + "platform": args.platform, "data_dir": str(data), + "candidate_dirs": [str(p) for p in candidates], "linked": 0, + "skipped_existing": 0, "missing": 0, "image_set_verified": False} + + +def finish(error=None): + if error: + result.update(ok=False, error=error) + print(json.dumps(result, separators=(",", ":"))) + sys.exit(1 if error else 0) + + +# A receipt belongs to the rootfs that selected the set. Kernel and initrd may +# use the build layout or installed layout, but every byte must match that receipt. +rootfs = data / rootfs_rel +if rootfs.exists(): + rootfs = rootfs.resolve() +else: + rootfs = next((p / rootfs_rel for p in candidates if (p / rootfs_rel).is_file()), None) +if rootfs is None: + result["missing"] = 4 + finish("The selected Browser artifact store has no image set." if args.artifact_data_dir else None) +source_vm = rootfs.parent +source_data = source_vm.parent +plan = {rootfs_rel: rootfs, manifest_rel: source_vm / "browser-vm-rootfs-manifest.json"} +for dest, alternatives in [("bin/vmlinux", [source_data / "bin/vmlinux", source_vm / "vmlinux"]), + (initrd_rel, [source_data / initrd_rel, source_vm / "initrd"])]: + plan[dest] = next((p for p in alternatives if p.is_file()), alternatives[0]) +# Existing destination files always retain ownership. An incompatible mixture +# fails before setup creates any links; it is repaired by installing a full set. +for rel in plan: + dest = data / rel + if dest.is_symlink() and not dest.exists(): + finish(f"Broken Browser artifact link requires repair: {dest}") + if dest.exists(): + plan[rel] = dest + result["skipped_existing"] += 1 + if not plan[rel].is_file(): + result["missing"] += 1 +if result["missing"]: + finish("Browser image set is incomplete. Install its rootfs, kernel, initrd and matching build manifest together.") + +env = {**os.environ, "ELASTOS_BROWSER_VM_PLATFORM": args.platform, + "ELASTOS_BROWSER_VM_DATA_DIR": str(data), "ELASTOS_BROWSER_VM_STAGED_ROOTFS": "", + "ELASTOS_BROWSER_VM_ROOTFS": str(plan[rootfs_rel]), + "ELASTOS_BROWSER_VM_ROOTFS_MANIFEST": str(plan[manifest_rel]), + "ELASTOS_BROWSER_VM_KERNEL": str(plan["bin/vmlinux"]), + ("ELASTOS_BROWSER_VM_INITRAMFS" if args.platform == "darwin-arm64" + else "ELASTOS_BROWSER_VM_INITRD"): str(plan[initrd_rel])} +try: + verified = subprocess.run([str(repo_root / "scripts/browser-vm-artifact-preflight.sh"), + "--verify-image-set"], env=env, capture_output=True, text=True, + timeout=120) + proof = json.loads(verified.stdout) + if verified.returncode or proof.get("schema") != "elastos.browser.vm-image-set/v1" or proof.get("ok") is not True: + finish("Browser image set failed verification. Rebuild or install a verified set; existing artifacts were preserved.") +except (OSError, ValueError, subprocess.SubprocessError) as exc: + finish(f"Browser image verification could not finish: {exc}") + +if args.platform.startswith("linux-"): + crosvm = source_data / "bin/crosvm" + if crosvm.is_file(): + plan["bin/crosvm"] = crosvm + elif not (data / "bin/crosvm").is_file(): + result["missing"] += 1 +# Install the receipt last so a first-time consumer cannot admit a partial set. +# This developer projection is read-only; Runtime package admission owns updates. +ordered = [rel for rel in plan if rel != manifest_rel] + [manifest_rel] +created = [] +try: + for rel in ordered: + dest = data / rel + if dest.exists(): + continue + dest.parent.mkdir(parents=True, exist_ok=True) + dest.symlink_to(plan[rel].resolve()) + created.append(dest) +except OSError as exc: + for dest in reversed(created): + dest.unlink() + finish(f"Browser artifact links could not be installed: {exc}") +result.update(linked=len(created), image_set_verified=True) +finish() PY diff --git a/scripts/setup-source-home-browser-rootfs-refresh-smoke.sh b/scripts/setup-source-home-browser-rootfs-refresh-smoke.sh deleted file mode 100755 index b7e94d202..000000000 --- a/scripts/setup-source-home-browser-rootfs-refresh-smoke.sh +++ /dev/null @@ -1,195 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -tmp_dir="$(mktemp -d)" -trap 'rm -rf "$tmp_dir"' EXIT - -find_e2fs_tool() { - local name="$1" - local candidate - if command -v "$name" >/dev/null 2>&1; then - command -v "$name" - return - fi - for candidate in \ - "/opt/homebrew/opt/e2fsprogs/sbin/$name" \ - "/usr/local/opt/e2fsprogs/sbin/$name" \ - "/usr/sbin/$name" \ - "/sbin/$name" - do - if [[ -x "$candidate" ]]; then - printf '%s\n' "$candidate" - return - fi - done - echo "$name not found" >&2 - exit 2 -} - -mke2fs="$(find_e2fs_tool mke2fs)" -debugfs="$(find_e2fs_tool debugfs)" -rootfs="$tmp_dir/rootfs.ext4" -old_relay="$tmp_dir/old-relay" -old_bridge="$tmp_dir/old-bridge" -current_relay="$tmp_dir/current-relay" -current_bridge="$tmp_dir/current-bridge" - -node - "$old_relay" "$old_bridge" "$current_relay" "$current_bridge" <<'NODE' -const fs = require("node:fs"); - -function elf(marker) { - const header = Buffer.alloc(64); - header[0] = 0x7f; - header[1] = 0x45; - header[2] = 0x4c; - header[3] = 0x46; - header[4] = 2; - header[5] = 1; - header[6] = 1; - header.writeUInt16LE(2, 16); - header.writeUInt16LE(183, 18); - return Buffer.concat([header, Buffer.from(`\n${marker}\n`)]); -} - -const [oldRelay, oldBridge, currentRelay, currentBridge] = process.argv.slice(2); -fs.writeFileSync(oldRelay, elf("old-runtime-relay"), { mode: 0o755 }); -fs.writeFileSync(oldBridge, elf("old-guest-control-bridge"), { mode: 0o755 }); -fs.writeFileSync(currentRelay, elf("current-runtime-relay"), { mode: 0o755 }); -fs.writeFileSync(currentBridge, elf("current-guest-control-bridge"), { mode: 0o755 }); -NODE - -truncate -s 16M "$rootfs" -"$mke2fs" -q -t ext4 -F "$rootfs" -for directory in /opt /opt/elastos /opt/elastos/bin; do - "$debugfs" -w -R "mkdir $directory" "$rootfs" >/dev/null 2>&1 -done -"$debugfs" -w -R "write $old_relay /opt/elastos/bin/browser-vm-runtime-relay" \ - "$rootfs" >/dev/null 2>&1 -"$debugfs" -w -R "write $old_bridge /opt/elastos/bin/browser-vm-guest-control-bridge" \ - "$rootfs" >/dev/null 2>&1 - -functions_file="$tmp_dir/setup-source-home-functions.sh" -awk ' - /^echo "\[setup-source-home\] repo:/ { exit } - /^SOURCE_HOME_BINARY_NAMES_JSON=/ { next } - { print } -' "$repo_root/scripts/setup-source-home.sh" >"$functions_file" -source "$functions_file" - -resolve_root="$tmp_dir/resolve-root" -rust_target="aarch64-unknown-linux-musl" -mkdir -p \ - "$resolve_root/elastos/tools/browser-vm-runtime-relay" \ - "$resolve_root/elastos/tools/browser-vm-guest-control-bridge" \ - "$resolve_root/elastos/target/$rust_target/release" \ - "$resolve_root/relative-target/$rust_target/release" \ - "$resolve_root/elastos/tools/browser-vm-runtime-relay/target/$rust_target/release" \ - "$resolve_root/elastos/tools/browser-vm-guest-control-bridge/target/$rust_target/release" -printf '[package]\nname = "browser-vm-runtime-relay"\nversion = "0.0.0"\n' \ - >"$resolve_root/elastos/tools/browser-vm-runtime-relay/Cargo.toml" -printf '[package]\nname = "browser-vm-guest-control-bridge"\nversion = "0.0.0"\n' \ - >"$resolve_root/elastos/tools/browser-vm-guest-control-bridge/Cargo.toml" -cp "$current_relay" "$resolve_root/elastos/target/$rust_target/release/browser-vm-runtime-relay" -cp "$current_bridge" "$resolve_root/elastos/target/$rust_target/release/browser-vm-guest-control-bridge" -cp "$old_relay" \ - "$resolve_root/elastos/tools/browser-vm-runtime-relay/target/$rust_target/release/browser-vm-runtime-relay" -cp "$old_bridge" \ - "$resolve_root/elastos/tools/browser-vm-guest-control-bridge/target/$rust_target/release/browser-vm-guest-control-bridge" -cp "$current_relay" "$resolve_root/relative-target/$rust_target/release/browser-vm-runtime-relay" -cp "$current_bridge" "$resolve_root/relative-target/$rust_target/release/browser-vm-guest-control-bridge" - -ROOT="$resolve_root" -unset ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN -unset ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN -unset CARGO_TARGET_DIR -default_runtime_relay="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-runtime-relay" \ - "ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN" \ - "browser-vm-runtime-relay" \ - "browser-vm-runtime-relay" \ - "linux-arm64")" -default_guest_bridge="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-guest-control-bridge" \ - "ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN" \ - "browser-vm-guest-control-bridge" \ - "browser-vm-guest-control-bridge" \ - "linux-arm64")" -[[ "$default_runtime_relay" == "$resolve_root/elastos/target/$rust_target/release/browser-vm-runtime-relay" ]] -[[ "$default_guest_bridge" == "$resolve_root/elastos/target/$rust_target/release/browser-vm-guest-control-bridge" ]] - -workspace_manifest="$resolve_root/elastos/standalone-workspace/Cargo.toml" -mkdir -p "$(dirname "$workspace_manifest")" -printf '[workspace]\nmembers = []\n' >"$workspace_manifest" -[[ "$(cargo_target_root_for_manifest "$workspace_manifest")" == "$resolve_root/elastos/standalone-workspace/target" ]] - -abs_target="$tmp_dir/absolute-target" -mkdir -p "$abs_target/$rust_target/release" -cp "$current_relay" "$abs_target/$rust_target/release/browser-vm-runtime-relay" -cp "$current_bridge" "$abs_target/$rust_target/release/browser-vm-guest-control-bridge" -export CARGO_TARGET_DIR="$abs_target" -abs_runtime_relay="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-runtime-relay" \ - "ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN" \ - "browser-vm-runtime-relay" \ - "browser-vm-runtime-relay" \ - "linux-arm64")" -abs_guest_bridge="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-guest-control-bridge" \ - "ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN" \ - "browser-vm-guest-control-bridge" \ - "browser-vm-guest-control-bridge" \ - "linux-arm64")" -[[ "$abs_runtime_relay" == "$abs_target/$rust_target/release/browser-vm-runtime-relay" ]] -[[ "$abs_guest_bridge" == "$abs_target/$rust_target/release/browser-vm-guest-control-bridge" ]] - -export CARGO_TARGET_DIR="relative-target" -relative_runtime_relay="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-runtime-relay" \ - "ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN" \ - "browser-vm-runtime-relay" \ - "browser-vm-runtime-relay" \ - "linux-arm64")" -relative_guest_bridge="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-guest-control-bridge" \ - "ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN" \ - "browser-vm-guest-control-bridge" \ - "browser-vm-guest-control-bridge" \ - "linux-arm64")" -[[ "$relative_runtime_relay" == "$resolve_root/relative-target/$rust_target/release/browser-vm-runtime-relay" ]] -[[ "$relative_guest_bridge" == "$resolve_root/relative-target/$rust_target/release/browser-vm-guest-control-bridge" ]] -[[ "$relative_runtime_relay" != "$resolve_root/elastos/tools/browser-vm-runtime-relay/target/$rust_target/release/browser-vm-runtime-relay" ]] -[[ "$relative_guest_bridge" != "$resolve_root/elastos/tools/browser-vm-guest-control-bridge/target/$rust_target/release/browser-vm-guest-control-bridge" ]] - -ROOT="$repo_root" -DATA_DIR="$tmp_dir/data" -BROWSER_VM_ROOTFS_BACKUP="" -export ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN="$current_relay" -export ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN="$current_bridge" -unset CARGO_TARGET_DIR - -refresh_browser_vm_native_helpers "$rootfs" "$debugfs" "linux-arm64" - -actual_relay="$tmp_dir/actual-relay" -actual_bridge="$tmp_dir/actual-bridge" -"$debugfs" -R "cat /opt/elastos/bin/browser-vm-runtime-relay" \ - "$rootfs" >"$actual_relay" 2>/dev/null -"$debugfs" -R "cat /opt/elastos/bin/browser-vm-guest-control-bridge" \ - "$rootfs" >"$actual_bridge" 2>/dev/null -cmp "$current_relay" "$actual_relay" -cmp "$current_bridge" "$actual_bridge" - -if [[ -z "$BROWSER_VM_ROOTFS_BACKUP" || ! -f "$BROWSER_VM_ROOTFS_BACKUP" ]]; then - echo "native helper refresh did not preserve one rollback rootfs" >&2 - exit 1 -fi -backup_relay="$tmp_dir/backup-relay" -backup_bridge="$tmp_dir/backup-bridge" -"$debugfs" -R "cat /opt/elastos/bin/browser-vm-runtime-relay" \ - "$BROWSER_VM_ROOTFS_BACKUP" >"$backup_relay" 2>/dev/null -"$debugfs" -R "cat /opt/elastos/bin/browser-vm-guest-control-bridge" \ - "$BROWSER_VM_ROOTFS_BACKUP" >"$backup_bridge" 2>/dev/null -cmp "$old_relay" "$backup_relay" -cmp "$old_bridge" "$backup_bridge" - -printf '%s\n' '{"schema":"elastos.setup-source-home.browser-rootfs-refresh-smoke/v1","ok":true}' diff --git a/scripts/setup-source-home-collaboration-config-smoke.sh b/scripts/setup-source-home-collaboration-config-smoke.sh index d8204108f..eaa64fb82 100755 --- a/scripts/setup-source-home-collaboration-config-smoke.sh +++ b/scripts/setup-source-home-collaboration-config-smoke.sh @@ -86,9 +86,8 @@ set -euo pipefail functions_file="$1" repo_root="$2" set -- -source "$functions_file" -ROOT="$repo_root" -browser_vm_backup_retention +eval "$(sed -n '/^backup_retention() {/,/^}/p' "$repo_root/scripts/browser-vm-target-refresh.sh")" +backup_retention EOF chmod 755 "$retention_runner" @@ -153,15 +152,15 @@ chmod 600 "$valid_a" "$valid_b" "$invalid" chmod 644 "$wrong_mode" if [[ "$(run_retention)" != "1" ]]; then - echo "setup-source-home did not keep exactly one Browser VM backup by default" >&2 + echo "Browser host refresh did not keep exactly one helper backup by default" >&2 exit 1 fi if [[ "$(env PATH="$fake_bin_dir:$PATH" ELASTOS_BROWSER_VM_BACKUP_RETENTION=1 "$retention_runner" "$functions_file" "$repo_root")" != "1" ]]; then - echo "setup-source-home did not accept explicit one-backup retention" >&2 + echo "Browser host refresh did not accept explicit one-backup retention" >&2 exit 1 fi if env PATH="$fake_bin_dir:$PATH" ELASTOS_BROWSER_VM_BACKUP_RETENTION=2 "$retention_runner" "$functions_file" "$repo_root" >/dev/null 2>"$tmp_dir/retention.err"; then - echo "setup-source-home accepted Browser VM backup retention above one" >&2 + echo "Browser host refresh accepted backup retention above one" >&2 exit 1 fi grep -q "ELASTOS_BROWSER_VM_BACKUP_RETENTION must be 1" "$tmp_dir/retention.err" @@ -325,19 +324,16 @@ import sys source = pathlib.Path(sys.argv[1]).read_text(encoding="utf-8") gate = 'require_minimum_free_space "${DATA_DIR}"' source_gate = 'require_minimum_free_space "${ROOT}"' -retention_gate = 'browser_vm_backup_retention >/dev/null' config_only = 'if [[ "${SETUP_SOURCE_HOME_CONFIG_ONLY:-0}" == "1" ]]; then' config_toml = 'CONFIG_TOML="${DATA_DIR}/config.toml"' if gate not in source: raise SystemExit("setup-source-home is missing the free-space gate") -if not (source.index(config_only) < source.index(retention_gate) < source.index(source_gate) < source.index(gate) < source.index(config_toml)): +if not (source.index(config_only) < source.index(source_gate) < source.index(gate) < source.index(config_toml)): raise SystemExit("setup-source-home free-space gate is not positioned between config-only and build setup") refresh = pathlib.Path(sys.argv[1]).with_name("browser-vm-target-refresh.sh").read_text(encoding="utf-8") -setup_retention = source.split("browser_vm_backup_retention() {", 1)[1].split("\n}", 1)[0] -refresh_retention = refresh.split("backup_retention() {", 1)[1].split("\n}", 1)[0] -if setup_retention != refresh_retention: - raise SystemExit("source setup and Browser refresh disagree on backup retention") +if "browser_vm_backup_retention" in source or "refresh_browser_vm_rootfs_files" in source: + raise SystemExit("source setup must consume image sets without guest mutation or rollback copies") if refresh.index("backup_retention >/dev/null") > refresh.index('install_with_backup "$selkies_source"'): raise SystemExit("Browser refresh validates retention after starting installation") PY diff --git a/scripts/setup-source-home.sh b/scripts/setup-source-home.sh index 75acaa172..06cfea901 100755 --- a/scripts/setup-source-home.sh +++ b/scripts/setup-source-home.sh @@ -14,14 +14,9 @@ resolved from HOME/XDG_DATA_HOME. Configure tool paths with: ELASTOS_CARGO_BIN ELASTOS_NODE_BIN - ELASTOS_DEBUGFS_BIN ELASTOS_COLLABORATION_STARTUP_MODE (configured|isolated) ELASTOS_COLLABORATION_STARTUP_CONFIG_INPUT - ELASTOS_BROWSER_NATIVE_PROXY_BIN - ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN - ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN ELASTOS_BROWSER_VM_ARTIFACT_DATA_DIR - ELASTOS_BROWSER_VM_BACKUP_RETENTION (must be 1) To target a non-default runtime root, set HOME or XDG_DATA_HOME before running. ELASTOS_DATA_DIR is intentionally not accepted as a gateway data-root override. @@ -163,28 +158,6 @@ find_node() { exit 1 } -find_debugfs() { - if [[ -n "${ELASTOS_DEBUGFS_BIN:-}" && -x "${ELASTOS_DEBUGFS_BIN}" ]]; then - printf '%s\n' "${ELASTOS_DEBUGFS_BIN}" - return - fi - if command -v debugfs >/dev/null 2>&1; then - command -v debugfs - return - fi - for candidate in \ - /opt/homebrew/opt/e2fsprogs/sbin/debugfs \ - /usr/local/opt/e2fsprogs/sbin/debugfs \ - /usr/sbin/debugfs \ - /sbin/debugfs - do - if [[ -x "$candidate" ]]; then - printf '%s\n' "$candidate" - return - fi - done -} - find_turnserver() { if [[ -n "${ELASTOS_BROWSER_VM_TURNSERVER_BIN:-}" && -x "${ELASTOS_BROWSER_VM_TURNSERVER_BIN}" ]]; then printf '%s\n' "${ELASTOS_BROWSER_VM_TURNSERVER_BIN}" @@ -238,193 +211,6 @@ find_vz_turn_program() { return 1 } -browser_vm_target_platform() { - case "$(uname -s)-$(uname -m)" in - Linux-x86_64) printf '%s\n' "linux-amd64" ;; - Linux-aarch64|Linux-arm64|Darwin-arm64) printf '%s\n' "linux-arm64" ;; - *) return 1 ;; - esac -} - -resolve_browser_vm_native_proxy_source() { - local target_platform="$1" - local candidate - - if [[ -n "${ELASTOS_BROWSER_NATIVE_PROXY_BIN:-}" ]]; then - if [[ ! -x "${ELASTOS_BROWSER_NATIVE_PROXY_BIN}" ]]; then - echo "ELASTOS_BROWSER_NATIVE_PROXY_BIN is not executable: ${ELASTOS_BROWSER_NATIVE_PROXY_BIN}" >&2 - return 2 - fi - printf '%s\n' "${ELASTOS_BROWSER_NATIVE_PROXY_BIN}" - return - fi - - case "$target_platform" in - linux-arm64) - for candidate in \ - "${ROOT}/elastos/tools/browser-native-proxy-engine/target/aarch64-unknown-linux-musl/release/browser-native-proxy-engine" \ - "${ROOT}/elastos/tools/browser-native-proxy-engine/target/aarch64-unknown-linux-gnu/release/browser-native-proxy-engine" - do - if [[ -x "$candidate" ]]; then - printf '%s\n' "$candidate" - return - fi - done - ;; - linux-amd64) - for candidate in \ - "${ROOT}/elastos/tools/browser-native-proxy-engine/target/x86_64-unknown-linux-musl/release/browser-native-proxy-engine" \ - "${ROOT}/elastos/tools/browser-native-proxy-engine/target/x86_64-unknown-linux-gnu/release/browser-native-proxy-engine" - do - if [[ -x "$candidate" ]]; then - printf '%s\n' "$candidate" - return - fi - done - ;; - esac - return 1 -} - -browser_vm_guest_rust_target() { - case "$1" in - linux-arm64) printf '%s\n' "aarch64-unknown-linux-musl" ;; - linux-amd64) printf '%s\n' "x86_64-unknown-linux-musl" ;; - *) return 1 ;; - esac -} - -build_browser_vm_guest_helper() { - local manifest="$1" - local rust_target="$2" - local linker_env="" - local linker="" - - case "$rust_target" in - aarch64-unknown-linux-musl) - linker_env="CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER" - ;; - x86_64-unknown-linux-musl) - linker_env="CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER" - ;; - *) - echo "unsupported Browser VM guest Rust target: $rust_target" >&2 - exit 1 - ;; - esac - - if [[ "$(uname -s)" == "Darwin" && -z "${!linker_env:-}" ]]; then - local rustc_bin - local rust_sysroot - rustc_bin="$(command -v rustc || true)" - if [[ -z "$rustc_bin" ]]; then - echo "rustc not found; Browser VM guest helper cross-build cannot locate rust-lld" >&2 - exit 1 - fi - rust_sysroot="$("$rustc_bin" --print sysroot)" - linker="$(find "$rust_sysroot" -type f -name rust-lld -perm -111 -print -quit 2>/dev/null || true)" - if [[ -z "$linker" ]]; then - echo "rust-lld not found in $rust_sysroot; Browser VM guest helper cross-build is unavailable" >&2 - exit 1 - fi - env "$linker_env=$linker" "$CARGO_BIN" build --quiet \ - --manifest-path "$manifest" \ - --target "$rust_target" \ - --release - return - fi - - "$CARGO_BIN" build --quiet \ - --manifest-path "$manifest" \ - --target "$rust_target" \ - --release -} - -build_browser_vm_guest_helpers() { - local target_platform - local rust_target - target_platform="$(browser_vm_target_platform || true)" - rust_target="$(browser_vm_guest_rust_target "$target_platform" || true)" - if [[ -z "$rust_target" ]]; then - echo "Browser VM guest helper build is unsupported on ${target_platform:-this platform}" >&2 - exit 1 - fi - ensure_rust_target_installed "$rust_target" - - echo "[setup-source-home] build Browser VM guest relay helpers" - build_browser_vm_guest_helper \ - "${ROOT}/elastos/tools/browser-vm-runtime-relay/Cargo.toml" \ - "$rust_target" - build_browser_vm_guest_helper \ - "${ROOT}/elastos/tools/browser-vm-guest-control-bridge/Cargo.toml" \ - "$rust_target" -} - -resolve_browser_vm_guest_helper_source() { - local label="$1" - local env_name="$2" - local crate_name="$3" - local binary_name="$4" - local target_platform="$5" - local explicit="${!env_name:-}" - local rust_target - local candidate - local manifest - - if [[ -n "$explicit" ]]; then - if [[ ! -x "$explicit" ]]; then - echo "$env_name is not executable: $explicit" >&2 - return 2 - fi - printf '%s\n' "$explicit" - return - fi - - rust_target="$(browser_vm_guest_rust_target "$target_platform" || true)" - if [[ -z "$rust_target" ]]; then - return 1 - fi - manifest="${ROOT}/elastos/tools/${crate_name}/Cargo.toml" - candidate="$(cargo_target_root_for_manifest "$manifest")/${rust_target}/release/${binary_name}" - if [[ ! -x "$candidate" ]]; then - echo "$label source build is missing: $candidate" >&2 - return 1 - fi - printf '%s\n' "$candidate" -} - -validate_linux_guest_binary() { - local label="$1" - local path="$2" - local target_platform="$3" - python3 - "$label" "$path" "$target_platform" <<'PY' -import pathlib -import struct -import sys - -label, path, target_platform = sys.argv[1:] -data = pathlib.Path(path).read_bytes()[:20] -if len(data) < 20 or data[:4] != b"\x7fELF": - raise SystemExit(f"{label} must be a Linux ELF guest binary for {target_platform}") -if data[5] == 1: - endian = "<" -elif data[5] == 2: - endian = ">" -else: - raise SystemExit(f"{label} has an invalid ELF data encoding") -machine = struct.unpack(endian + "H", data[18:20])[0] -expected = { - "linux-amd64": 62, - "linux-arm64": 183, -}[target_platform] -if machine != expected: - names = {62: "x86_64", 183: "aarch64"} - got = names.get(machine, f"e_machine={machine}") - want = names[expected] - raise SystemExit(f"{label} must be {want} for {target_platform}, got {got}") -PY -} - configure_rust_toolchain_env() { local cargo_bin="$1" local cargo_home_guess @@ -445,65 +231,6 @@ configure_rust_toolchain_env() { fi } -clone_or_copy_file() { - local source="$1" - local dest="$2" - cp -c "$source" "$dest" 2>/dev/null || - cp --reflink=auto -p "$source" "$dest" 2>/dev/null || - cp -p "$source" "$dest" - touch "$dest" -} - -browser_vm_backup_retention() { - local retention="${ELASTOS_BROWSER_VM_BACKUP_RETENTION:-1}" - if [[ "$retention" != "1" ]]; then - echo "ELASTOS_BROWSER_VM_BACKUP_RETENTION must be 1" >&2 - exit 2 - fi - printf '%s\n' "$retention" -} - -file_mtime() { - case "$(uname -s)" in - Darwin) stat -f '%m' "$1" ;; - Linux) stat -c '%Y' "$1" ;; - *) return 1 ;; - esac -} - -prune_file_backups() { - local source="$1" - local keep="$2" - local candidate - local oldest_index - local oldest_mtime - local candidate_mtime - local index - local backups=() - - for candidate in "${source}".before-*; do - if [[ -f "$candidate" && ! -L "$candidate" ]]; then - backups+=("$candidate") - fi - done - while (( ${#backups[@]} > keep )); do - oldest_index=0 - oldest_mtime="$(file_mtime "${backups[0]}")" - for (( index = 1; index < ${#backups[@]}; index += 1 )); do - candidate_mtime="$(file_mtime "${backups[$index]}")" - if (( candidate_mtime < oldest_mtime )); then - oldest_index="$index" - oldest_mtime="$candidate_mtime" - fi - done - candidate="${backups[$oldest_index]}" - rm -f -- "$candidate" - echo "[setup-source-home] pruned stale Browser VM backup: ${candidate}" - unset "backups[$oldest_index]" - backups=("${backups[@]}") - done -} - require_minimum_free_space() { local path="$1" # A full cold setup measures roughly 10 GiB across the workspace release @@ -541,7 +268,6 @@ DATA_DIR="${DEFAULT_DATA_DIR}" PLATFORM="$(detect_platform)" CARGO_BIN="$(find_cargo)" NODE_BIN="$(find_node)" -BROWSER_VM_ROOTFS_BACKUP="" SOURCE_HOME_KUBO_INSTALLED="0" configure_rust_toolchain_env "$CARGO_BIN" export PATH="$(dirname "$CARGO_BIN"):$(dirname "$NODE_BIN"):${PATH}" @@ -858,514 +584,6 @@ sign_browser_vz_supervisor() { "${ROOT}/scripts/dev/sign-elastos-vz/sign.sh" "$binary" } -refresh_browser_vm_rootfs_file() { - local rootfs="$1" - local source="$2" - local guest_path="$3" - local mode="$4" - local label="$5" - local debugfs="$6" - local current_copy - local staged_source - local updated_copy - local commands_file - current_copy="$(mktemp)" - staged_source="$(mktemp)" - updated_copy="$(mktemp)" - commands_file="$(mktemp)" - cp "$source" "$staged_source" - - if "$debugfs" -R "cat ${guest_path}" "$rootfs" > "$current_copy" 2>/dev/null && - cmp -s "$source" "$current_copy"; then - rm -f "$current_copy" "$staged_source" "$updated_copy" "$commands_file" - return - fi - - if [[ -z "$BROWSER_VM_ROOTFS_BACKUP" ]]; then - BROWSER_VM_ROOTFS_BACKUP="${rootfs}.before-${label}-$(date -u +%Y%m%dT%H%M%SZ)" - clone_or_copy_file "$rootfs" "$BROWSER_VM_ROOTFS_BACKUP" - fi - - cat > "$commands_file" </dev/null - "$debugfs" -R "cat ${guest_path}" "$rootfs" > "$updated_copy" 2>/dev/null - if ! cmp -s "$source" "$updated_copy"; then - echo "Browser VM rootfs ${label} refresh did not verify; backup kept at ${BROWSER_VM_ROOTFS_BACKUP}" >&2 - exit 1 - fi - rm -f "$current_copy" "$staged_source" "$updated_copy" "$commands_file" - echo "[setup-source-home] refreshed Browser VM rootfs ${label}: ${rootfs}" -} - -extract_browser_vm_selkies_start() { - local target="$1" - python3 - "${ROOT}/scripts/build/stage-browser-vm-target.sh" "$target" <<'PY' -import pathlib -import sys - -stage_script = pathlib.Path(sys.argv[1]) -target = pathlib.Path(sys.argv[2]) -text = stage_script.read_text() -start_marker = 'cat > "$staging_dir/opt/elastos/bin/browser-vm-selkies-start" <<\'SH\'\n' -end_marker = '\nSH\nchmod 755 "$staging_dir/opt/elastos/bin/browser-vm-selkies-start"' -try: - start = text.index(start_marker) + len(start_marker) - end = text.index(end_marker, start) -except ValueError as exc: - raise SystemExit(f"could not extract browser-vm-selkies-start from stage script: {exc}") -target.write_text(text[start:end]) -PY -} - -extract_browser_vm_init() { - local target="$1" - python3 - "${ROOT}/scripts/build/stage-browser-vm-target.sh" "$target" <<'PY' -import pathlib -import sys - -stage_script = pathlib.Path(sys.argv[1]) -target = pathlib.Path(sys.argv[2]) -text = stage_script.read_text() -start_marker = 'cat > "$staging_dir/opt/elastos/bin/browser-vm-init" <<\'SH\'\n' -end_marker = '\nSH\nchmod 755 "$staging_dir/opt/elastos/bin/browser-vm-init"' -try: - start = text.index(start_marker) + len(start_marker) - end = text.index(end_marker, start) -except ValueError as exc: - raise SystemExit(f"could not extract browser-vm-init from stage script: {exc}") -target.write_text(text[start:end]) -PY -} - -patch_browser_vm_selkies_app_source() { - local target="$1" - python3 - "$target" <<'PY' -from pathlib import Path -import re -import sys - -path = Path(sys.argv[1]) -text = path.read_text() -marker = ' self.webrtcbin.set_property("latency", 0)\n' -relay_patch = ''' elastos_ice_transport_policy = os.environ.get("ELASTOS_BROWSER_VM_ICE_TRANSPORT_POLICY", "").strip().lower() - if elastos_ice_transport_policy: - if elastos_ice_transport_policy not in ("all", "relay"): - raise GSTWebRTCAppError("ELASTOS_BROWSER_VM_ICE_TRANSPORT_POLICY must be all or relay") - try: - policy_value = getattr(GstWebRTC.WebRTCICETransportPolicy, elastos_ice_transport_policy.upper()) - except AttributeError: - policy_value = elastos_ice_transport_policy - self.webrtcbin.set_property("ice-transport-policy", policy_value) - logger.info("using ICE transport policy: %s", elastos_ice_transport_policy) -''' -turn_marker = ''' if self.turn_servers: - for i, turn_server in enumerate(self.turn_servers): - logger.info("updating TURN server") - if i == 0: - self.webrtcbin.set_property("turn-server", turn_server) - else: - self.webrtcbin.emit("add-turn-server", turn_server) -''' -turn_relay_patch = ''' if elastos_ice_transport_policy: - self.webrtcbin.set_property("ice-transport-policy", policy_value) - logger.info("confirmed ICE transport policy after TURN setup: %s", elastos_ice_transport_policy) -''' -if "elastos_ice_transport_policy" not in text: - if marker not in text: - raise SystemExit("Selkies relay policy patch target not found") - text = text.replace(marker, marker + relay_patch, 1) -if "confirmed ICE transport policy after TURN setup" not in text: - if turn_marker not in text: - raise SystemExit("Selkies TURN relay policy patch target not found") - text = text.replace(turn_marker, turn_marker + turn_relay_patch, 1) -if "confirmed ICE transport policy after TURN setup" not in text: - raise SystemExit("Selkies relay policy patch incomplete") -ice_log_marker = ' logger.debug("received ICE candidate: %d %s", mlineindex, candidate)\n' -ice_log_patch = ' logger.info("emitting ICE candidate: %d %s", mlineindex, candidate)\n' -if ice_log_patch not in text: - if ice_log_marker not in text: - raise SystemExit("Selkies ICE candidate log patch target not found") - text = text.replace(ice_log_marker, ice_log_patch, 1) -rtp_extensions_marker = ''' rtp_id_iteration = 0 - return_result = True -''' -previous_rtp_extensions_patch = ''' # Selkies 1.6.1 RTP header extensions are unstable in the ElastOS - # combined audio/video product session. Runtime TURN plus NACK/FIR keeps - # the media path reliable without mutating RTP extension caps here. - return True -''' -if previous_rtp_extensions_patch in text: - text = text.replace(previous_rtp_extensions_patch, rtp_extensions_marker, 1) -elif rtp_extensions_marker not in text: - raise SystemExit("Selkies RTP extension patch removal target not found") -opusenc_member_marker = " self.rtpgccbwe = None\n" -opusenc_member_patch = " self.rtpgccbwe = None\n self.opusenc = None\n" -if opusenc_member_patch not in text: - if opusenc_member_marker not in text: - raise SystemExit("Selkies opusenc member patch target not found") - text = text.replace(opusenc_member_marker, opusenc_member_patch, 1) -video_only_start = """ if audio_only: - self.build_audio_pipeline() - else: - self.build_video_pipeline() -""" -audio_video_start = """ if audio_only: - self.build_audio_pipeline() - else: - self.build_video_pipeline() - self.build_audio_pipeline() -""" -audio_video_pattern = ( - r" if audio_only:\n" - r" self\.build_audio_pipeline\(\)\n" - r" else:\n" - r" self\.build_video_pipeline\(\)\n" - r"(?: self\.build_audio_pipeline\(\)\n)+" -) -text, pipeline_replacements = re.subn(audio_video_pattern, video_only_start, text, count=1) -if pipeline_replacements == 0 and video_only_start not in text: - raise SystemExit("Selkies video/audio split pipeline patch target not found") -audio_extension_block = """ # Add WebRTC RTP extensions - extensions_return = self.rtp_add_extensions(rtpopuspay, audio=True) - if not extensions_return: - logger.warning("WebRTC RTP extension configuration failed with audio, this may lead to suboptimal performance") -""" -previous_audio_extension_patch = """ # Selkies 1.6.1 can corrupt combined audio/video SDP when audio RTP - # header extensions are attached. Keep the product session audio track - # simple and let WebRTC/NACK handle media recovery through Runtime TURN. - extensions_return = True -""" -audio_extension_patch = """ # Selkies 1.6.1 audio RTP header extensions are fragile in the split - # product audio peer. Keep the audio track - # simple and let WebRTC/NACK handle media recovery through Runtime TURN. - extensions_return = True -""" -if audio_extension_block in text: - text = text.replace(audio_extension_block, audio_extension_patch, 1) -elif previous_audio_extension_patch in text: - text = text.replace(previous_audio_extension_patch, audio_extension_patch, 1) -elif "Selkies 1.6.1 audio RTP header extensions are fragile" not in text: - raise SystemExit("Selkies audio RTP extension patch target not found") -pulsesrc_named = ' pulsesrc = Gst.ElementFactory.make("pulsesrc", "pulsesrc")\n' -pulsesrc_unnamed = ' pulsesrc = Gst.ElementFactory.make("pulsesrc")\n' -pulsesrc_device = ' pulsesrc.set_property("device", "auto_null.monitor")\n' -if pulsesrc_named in text: - text = text.replace(pulsesrc_named, pulsesrc_unnamed, 1) -elif pulsesrc_unnamed not in text: - raise SystemExit("Selkies pulsesrc patch target not found") -text = re.sub(r'^[ \t]*pulsesrc\.set_property\("device", .*\)\n', '', text, flags=re.MULTILINE) -text = text.replace(pulsesrc_unnamed, pulsesrc_unnamed + pulsesrc_device, 1) -opusenc_named = ' opusenc = Gst.ElementFactory.make("opusenc", "opusenc")\n' -opusenc_unnamed = ' opusenc = Gst.ElementFactory.make("opusenc")\n' -if opusenc_named in text: - text = text.replace(opusenc_named, opusenc_unnamed, 1) -elif opusenc_unnamed not in text: - raise SystemExit("Selkies opusenc patch target not found") -opusenc_bitrate_marker = ' opusenc.set_property("bitrate", self.audio_bitrate)\n' -opusenc_bitrate_patch = ' opusenc.set_property("bitrate", self.audio_bitrate)\n self.opusenc = opusenc\n' -if opusenc_bitrate_patch not in text: - if opusenc_bitrate_marker not in text: - raise SystemExit("Selkies opusenc reference patch target not found") - text = text.replace(opusenc_bitrate_marker, opusenc_bitrate_patch, 1) -opusenc_update_block = """ element = Gst.Bin.get_by_name(self.pipeline, "opusenc") - element.set_property("bitrate", bitrate) -""" -opusenc_update_patch = """ element = self.opusenc or Gst.Bin.get_by_name(self.pipeline, "opusenc") - if element is None: - raise GSTWebRTCAppError("Audio encoder is unavailable") - element.set_property("bitrate", bitrate) -""" -if opusenc_update_block in text: - text = text.replace(opusenc_update_block, opusenc_update_patch, 1) -elif opusenc_update_patch not in text: - raise SystemExit("Selkies audio bitrate update patch target not found") -audio_queue_named = ' rtpopuspay_queue = Gst.ElementFactory.make("queue", "rtpopuspay_queue")\n' -audio_queue_unnamed = ' rtpopuspay_queue = Gst.ElementFactory.make("queue")\n' -if audio_queue_named in text: - text = text.replace(audio_queue_named, audio_queue_unnamed, 1) -elif audio_queue_unnamed not in text: - raise SystemExit("Selkies audio queue patch target not found") -audio_add_block = """ # Add all elements to the pipeline. - pipeline_elements = [pulsesrc, pulsesrc_capsfilter, opusenc, rtpopuspay, rtpopuspay_queue, rtpopuspay_capsfilter] - - for pipeline_element in pipeline_elements: - self.pipeline.add(pipeline_element) -""" -audio_add_strict_block = """ # Add all elements to the pipeline. - pipeline_elements = [pulsesrc, pulsesrc_capsfilter, opusenc, rtpopuspay, rtpopuspay_queue, rtpopuspay_capsfilter] - - for pipeline_element in pipeline_elements: - if pipeline_element is None: - raise GSTWebRTCAppError("Audio pipeline element is unavailable") - if not self.pipeline.add(pipeline_element): - raise GSTWebRTCAppError("Failed to add {} to pipeline".format(pipeline_element.get_name())) -""" -audio_add_patch = """ # Add all elements to the pipeline. - pipeline_elements = [pulsesrc, pulsesrc_capsfilter, opusenc, rtpopuspay, rtpopuspay_queue, rtpopuspay_capsfilter] - - for pipeline_element in pipeline_elements: - if pipeline_element is None: - raise GSTWebRTCAppError("Audio pipeline element is unavailable") - self.pipeline.add(pipeline_element) -""" -if audio_add_strict_block in text: - text = text.replace(audio_add_strict_block, audio_add_patch, 1) -elif audio_add_block in text: - text = text.replace(audio_add_block, audio_add_patch, 1) -elif audio_add_patch not in text: - raise SystemExit("Selkies audio pipeline add patch target not found") -audio_offer_marker = ' logger.info("{} pipeline started".format("audio" if audio_only else "video"))\n' -audio_offer_patch = """ logger.info("{} pipeline started".format("audio" if audio_only else "video")) - if audio_only: - logger.info("forcing audio SDP offer for split product audio peer") - self.__on_negotiation_needed(self.webrtcbin) -""" -if audio_offer_patch not in text: - if audio_offer_marker not in text: - raise SystemExit("Selkies split audio offer patch target not found") - text = text.replace(audio_offer_marker, audio_offer_patch, 1) -if "Failed to add {} to pipeline" in text: - raise SystemExit("Selkies audio pipeline still contains the obsolete strict add check") -if "emitting ICE candidate" not in text: - raise SystemExit("Selkies must log outbound ICE candidates at info level") -path.write_text(text) -PY -} - -extract_browser_vm_selkies_app() { - local rootfs="$1" - local target="$2" - local debugfs="$3" - local guest_path="/usr/local/lib/python3.11/dist-packages/selkies_gstreamer/gstwebrtc_app.py" - - if ! "$debugfs" -R "cat ${guest_path}" "$rootfs" > "$target" 2>/dev/null; then - echo "Browser VM rootfs Selkies gstwebrtc_app.py was not found: ${rootfs}" >&2 - exit 1 - fi - patch_browser_vm_selkies_app_source "$target" -} - -write_browser_vm_target_manifest() { - local target="$1" - cat > "$target" <<'JSON' -{ - "schema": "elastos.browser.vm-target/v1", - "engine": "chromium_microvm", - "network_mode": "runtime_net_only", - "direct_network": false, - "wallet_injection": false, - "media_transport": "runtime_relay", - "display_mode": "webrtc_remote_display", - "guarantee_level": "mechanism_microvm", - "display_backend": "vm_selkies_gstreamer_webrtc", - "runtime_exit_transport": "vsock_relay", - "control_transport": "vsock_relay", - "control_port": 19092 -} -JSON -} - -refresh_browser_vm_native_helpers() { - local rootfs="$1" - local debugfs="$2" - local target_platform="$3" - local runtime_relay_source - local guest_control_bridge_source - - runtime_relay_source="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-runtime-relay" \ - "ELASTOS_BROWSER_VM_RUNTIME_RELAY_BIN" \ - "browser-vm-runtime-relay" \ - "browser-vm-runtime-relay" \ - "$target_platform")" - guest_control_bridge_source="$(resolve_browser_vm_guest_helper_source \ - "browser-vm-guest-control-bridge" \ - "ELASTOS_BROWSER_VM_GUEST_CONTROL_BRIDGE_BIN" \ - "browser-vm-guest-control-bridge" \ - "browser-vm-guest-control-bridge" \ - "$target_platform")" - validate_linux_guest_binary "browser-vm-runtime-relay" \ - "$runtime_relay_source" "$target_platform" - validate_linux_guest_binary "browser-vm-guest-control-bridge" \ - "$guest_control_bridge_source" "$target_platform" - refresh_browser_vm_rootfs_file "$rootfs" "$runtime_relay_source" \ - "/opt/elastos/bin/browser-vm-runtime-relay" "0100755" \ - "runtime-relay" "$debugfs" - refresh_browser_vm_rootfs_file "$rootfs" "$guest_control_bridge_source" \ - "/opt/elastos/bin/browser-vm-guest-control-bridge" "0100755" \ - "guest-control-bridge" "$debugfs" -} - -refresh_browser_vm_rootfs_files() { - local rootfs="${ELASTOS_BROWSER_VM_ROOTFS:-${DATA_DIR}/browser-vm/rootfs.ext4}" - local control_source="${ROOT}/scripts/browser-selkies-control-service.mjs" - local vz_transport_bootstrap_source="${ROOT}/scripts/browser-vm-vz-transport-bootstrap.mjs" - - if [[ ! -f "$rootfs" ]]; then - return - fi - - local debugfs - debugfs="$(find_debugfs || true)" - if [[ -z "$debugfs" ]]; then - echo "Browser VM rootfs exists but debugfs was not found; install e2fsprogs or set ELASTOS_DEBUGFS_BIN so setup-source-home can refresh VM guest files." >&2 - exit 1 - fi - - local target_platform - local native_proxy_source="" - target_platform="$(browser_vm_target_platform || true)" - if [[ -z "$target_platform" ]]; then - echo "Browser VM rootfs helper refresh is unsupported on this platform" >&2 - exit 1 - fi - if [[ -n "$target_platform" ]]; then - if ! native_proxy_source="$(resolve_browser_vm_native_proxy_source "$target_platform")"; then - if [[ -n "${ELASTOS_BROWSER_NATIVE_PROXY_BIN:-}" ]]; then - exit 1 - fi - native_proxy_source="" - fi - fi - - local init_source - local selkies_start_source - local selkies_app_source - local manifest_source - init_source="$(mktemp)" - selkies_start_source="$(mktemp)" - selkies_app_source="$(mktemp)" - manifest_source="$(mktemp)" - extract_browser_vm_init "$init_source" - extract_browser_vm_selkies_start "$selkies_start_source" - extract_browser_vm_selkies_app "$rootfs" "$selkies_app_source" "$debugfs" - write_browser_vm_target_manifest "$manifest_source" - refresh_browser_vm_rootfs_file "$rootfs" "$manifest_source" \ - "/etc/elastos/browser-vm-target.json" "0100644" \ - "target-manifest" "$debugfs" - refresh_browser_vm_rootfs_file "$rootfs" "$control_source" \ - "/opt/elastos/bin/browser-selkies-control-service.mjs" "0100644" \ - "control-service" "$debugfs" - refresh_browser_vm_rootfs_file "$rootfs" "$vz_transport_bootstrap_source" \ - "/opt/elastos/bin/browser-vm-vz-transport-bootstrap.mjs" "0100755" \ - "vz-transport-bootstrap" "$debugfs" - refresh_browser_vm_native_helpers "$rootfs" "$debugfs" "$target_platform" - refresh_browser_vm_rootfs_file "$rootfs" "$init_source" \ - "/opt/elastos/bin/browser-vm-init" "0100755" \ - "vm-init" "$debugfs" - refresh_browser_vm_rootfs_file "$rootfs" "$selkies_app_source" \ - "/usr/local/lib/python3.11/dist-packages/selkies_gstreamer/gstwebrtc_app.py" "0100644" \ - "selkies-gstwebrtc-app" "$debugfs" - "$debugfs" -w -R "rm /usr/local/lib/python3.11/dist-packages/selkies_gstreamer/__pycache__/gstwebrtc_app.cpython-311.pyc" \ - "$rootfs" >/dev/null 2>&1 || true - refresh_browser_vm_rootfs_file "$rootfs" "$selkies_start_source" \ - "/opt/elastos/bin/browser-vm-selkies-start" "0100755" \ - "selkies-start" "$debugfs" - if [[ -n "$native_proxy_source" ]]; then - validate_linux_guest_binary "browser-native-proxy-engine" \ - "$native_proxy_source" "$target_platform" - refresh_browser_vm_rootfs_file "$rootfs" "$native_proxy_source" \ - "/opt/elastos/bin/browser-native-proxy-engine" "0100755" \ - "native-proxy" "$debugfs" - fi - rm -f "$init_source" "$selkies_start_source" "$selkies_app_source" "$manifest_source" - prune_file_backups "$rootfs" "$(browser_vm_backup_retention)" -} - -resolve_existing_symlink_target() { - local path="$1" - local target - if [[ ! -L "$path" ]]; then - printf '%s\n' "$path" - return - fi - target="$(readlink "$path")" - if [[ "$target" == /* ]]; then - printf '%s\n' "$target" - else - printf '%s/%s\n' "$(cd "$(dirname "$path")" && pwd -P)" "$target" - fi -} - -refresh_browser_vm_initrd_path() { - local requested_initrd="$1" - local initrd - local source="${ROOT}/scripts/browser-selkies-control-service.mjs" - - if [[ ! -f "$requested_initrd" ]]; then - return - fi - initrd="$(resolve_existing_symlink_target "$requested_initrd")" - if ! command -v cpio >/dev/null 2>&1; then - echo "Browser VM initrd exists but cpio was not found; install cpio so setup-source-home can refresh browser-selkies-control-service.mjs." >&2 - exit 1 - fi - if ! command -v gzip >/dev/null 2>&1; then - echo "Browser VM initrd exists but gzip was not found; install gzip so setup-source-home can refresh browser-selkies-control-service.mjs." >&2 - exit 1 - fi - - local work_dir - work_dir="$(mktemp -d)" - gzip -dc "$initrd" | (cd "$work_dir" && cpio -id --quiet) - mkdir -p "$work_dir/bin" - - if [[ -f "$work_dir/bin/browser-selkies-control-service.mjs" ]] && - cmp -s "$source" "$work_dir/bin/browser-selkies-control-service.mjs"; then - rm -rf "$work_dir" - return - fi - - install -m 644 "$source" "$work_dir/bin/browser-selkies-control-service.mjs" - local backup="${initrd}.before-selkies-control-$(date -u +%Y%m%dT%H%M%SZ)" - clone_or_copy_file "$initrd" "$backup" - (cd "$work_dir" && find . -print0 | cpio --null -o --format=newc 2>/dev/null | gzip -9) > "${initrd}.new" - mv "${initrd}.new" "$initrd" - - local verify_dir - verify_dir="$(mktemp -d)" - gzip -dc "$initrd" | (cd "$verify_dir" && cpio -id --quiet) - if ! cmp -s "$source" "$verify_dir/bin/browser-selkies-control-service.mjs"; then - echo "Browser VM initrd control-service refresh did not verify; backup kept at ${backup}" >&2 - exit 1 - fi - rm -rf "$work_dir" "$verify_dir" - prune_file_backups "$initrd" "$(browser_vm_backup_retention)" - if [[ "$requested_initrd" != "$initrd" ]]; then - echo "[setup-source-home] refreshed Browser VM initrd control service: ${requested_initrd} -> ${initrd}" - else - echo "[setup-source-home] refreshed Browser VM initrd control service: ${initrd}" - fi -} - -refresh_browser_vm_initrd_control_service() { - local initrds=() - - if [[ -n "${ELASTOS_BROWSER_VM_INITRD:-}" ]]; then - initrds+=("${ELASTOS_BROWSER_VM_INITRD}") - fi - if [[ -n "${ELASTOS_BROWSER_VM_INITRAMFS:-}" ]]; then - initrds+=("${ELASTOS_BROWSER_VM_INITRAMFS}") - fi - if [[ "${#initrds[@]}" -eq 0 ]]; then - initrds+=("${DATA_DIR}/browser-vm/initrd" "${DATA_DIR}/bin/initrd") - fi - - local seen=":" - local initrd - for initrd in "${initrds[@]}"; do - if [[ "$seen" == *":${initrd}:"* ]]; then - continue - fi - seen="${seen}${initrd}:" - refresh_browser_vm_initrd_path "$initrd" - done -} - install_browser_runtime_helpers() { echo "[setup-source-home] install Browser runtime helper scripts" mkdir -p "${DATA_DIR}/bin" "${DATA_DIR}/scripts" @@ -1446,8 +664,8 @@ EOF "${ROOT}/scripts/setup-source-home-browser-artifacts.sh" \ --data-dir "${DATA_DIR}" \ --platform "${PLATFORM}" - refresh_browser_vm_initrd_control_service - refresh_browser_vm_rootfs_files + # Guest images are build outputs. Setup consumes the verified set and keeps + # its bytes and receipt intact; guest changes go through the image builder. } start_browser_runtime_turn() { @@ -1757,7 +975,6 @@ if [[ "${SETUP_SOURCE_HOME_CONFIG_ONLY:-0}" == "1" ]]; then exit 0 fi -browser_vm_backup_retention >/dev/null require_minimum_free_space "${ROOT}" require_minimum_free_space "${DATA_DIR}" @@ -1777,7 +994,6 @@ if [[ "$PLATFORM" == "darwin-arm64" ]]; then echo "[setup-source-home] build Browser VZ engine supervisor" "$CARGO_BIN" build --manifest-path "${ROOT}/elastos/Cargo.toml" --release -p elastos-vz --bin browser-vz-engine-supervisor fi -build_browser_vm_guest_helpers source_home_binary_manifest_path() { local name="$1" diff --git a/state.md b/state.md index e8b80323c..b194e03f2 100644 --- a/state.md +++ b/state.md @@ -126,13 +126,31 @@ selection. Every matrix row remains pending installed qualification for the Browser maturity changes. Existing installed observations retain their original source bindings and verdicts. +Source-home setup now verifies rootfs, kernel and initrd as one build-manifest +set before creating artifact links. It includes the manifest in managed Runtime +projections and preserves existing and shared files. Automatic guest patching +and its guest-helper cross-build were removed from setup; guest changes use the +image builder. Target maintenance verifies image integrity and guest parity +before host-helper writes. Its guest mismatch path preserves the image, initrd +and receipt. This replaces the known setup path that changed image bytes while +leaving a stale receipt. Automatic package acquisition, publisher admission, +source-bound guest builds, atomic host/guest updates and product proof remain open. +The shared image-set check and readiness cache use the same initrd selection as +the Linux crosvm and Mac VZ launchers. Seventeen setup cases and 27 artifact/control +cases passed, including a real ext4/cpio maintenance rejection that preserves +guest and host files. The repository entropy, display-mode and formatting gates +passed. A read-only admission attempt against the existing Mac artifact store +rejected the changed image in 2.9 seconds; source hashes and the isolated test +Home remained unchanged. These are installation-source and rejection results; +positive installed Browser operation and independent review remain open. + The existing Mac image has a SHA-256 that differs from its sidecar manifest. The previous rootfs preflight skipped the receipt when `debugfs` was available and passed that image. The revised check rejects it in both inspection modes. Seventeen regression cases passed, including a real 64 MiB ext4 fixture, changed image bytes, missing receipts, wrong architecture and missing guest dependencies. Read-only checks of the existing 8 GiB image took about three seconds per hash -on this Mac. Runtime admission and its cached identity still need implementation; +on this Mac. Runtime package admission still needs implementation; these checks do not establish launch or media readiness. The embedded Browser control helper also differs from current source, so source parity and the artifact receipt still require repair. The user's active test Home lacks the required From 21966efe5c43b8f7859734c24c8fc7fa405366a8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 20:44:57 -0500 Subject: [PATCH 080/535] fix(browser): resolve image builder tools outside user PATH --- scripts/browser-entropy-check.mjs | 2 +- scripts/build/build-browser-vm-rootfs.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index a8845313d..36331dfe9 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -1648,7 +1648,7 @@ assert( browserVmRootfsBuild.includes("/opt/gst-web/index.html") && browserVmRootfsBuild.includes("python3 -m pip install") && browserVmRootfsBuild.includes("linux-libc-dev") && - browserVmRootfsBuild.includes("mke2fs -q -t ext4") && + browserVmRootfsBuild.includes('"$mke2fs_bin" -q -t ext4') && !browserVmRootfsBuild.includes("docker ") && !browserVmRootfsBuild.includes("Docker is used here only as an") && browserVmTargetStageSmoke.includes("elastos.browser.vm-target-stage-smoke/v1") && diff --git a/scripts/build/build-browser-vm-rootfs.sh b/scripts/build/build-browser-vm-rootfs.sh index cdc928b98..e581af873 100755 --- a/scripts/build/build-browser-vm-rootfs.sh +++ b/scripts/build/build-browser-vm-rootfs.sh @@ -135,7 +135,7 @@ as_root() { } require_cmd cargo -require_cmd mke2fs +mke2fs_bin="$(resolve_cmd mke2fs)" require_cmd cpio require_cmd gzip require_cmd python3 @@ -885,7 +885,7 @@ echo "[browser-vm-rootfs] run rootfs preflight" echo "[browser-vm-rootfs] pack ext4 image" rm -f "$rootfs_image" -as_root mke2fs -q -t ext4 -d "$rootfs_dir" -F "$rootfs_image" "$rootfs_size" +as_root "$mke2fs_bin" -q -t ext4 -d "$rootfs_dir" -F "$rootfs_image" "$rootfs_size" as_root chown "$(id -u):$(id -g)" "$rootfs_image" python3 - "$out_dir" "$target_platform" "$rootfs_image" "$kernel_image" "$initrd_image" <<'PY' From ee5a84025f235497b553a04c52551f7974ddf704 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 21:52:22 -0500 Subject: [PATCH 081/535] fix(browser): retain online state in isolated guests --- ...browser-runtime-online-projection.test.mjs | 24 ++++++++++++++++++- .../browser-selkies-control-service-smoke.sh | 4 ++-- scripts/browser-selkies-control-service.mjs | 6 ++++- 3 files changed, 30 insertions(+), 4 deletions(-) diff --git a/scripts/browser-runtime-online-projection.test.mjs b/scripts/browser-runtime-online-projection.test.mjs index 21dc1fe8a..5cdd56382 100644 --- a/scripts/browser-runtime-online-projection.test.mjs +++ b/scripts/browser-runtime-online-projection.test.mjs @@ -45,7 +45,7 @@ test("Runtime online projection returns a bounded verified state", async () => { method: "Network.overrideNetworkState", params: { offline: false, - latency: 0, + latency: 1, downloadThroughput: -1, uploadThroughput: -1, connectionType: "other", @@ -65,6 +65,28 @@ test("Runtime online projection returns a bounded verified state", async () => { ]); }); +test("a no-NIC guest keeps an explicit online override", async () => { + let online = false; + const cdp = { + async request(method, params) { + if (method === "Network.overrideNetworkState") { + // Chromium's SetNetworkStateOverride clears an all-disabled override, + // restoring the guest's actual offline state. Connection type alone + // does not activate it. + const enabled = params.offline || params.latency > 0 || + params.downloadThroughput >= 0 || params.uploadThroughput >= 0; + online = enabled && !params.offline; + } + if (method === "Runtime.evaluate") { + return { result: { value: JSON.stringify({ online }) } }; + } + return {}; + }, + }; + const state = await projectRuntimeProxyOnlineState(cdp, new URL("http://127.0.0.1:19094/")); + assert.equal(state.online, true); +}); + test("online projection brackets initial navigation and follows later navigation", () => { const beforeProjection = controlServiceSource.indexOf( '"before_initial_navigation"', diff --git a/scripts/browser-selkies-control-service-smoke.sh b/scripts/browser-selkies-control-service-smoke.sh index d543b3f6c..31bda5479 100755 --- a/scripts/browser-selkies-control-service-smoke.sh +++ b/scripts/browser-selkies-control-service-smoke.sh @@ -491,7 +491,7 @@ server.on("upgrade", (req, socket) => { const projection = message.params || {}; if ( projection.offline !== false || - projection.latency !== 0 || + projection.latency !== 1 || projection.downloadThroughput !== -1 || projection.uploadThroughput !== -1 || projection.connectionType !== "other" @@ -1306,7 +1306,7 @@ if (response.active_pages !== 0) throw new Error("failed open leaked an active S const projection = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); if ( projection.offline !== false || - projection.latency !== 0 || + projection.latency !== 1 || projection.downloadThroughput !== -1 || projection.uploadThroughput !== -1 || projection.connectionType !== "other" diff --git a/scripts/browser-selkies-control-service.mjs b/scripts/browser-selkies-control-service.mjs index 56bb9bafd..1626eff4a 100755 --- a/scripts/browser-selkies-control-service.mjs +++ b/scripts/browser-selkies-control-service.mjs @@ -3023,7 +3023,11 @@ export async function projectRuntimeProxyOnlineState(cdp, runtimeFetchProxyUrl) await cdp.request("Network.enable"); await cdp.request("Network.overrideNetworkState", { offline: false, - latency: 0, + // Chromium clears an override when latency is zero and both throughputs + // are disabled. Keep a 1 ms navigator RTT hint so a no-NIC guest stays + // online through Runtime. This command changes navigator state only; + // Runtime's proxy still owns traffic and applies no CDP throttling. + latency: 1, downloadThroughput: -1, uploadThroughput: -1, connectionType: "other", From 664c81521adf33c6470bb2ed7570abbc6f902885 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 22:20:44 -0500 Subject: [PATCH 082/535] fix(net): defer stream destination policy to Exit --- capsules/net-provider/src/main.rs | 62 ++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/capsules/net-provider/src/main.rs b/capsules/net-provider/src/main.rs index 0f10957dc..a3f97573b 100644 --- a/capsules/net-provider/src/main.rs +++ b/capsules/net-provider/src/main.rs @@ -219,8 +219,10 @@ impl NetProvider { let Some(host) = url.host_str() else { return Response::error("invalid_request", "stream target requires a host"); }; - if let Err(err) = validate_public_host(host) { - return Response::error("private_network_blocked", err); + // Runtime hands this request to the selected Exit, which owns destination + // policy, including exact private-target grants. Net opens no connection. + if let Err(err) = validate_host_shape(host) { + return Response::error("invalid_request", err); } self.exit_unavailable( "stream", @@ -299,6 +301,18 @@ fn configured_exit_count(config: &Value) -> usize { fn validate_public_host(host: &str) -> Result<(), String> { let host = host.trim().trim_matches(['[', ']']); + validate_host_shape(host)?; + let lower = host.to_ascii_lowercase(); + if lower == "localhost" || lower.ends_with(".localhost") || lower.ends_with(".local") { + return Err(format!("private host blocked: {host}")); + } + if let Ok(ip) = host.parse::() { + return validate_public_ip(ip).map_err(|_| format!("private IP blocked: {host}")); + } + Ok(()) +} + +fn validate_host_shape(host: &str) -> Result<(), String> { if host.is_empty() { return Err("host must not be empty".to_string()); } @@ -308,13 +322,6 @@ fn validate_public_host(host: &str) -> Result<(), String> { { return Err(format!("invalid host: {host}")); } - let lower = host.to_ascii_lowercase(); - if lower == "localhost" || lower.ends_with(".localhost") || lower.ends_with(".local") { - return Err(format!("private host blocked: {host}")); - } - if let Ok(ip) = host.parse::() { - return validate_public_ip(ip).map_err(|_| format!("private IP blocked: {host}")); - } Ok(()) } @@ -451,6 +458,43 @@ mod tests { } } + #[test] + fn stream_defers_destination_policy_to_runtime_exit() { + let provider = NetProvider::new(); + for target in [ + "tcp://localhost:61511", + "tcp://127.0.0.1:61511", + "tcp://192.168.1.1:80", + "tcp://[::1]:61511", + "tls://example.com:443", + ] { + // This is a denied network operation until Runtime obtains an Exit receipt. + assert_eq!( + error_code(provider.stream(target, None, None, None)), + "exit_unavailable", + "{target}" + ); + } + } + + #[test] + fn stream_rejects_invalid_targets_before_exit_handoff() { + let provider = NetProvider::new(); + for target in [ + "not a URL", + "file:///etc/hosts", + "http://example.com/", + "tcp:///", + "tcp://bad host:80", + ] { + assert_eq!( + error_code(provider.stream(target, None, None, None)), + "invalid_request", + "{target}" + ); + } + } + #[test] fn http_rejects_unsupported_methods_and_schemes() { let provider = NetProvider::new(); From 06b32c65e0a2e07786f9acf809bb2e84b4f4f5f3 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 22:21:33 -0500 Subject: [PATCH 083/535] fix(browser): scope source-home IPC to each Runtime --- docs/BROWSER_VM_TARGET.md | 11 ++- scripts/browser-entropy-check.mjs | 4 +- scripts/browser-mac-vm-proof-config.test.mjs | 98 ++++++++++++++++++++ scripts/browser-mac-vm-proof.sh | 29 +++++- scripts/browser-source-home-config-smoke.sh | 65 ++++++++++++- scripts/browser-source-home-config.mjs | 15 ++- 6 files changed, 208 insertions(+), 14 deletions(-) create mode 100644 scripts/browser-mac-vm-proof-config.test.mjs diff --git a/docs/BROWSER_VM_TARGET.md b/docs/BROWSER_VM_TARGET.md index b0555f063..dd9f1ca65 100644 --- a/docs/BROWSER_VM_TARGET.md +++ b/docs/BROWSER_VM_TARGET.md @@ -253,9 +253,14 @@ be backed by a local crosvm/VZ substrate or by a remote/operator VM provider reached through approved Runtime/Carrier/SSH-tunnel plumbing. Browser UI still talks only to Runtime, and the Browser Engine Adapter still requires `runtime_net_only`, `direct_network=false`, and `media_transport=runtime_relay`. -Source-home config writes a stable `/tmp/elastos-browser-vm-control-.sock` -control socket by default so the Browser VM supervisor has a single launch -target instead of relying on ambient shell state. +Source-home config defaults to +`/tmp/elastos-browser---vm-control.sock`. The hash uses +the first 16 hex characters of SHA-256 over the normalized absolute Runtime +data-directory path, so each Runtime has a stable launch target. The Mac VM +proof reads `browser-vm-product.supervisor.control_socket_path` from the target +data directory's `config/browser-engine-adapter.json`; an explicit +`ELASTOS_BROWSER_VM_CONTROL_SOCKET` overrides that selection. A missing or +invalid configured socket stops the proof before it contacts Home. `scripts/browser-vm-control-service.mjs` is the local Unix-socket control-plane contract. It serves `GET /status`, `POST /pages`, page-scoped `POST /shutdown`, diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index 36331dfe9..dd04de81b 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -1032,7 +1032,7 @@ assert( !browserSourceHomeConfig.includes("preferred_display_mode") && browserSourceHomeConfig.includes("relay_ipc") && browserSourceHomeConfig.includes("relay_ipc: true") && - browserSourceHomeConfig.includes("-relay.sock") && + browserSourceHomeConfig.includes('runtimeSocketPath(args, "exit-relay")') && browserSourceHomeConfig.includes("browser-local-exit.json") && browserSourceHomeConfig.includes("elastos.browser.local-exit.config/v1") && browserSourceHomeConfig.includes("runtimeGatewayPrivateTargets") && @@ -1041,7 +1041,7 @@ assert( browserSourceHomeConfig.includes('ports: [80, 443]') && browserSourceHomeConfig.includes("relay_ipc_path") && browserSourceHomeConfig.includes("control_socket_path") && - browserSourceHomeConfig.includes("/tmp/elastos-browser-vm-control-${args.platform}.sock") && + browserSourceHomeConfig.includes('runtimeSocketPath(args, "vm-control")') && browserSourceHomeConfig.includes("ELASTOS_BROWSER_VM_CONTROL_SOCKET") && !browserSourceHomeConfig.includes("ELASTOS_BROWSER_VM_PROFILE_DISK_ROOT") && browserSourceHomeConfig.includes("ELASTOS_BROWSER_VM_EGRESS_MAX_SESSIONS") && diff --git a/scripts/browser-mac-vm-proof-config.test.mjs b/scripts/browser-mac-vm-proof-config.test.mjs new file mode 100644 index 000000000..901adee75 --- /dev/null +++ b/scripts/browser-mac-vm-proof-config.test.mjs @@ -0,0 +1,98 @@ +import assert from "node:assert/strict"; +import { execFileSync, spawnSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import test from "node:test"; + +const proofPath = fileURLToPath(new URL("./browser-mac-vm-proof.sh", import.meta.url)); +const generatorPath = fileURLToPath(new URL("./browser-source-home-config.mjs", import.meta.url)); +const configPath = dataDir => path.join(dataDir, "config/browser-engine-adapter.json"); +const adapter = socket => ({ id: "browser-vm-product", supervisor: { control_socket_path: socket } }); + +function fixture(t) { + const root = mkdtempSync(path.join(os.tmpdir(), "browser-proof-config-")); + t.after(() => rmSync(root, { recursive: true, force: true })); + return root; +} + +function probe(dataDir, override = "") { + // Run the actual consumer with inert HTTP and stop at its first socket probe, + // before either virtual-auth journey can run. + return spawnSync("bash", ["-c", ` +exec 3>&2 +curl() { + while [[ $# -gt 0 ]]; do + if [[ "$1" == "--unix-socket" ]]; then + printf 'CONTROL_SOCKET=%s\\n' "$2" >&3 + exit 77 + fi + shift + done + printf 'HOME_PROBE\\n' >&2 + printf '200' +} +export -f curl +exec bash "$1" +`, "proof-config-test", proofPath], { + cwd: path.dirname(path.dirname(proofPath)), + env: { ...process.env, ELASTOS_NODE_BIN: process.execPath, + ELASTOS_MAC_BROWSER_DATA_DIR: dataDir, ELASTOS_BROWSER_VM_CONTROL_SOCKET: override, + ELASTOS_BROWSER_MAC_VM_PROOF_AUTH_PROFILE: "" }, + encoding: "utf8", timeout: 10_000, + }); +} + +function expectSocket(result, socket) { + assert.equal(result.error, undefined); + assert.equal(result.status, 77, result.stderr); + assert.ok(result.stderr.includes(`CONTROL_SOCKET=${socket}\n`), result.stderr); +} + +test("Mac proof follows each target's generated socket and selects the product adapter by ID", t => { + const root = fixture(t); + const sockets = []; + for (const home of ["Home One", "Home Two"]) { + const dataDir = path.join(root, home, "Library/Application Support/elastos"); + execFileSync(process.execPath, [generatorPath, "--data-dir", dataDir, "--platform", "darwin-arm64"]); + const config = JSON.parse(readFileSync(configPath(dataDir), "utf8")); + const socket = config.adapters.find(entry => entry.id === "browser-vm-product").supervisor.control_socket_path; + config.adapters.unshift({ id: "other-engine", supervisor: { control_socket_path: "/tmp/foreign-engine.sock" } }); + writeFileSync(configPath(dataDir), JSON.stringify(config)); + expectSocket(probe(dataDir), socket); + sockets.push(socket); + } + assert.notEqual(sockets[0], sockets[1]); +}); + +test("explicit socket override wins and works without target config", t => { + const root = fixture(t); + const socket = path.join(root, "explicit control.sock"); + expectSocket(probe(root, socket), socket); + mkdirSync(path.dirname(configPath(root)), { recursive: true }); + writeFileSync(configPath(root), JSON.stringify({ adapters: [adapter("/tmp/configured.sock")] })); + expectSocket(probe(root, socket), socket); +}); + +test("missing, malformed, ambiguous or invalid target config fails before Home is contacted", t => { + const root = fixture(t); + const cases = [ + undefined, "{broken-json", {}, + { adapters: [adapter("/tmp/one.sock"), adapter("/tmp/two.sock")] }, + { adapters: [adapter(undefined)] }, + { adapters: [adapter("relative.sock")] }, + { adapters: [adapter("/tmp/control.sock\n")] }, + ]; + for (const [index, config] of cases.entries()) { + const dataDir = path.join(root, String(index)); + if (config !== undefined) { + mkdirSync(path.dirname(configPath(dataDir)), { recursive: true }); + writeFileSync(configPath(dataDir), typeof config === "string" ? config : JSON.stringify(config)); + } + const result = probe(dataDir); + assert.equal(result.status, 2, result.stderr); + assert.match(result.stderr, /Cannot resolve Browser VM control socket/); + assert.doesNotMatch(result.stderr, /HOME_PROBE|CONTROL_SOCKET=/); + } +}); diff --git a/scripts/browser-mac-vm-proof.sh b/scripts/browser-mac-vm-proof.sh index febd28a50..12af88c01 100755 --- a/scripts/browser-mac-vm-proof.sh +++ b/scripts/browser-mac-vm-proof.sh @@ -49,7 +49,7 @@ Environment: ELASTOS_BROWSER_MAC_VM_PROFILE_RESET_PROOF default: 0; set 1 to reset the virtual test principal profile after proof pages close ELASTOS_BROWSER_MAC_VM_PROOF_AUTH_PROFILE optional persistent virtual-auth profile dir for authenticated ela.city proof HOME_VIRTUAL_AUTH_BROWSER_DIAGNOSTIC_CLICK_OPTIONAL default: 1 for this proof - ELASTOS_BROWSER_VM_CONTROL_SOCKET default: /tmp/elastos-browser-vm-control-darwin-arm64.sock + ELASTOS_BROWSER_VM_CONTROL_SOCKET optional override; default: browser-vm-product supervisor.control_socket_path in the target adapter config ELASTOS_MAC_BROWSER_DATA_DIR default: ~/elastos-mac-test-home/Library/Application Support/elastos ELASTOS_NODE_BIN optional node binary USAGE @@ -100,8 +100,33 @@ embedded_open_urls="${HOME_VIRTUAL_AUTH_BROWSER_OPEN_URLS:-}" if [[ -n "$click_href_re" ]]; then embedded_open_urls="$proof_url" fi -control_socket="${ELASTOS_BROWSER_VM_CONTROL_SOCKET:-/tmp/elastos-browser-vm-control-darwin-arm64.sock}" data_dir="${ELASTOS_MAC_BROWSER_DATA_DIR:-${HOME}/elastos-mac-test-home/Library/Application Support/elastos}" +control_socket="${ELASTOS_BROWSER_VM_CONTROL_SOCKET:-}" +if [[ -z "$control_socket" ]]; then + control_socket="$("$node_bin" - "$data_dir/config/browser-engine-adapter.json" <<'NODE' +const fs = require("node:fs"); +const configPath = process.argv[2]; +function fail(reason) { + console.error(`Cannot resolve Browser VM control socket from ${configPath}: ${reason}`); + process.exit(2); +} +let config; +try { + config = JSON.parse(fs.readFileSync(configPath, "utf8")); +} catch { + fail("expected a readable JSON adapter config"); +} +const adapters = Array.isArray(config?.adapters) + ? config.adapters.filter(entry => entry?.id === "browser-vm-product") : []; +if (adapters.length !== 1) fail("expected exactly one browser-vm-product adapter"); +const socket = adapters[0].supervisor?.control_socket_path; +if (typeof socket !== "string" || !socket.startsWith("/") || /[\r\n\0]/.test(socket)) { + fail("supervisor.control_socket_path must be an absolute path without control characters"); +} +process.stdout.write(socket); +NODE +)" +fi installed_home="${data_dir}/capsules/home/browser/index.html" source_home="capsules/home/browser/index.html" tmp_dir="$(mktemp -d /tmp/elastos-browser-mac-vm-proof-XXXXXX)" diff --git a/scripts/browser-source-home-config-smoke.sh b/scripts/browser-source-home-config-smoke.sh index 22658cfe8..cfad18134 100755 --- a/scripts/browser-source-home-config-smoke.sh +++ b/scripts/browser-source-home-config-smoke.sh @@ -34,6 +34,64 @@ if [[ -z "$node_bin" ]]; then exit 2 fi +"$node_bin" - "$tmp_dir" <<'NODE' +const assert = require("node:assert/strict"); +const { execFileSync } = require("node:child_process"); +const fs = require("node:fs"); +const path = require("node:path"); +const tmpDir = process.argv[2]; + +function generate(dataDir, platform, name, controlSocket) { + const outDir = path.join(tmpDir, name); + const args = [ + "scripts/browser-source-home-config.mjs", + "--data-dir", dataDir, + "--platform", platform, + "--out-dir", outDir, + ]; + if (controlSocket) args.push("--vm-control-socket", controlSocket); + execFileSync(process.execPath, args); + const read = (file) => JSON.parse(fs.readFileSync(path.join(outDir, file), "utf8")); + const adapter = read("browser-engine-adapter.json").adapters[0]; + const backend = read("exit-provider.json").backends[0]; + const localExit = read("browser-local-exit.json"); + assert.equal(adapter.network_mode, "runtime_net_only"); + assert.equal(adapter.supervisor.control_socket_path, adapter.supervisor.env.ELASTOS_BROWSER_VM_CONTROL_SOCKET); + assert.equal(backend.relay_ipc.path, localExit.relay_ipc_path); + const sockets = [backend.adapter_ipc.path, backend.relay_ipc.path, adapter.supervisor.control_socket_path]; + assert.equal(new Set(sockets).size, 3, "each IPC role needs a distinct socket"); + for (const socket of sockets) { + assert.ok(path.isAbsolute(socket) && !/[\s\0]/.test(socket), "consumers require absolute IPC paths without whitespace"); + assert.ok(Buffer.byteLength(socket) < 104, "IPC defaults must fit the macOS Unix socket path limit"); + } + return sockets; +} + +const allDefaults = new Set(); +for (const platform of ["darwin-arm64", "linux-amd64", "linux-arm64"]) { + // Homes can have long paths with spaces and the same final data-dir name. + const homeRoot = path.join(tmpDir, "Long Home ".repeat(16), "Library", "Application Support"); + const dataDir = path.join(homeRoot, "one", "elastos"); + const sockets = generate(dataDir, platform, `${platform}-one`); + const other = generate(path.join(homeRoot, "two", "elastos"), platform, `${platform}-two`); + for (const socket of [...sockets, ...other]) { + assert.ok(!allDefaults.has(socket), "distinct Runtime data dirs and platforms need distinct IPC defaults"); + allDefaults.add(socket); + } + assert.deepEqual( + generate(`${dataDir}/../elastos/`, platform, `${platform}-repeat`), + sockets, + "IPC defaults must stay stable across normalized data-dir paths and output dirs", + ); + const explicitControl = `/tmp/elastos-config-smoke-${platform}-control.sock`; + assert.deepEqual( + generate(dataDir, platform, `${platform}-explicit`, explicitControl), + [sockets[0], sockets[1], explicitControl], + "an explicit VM control socket must preserve this Runtime's Exit IPC defaults", + ); +} +NODE + rejected_output="$tmp_dir/rejected.out" if "$node_bin" scripts/browser-source-home-config.mjs \ --data-dir "$tmp_dir/rejected-data" \ @@ -432,10 +490,11 @@ if ( throw new Error("Linux remote VZ source-home config must issue VZ transport authority"); } const defaultMacAdapter = defaultMacVmAdapter.adapters?.[0]; -if (defaultMacAdapter?.supervisor?.control_socket_path !== "/tmp/elastos-browser-vm-control-darwin-arm64.sock") { - throw new Error("Mac source-home Browser config must declare a default VM control socket"); +const defaultMacControlSocket = defaultMacAdapter?.supervisor?.control_socket_path; +if (!/^\/tmp\/elastos-browser-darwin-arm64-[0-9a-f]{16}-vm-control\.sock$/.test(defaultMacControlSocket)) { + throw new Error("Mac source-home Browser config must declare a Runtime-scoped default VM control socket"); } -if (defaultMacAdapter?.supervisor?.env?.ELASTOS_BROWSER_VM_CONTROL_SOCKET !== "/tmp/elastos-browser-vm-control-darwin-arm64.sock") { +if (defaultMacAdapter?.supervisor?.env?.ELASTOS_BROWSER_VM_CONTROL_SOCKET !== defaultMacControlSocket) { throw new Error("Mac source-home Browser supervisor must receive the default VM control socket"); } if (defaultMacAdapter?.supervisor?.env?.ELASTOS_BROWSER_VM_CONTROL_READY_TIMEOUT_MS !== "120000") { diff --git a/scripts/browser-source-home-config.mjs b/scripts/browser-source-home-config.mjs index 496e21c1c..fa036dda1 100755 --- a/scripts/browser-source-home-config.mjs +++ b/scripts/browser-source-home-config.mjs @@ -1,4 +1,5 @@ #!/usr/bin/env node +import { createHash } from "node:crypto"; import fs from "node:fs"; import net from "node:net"; import path from "node:path"; @@ -57,7 +58,7 @@ function usage() { Options: --out-dir Default: /config --vm-supervisor Default: /bin/browser-vm-engine-supervisor - --vm-control-socket Default: /tmp/elastos-browser-vm-control-.sock + --vm-control-socket Default: /tmp/elastos-browser---vm-control.sock --vm-control-launcher Optional launcher used to auto-start the VM control socket. Linux default: /bin/browser-vm-local-crosvm-launcher Darwin default: /bin/browser-vz-engine-supervisor @@ -119,6 +120,12 @@ function validateAbsolute(label, value) { } } +function runtimeSocketPath(args, role) { + // Keep Runtime IPC stable and short even for long data dirs with spaces. + const scope = createHash("sha256").update(path.resolve(args.dataDir)).digest("hex").slice(0, 16); + return `/tmp/elastos-browser-${args.platform}-${scope}-${role}.sock`; +} + function hasVmIceEnv(env) { return Boolean(env.ELASTOS_BROWSER_VM_ICE_SERVER || env.ELASTOS_BROWSER_VM_ICE_SERVERS_JSON); } @@ -574,7 +581,7 @@ function browserLocalExit(relaySocket, { allowPrivateTargets = false } = {}) { function vmBrowserEngineAdapter(args, sourceEnv = process.env, vzTransport = null) { const supervisor = args.vmSupervisor || path.join(args.dataDir, "bin/browser-vm-engine-supervisor"); validateAbsolute("--vm-supervisor", supervisor); - const controlSocket = args.vmControlSocket || `/tmp/elastos-browser-vm-control-${args.platform}.sock`; + const controlSocket = args.vmControlSocket || runtimeSocketPath(args, "vm-control"); validateAbsolute("--vm-control-socket", controlSocket); const controlLauncher = args.vmControlLauncher || (args.platform === "darwin-arm64" @@ -681,8 +688,8 @@ function main() { validateAbsolute("--data-dir", args.dataDir); const outDir = args.outDir || path.join(args.dataDir, "config"); validateAbsolute("--out-dir", outDir); - const adapterSocket = `/tmp/elastos-browser-source-home-${args.platform}.sock`; - const relaySocket = `/tmp/elastos-browser-source-home-${args.platform}-relay.sock`; + const adapterSocket = runtimeSocketPath(args, "exit-adapter"); + const relaySocket = runtimeSocketPath(args, "exit-relay"); const sourceEnv = runtimeTurnEnv(args); const selectedControlLauncher = args.vmControlLauncher || From 9f4e123a4aba87be6da9e47eba82c339c37c4f97 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 22:23:58 -0500 Subject: [PATCH 084/535] fix(browser): route guest loopback through Runtime proxy --- scripts/browser-entropy-check.mjs | 2 +- scripts/build/stage-browser-vm-target.sh | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index dd04de81b..ac334eaa5 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -1613,7 +1613,7 @@ assert( !browserVmTargetStage.includes('>"$ELASTOS_BROWSER_VM_SERIAL_LOG_DEV"') && browserVmTargetStage.includes('"control_socket_ready_timeout_ms": 60000') && browserVmTargetStage.includes('"control_request_timeout_ms": 120000') && - !browserVmTargetStage.includes("--proxy-bypass-list=<-loopback>") && + browserVmTargetStage.includes("--proxy-bypass-list=<-loopback>") && browserVmRootfsBuild.includes("elastos.browser.vm-rootfs-build/v1") && browserVmRootfsBuild.includes("debootstrap") && browserVmRootfsBuild.includes("elastos-tiny-initrd") && diff --git a/scripts/build/stage-browser-vm-target.sh b/scripts/build/stage-browser-vm-target.sh index 979ca20ac..c6c5064bb 100755 --- a/scripts/build/stage-browser-vm-target.sh +++ b/scripts/build/stage-browser-vm-target.sh @@ -1221,6 +1221,7 @@ ELASTOS_BROWSER_NATIVE_PROXY_ENGINE_CONFIG="$(cat <", "--host-resolver-rules=MAP * ~NOTFOUND, EXCLUDE 127.0.0.1", "--disable-dev-shm-usage", "--no-first-run", From dd36a332009325efea0240b33914dc6d290f4fb3 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 22:32:31 -0500 Subject: [PATCH 085/535] fix(browser): preserve authority after destination policy denial --- capsules/browser/browser/browser-status.js | 4 ++- .../browser/browser/browser-status.test.mjs | 25 +++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/capsules/browser/browser/browser-status.js b/capsules/browser/browser/browser-status.js index 2eab37278..86948540a 100644 --- a/capsules/browser/browser/browser-status.js +++ b/capsules/browser/browser/browser-status.js @@ -15,7 +15,6 @@ export function isAuthoritySessionError(error) { const text = String(error?.message || ""); return ( error?.status === 401 || - error?.status === 403 || /auth session not found|auth session is not active|home launch token auth session is not active|home launch token expired/i.test(text) ); } @@ -96,6 +95,9 @@ export function friendlyOpenError(error) { } } if (outcome?.state === "terminal_pre_effect_failure") { + if (error.status === 403) { + return "This page was blocked by your Exit Node settings."; + } return "Browser Engine failed to start cleanly. No Browser page or VM was acquired."; } if (outcome?.state === "terminal_post_effect_cleanup") { diff --git a/capsules/browser/browser/browser-status.test.mjs b/capsules/browser/browser/browser-status.test.mjs index 1171739b5..ba1fd5399 100644 --- a/capsules/browser/browser/browser-status.test.mjs +++ b/capsules/browser/browser/browser-status.test.mjs @@ -4,9 +4,34 @@ import test from "node:test"; import { collectWebrtcStats, friendlyOpenError, + isAuthoritySessionError, runtimeOpenOutcome, } from "./browser-status.js"; +test("destination policy denial preserves Browser authority and close ownership", () => { + const error = openError("terminal_pre_effect_failure"); + error.status = 403; + error.message = "private host blocked: localhost"; + assert.equal(isAuthoritySessionError(error), false); + assert.equal(friendlyOpenError(error), "This page was blocked by your Exit Node settings."); + + error.payload.outcome.state = "cleanup_pending"; + error.payload.outcome.effects.vm_acquired = true; + assert.match(friendlyOpenError(error), /cleanup is pending/); + assert.equal(isAuthoritySessionError({ status: 403, message: "permission denied" }), false); +}); + +test("authentication failures still request Home authority renewal", () => { + for (const error of [ + { status: 401, message: "unauthorized" }, + { status: 403, message: "Home launch token expired" }, + { status: 403, message: "auth session is not active" }, + ]) { + assert.equal(isAuthoritySessionError(error), true); + assert.match(friendlyOpenError(error), /session expired/); + } +}); + function openError(state, effects) { const error = new Error("browser-vz-engine-supervisor exited"); error.status = 503; From b0f648ee6a228395e6ab9f1c51f133cb9692e604 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 22:58:14 -0500 Subject: [PATCH 086/535] fix(browser): return settled navigation without a second wait --- scripts/browser-command-navigation.test.mjs | 89 +++++++++++++++++++++ scripts/browser-selkies-control-service.mjs | 5 +- 2 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 scripts/browser-command-navigation.test.mjs diff --git a/scripts/browser-command-navigation.test.mjs b/scripts/browser-command-navigation.test.mjs new file mode 100644 index 000000000..40fc34724 --- /dev/null +++ b/scripts/browser-command-navigation.test.mjs @@ -0,0 +1,89 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import test from "node:test"; +import vm from "node:vm"; + +const source = fs.readFileSync(new URL("./browser-selkies-control-service.mjs", import.meta.url), "utf8"); +const commandSource = source.slice( + source.indexOf("async function applyBrowserCommand("), + source.indexOf("\nfunction isRecoverableCommandNavigationFailure("), +); + +function commandHarness({ navigationError = null } = {}) { + let pendingDomEvent = false; + let elapsedMs = 0; + const calls = []; + const state = { url: "https://example.test/next", title: "Next" }; + const cdp = { + async request(method) { + calls.push(method); + if (method === "Page.getNavigationHistory") { + return { currentIndex: 1, entries: [{ id: 1 }, { id: 2 }, { id: 3 }] }; + } + if (["Page.reload", "Page.navigateToHistoryEntry"].includes(method)) pendingDomEvent = true; + return {}; + }, + async waitForEvent(method, timeoutMs) { + calls.push(method); + if (pendingDomEvent) { + pendingDomEvent = false; + return {}; + } + elapsedMs += timeoutMs; + throw new Error("event timed out"); + }, + }; + const context = vm.createContext({ + validateBrowserNavigationUrl: (url) => new URL(url).href, + withBrowserCdp: async (_page, _timeout, action) => action(cdp), + ensureBrowserFileChooserInterception: async () => {}, + installWalletBridge: async () => null, + normalizeWalletBridge: (wallet) => wallet, + navigateInitialBrowserPage: async () => { + if (navigationError) throw navigationError; + // The navigation observer consumes the document event before returning. + pendingDomEvent = true; + await cdp.waitForEvent("Page.domContentEventFired", 15000); + return {}; + }, + assertBrowserNavigationSucceeded: () => {}, + projectAndLogRuntimeProxyOnlineState: async () => {}, + refreshBrowserPageState: async () => state, + assertBrowserStateDidNotLandOnErrorPage: () => {}, + isRecoverableCommandNavigationFailure: () => false, + }); + vm.runInContext(`${commandSource}\nthis.runCommand = applyBrowserCommand;`, context); + const page = { debugger_url: "ws://example.test/devtools/page/1" }; + return { + page, calls, state, + elapsedMs: () => elapsedMs, + run: (command) => context.runCommand( + { browserControl: { timeoutMs: 20000 }, runtimeFetchProxyUrl: "http://runtime.test" }, + page, { command, url: state.url }, 20000, + ), + }; +} + +test("completed navigation returns page state without an extra event timeout", async () => { + const harness = commandHarness(); + assert.equal(await harness.run("navigate"), harness.state); + assert.equal(harness.elapsedMs(), 0, "a settled page must return without waiting for another DOM event"); + assert.equal(harness.page.navigationInProgress, false); +}); + +for (const command of ["reload", "back", "forward"]) { + test(`${command} still consumes its own navigation event`, async () => { + const harness = commandHarness(); + assert.equal(await harness.run(command), harness.state); + assert.equal(harness.elapsedMs(), 0); + assert.equal(harness.calls.filter((method) => method === "Page.domContentEventFired").length, 1); + }); +} + +test("terminal navigation failure preserves the error and clears in-progress state", async () => { + const error = new Error("destination policy denied"); + const harness = commandHarness({ navigationError: error }); + await assert.rejects(harness.run("navigate"), (actual) => actual === error); + assert.equal(harness.elapsedMs(), 0); + assert.equal(harness.page.navigationInProgress, false); +}); diff --git a/scripts/browser-selkies-control-service.mjs b/scripts/browser-selkies-control-service.mjs index 1626eff4a..a52922481 100755 --- a/scripts/browser-selkies-control-service.mjs +++ b/scripts/browser-selkies-control-service.mjs @@ -4706,7 +4706,10 @@ async function applyBrowserCommand(config, browserPage, event, timeoutMs) { } else { throw new Error("unsupported browser command"); } - await cdp.waitForEvent("Page.domContentEventFired", Math.min(timeoutMs, 15000), "domcontent").catch(() => {}); + // navigateInitialBrowserPage already observes and consumes this event. + if (command !== "navigate") { + await cdp.waitForEvent("Page.domContentEventFired", Math.min(timeoutMs, 15000), "domcontent").catch(() => {}); + } await projectAndLogRuntimeProxyOnlineState( cdp, config.runtimeFetchProxyUrl, From 8c127d2f27534ddd0a343949328046269da2565d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 23:03:25 -0500 Subject: [PATCH 087/535] fix(browser): settle guest HTTP failures without launch retries --- .../src/bin/browser-vz-engine-supervisor.rs | 202 ++++++++++++++---- 1 file changed, 166 insertions(+), 36 deletions(-) diff --git a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs index 4f85d17bf..681aaeb84 100644 --- a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs +++ b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs @@ -42,6 +42,7 @@ const EGRESS_COPY_BUFFER_BYTES: usize = 256 * 1024; const MAX_CONTROL_HTTP_HEADER_BYTES: usize = 64 * 1024; const MAX_CONTROL_HTTP_BODY_BYTES: usize = 16 * 1024 * 1024; const MAX_CONTROL_HTTP_RESPONSE_BYTES: usize = 64 * 1024 * 1024; +const MAX_GUEST_CONTROL_ERROR_CHARS: usize = 512; const MAX_SETTLEMENT_MESSAGE_CHARS: usize = 2 * 1024; const DEFAULT_CONTROL_PROXY_REQUEST_TIMEOUT_MS: u32 = 120_000; const BROWSER_VM_TARGET_VERSION: &str = match option_env!("ELASTOS_RELEASE_VERSION") { @@ -2668,9 +2669,10 @@ fn sdp_has_media_kind(sdp: &str, kind: &str) -> bool { } fn is_retryable_guest_control_open_error(error: &str) -> bool { - error.contains("Browser VM guest control HTTP 503") - || error.contains("Connection reset") - || error.contains("Broken pipe") + // Only raw connection failures can mean the control service is still starting. + // A completed HTTP response is a page-open outcome, including HTTP 503. + let error = error.to_ascii_lowercase(); + error.starts_with("connection reset") || error.starts_with("broken pipe") } fn is_guest_control_response_timeout(error: &str) -> bool { @@ -2996,13 +2998,7 @@ fn parse_http_json_response(response: &[u8]) -> Result { let split = response .windows(4) .position(|window| window == b"\r\n\r\n") - .ok_or_else(|| { - format!( - "Browser VM guest control returned an invalid HTTP response: len={} preview={}", - response.len(), - response_preview(response) - ) - })?; + .ok_or_else(|| "Browser VM guest control returned an invalid HTTP response".to_string())?; let (head, body) = response.split_at(split + 4); let head_text = std::str::from_utf8(head).map_err(|_| "Browser VM guest HTTP head is not UTF-8")?; @@ -3012,39 +3008,60 @@ fn parse_http_json_response(response: &[u8]) -> Result { .nth(1) .and_then(|value| value.parse::().ok()) .ok_or_else(|| "Browser VM guest HTTP status is invalid".to_string())?; - let parsed: Value = serde_json::from_slice(body) - .map_err(|err| format!("Browser VM guest control response is not JSON: {err}"))?; + let parsed = serde_json::from_slice::(body); if !(200..300).contains(&status) { let error = parsed - .get("error") + .as_ref() + .ok() + .and_then(|value| value.get("error")) .and_then(Value::as_str) - .unwrap_or("Browser VM guest control returned an error") - .to_string(); - let mut message = format!("Browser VM guest control HTTP {status}: {error}"); - if let Some(logs) = parsed.get("logs") { - let logs_text = serde_json::to_string(logs) - .unwrap_or_else(|_| "".to_string()); - let mut bounded = logs_text.chars().take(20_000).collect::(); - if logs_text.len() > bounded.len() { - bounded.push_str("..."); - } - message.push_str(" logs="); - message.push_str(&bounded); - } - return Err(message); + .unwrap_or("Browser VM guest control returned an error"); + // Guest log tails and call logs stay behind the private control /logs endpoint. + return Err(format!( + "Browser VM guest control HTTP {status}: {}", + brief_guest_control_error(error) + )); } - Ok(parsed) + parsed.map_err(|err| format!("Browser VM guest control response is not JSON: {err}")) } -fn response_preview(response: &[u8]) -> String { - let mut preview = String::new(); - for byte in response.iter().take(160) { - let _ = write!(preview, "{byte:02x}"); - } - if response.len() > 160 { - preview.push_str("..."); +fn brief_guest_control_error(error: &str) -> String { + let first_line: String = error + .lines() + .next() + .unwrap_or_default() + .chars() + .take(MAX_GUEST_CONTROL_ERROR_CHARS) + .filter(|character| !character.is_control()) + .collect(); + let brief = first_line + .split_whitespace() + .map(|word| { + if word.contains("://") || word.starts_with("turn:") || word.starts_with("turns:") { + "[redacted URL]" + } else { + word + } + }) + .collect::>() + .join(" "); + let lower = brief.to_ascii_lowercase(); + if brief.is_empty() + || [ + "credential", + "secret", + "password", + "authorization", + "bearer ", + "home_token", + ] + .iter() + .any(|marker| lower.contains(marker)) + { + return "Browser VM guest control returned an error (details in private guest logs)" + .to_string(); } - preview + brief.chars().take(MAX_GUEST_CONTROL_ERROR_CHARS).collect() } fn spawn_control_proxy( @@ -4555,6 +4572,119 @@ mod tests { assert_eq!(runtime_to_guest, 4); } + #[test] + fn guest_control_completed_http_errors_are_terminal_and_keep_brief_cause() { + for status in [503, 400, 403, 500] { + for cause in [ + "Chromium did not accept the Runtime online-state projection", + "page.goto: net::ERR_NAME_NOT_RESOLVED", + "Connection reset while opening the page", + "Broken pipe while opening the page", + ] { + let body = json!({ + "schema": "elastos.browser.selkies-control.error/v1", + "error": cause, + "logs": { + "browser-vm-control.log": { "tail": "Connection reset; Broken pipe" }, + "browser-vm-chromium.log": { "tail": "DevTools listening on ws://127.0.0.1:9222/private-control-id" }, + "turn": { "credential": "private-turn-credential", "auth_secret": "private-turn-secret" }, + }, + }); + let response = format!("HTTP/1.1 {status} Error\r\n\r\n{body}"); + let error = parse_http_json_response(response.as_bytes()).unwrap_err(); + + assert!(!is_retryable_guest_control_open_error(&error), "{error}"); + assert_eq!( + error, + format!("Browser VM guest control HTTP {status}: {cause}") + ); + assert!(!error.contains("logs=")); + assert!(!error.contains("private-")); + assert!(!error.contains("ws://")); + } + } + } + + #[test] + fn guest_control_non_json_http_error_keeps_status_and_discards_private_body() { + assert_eq!( + parse_http_json_response(b"credential=private-value").unwrap_err(), + "Browser VM guest control returned an invalid HTTP response" + ); + for body in ["Broken pipe; credential=private-value", "{\"error\":", ""] { + let response = format!("HTTP/1.1 503 Service Unavailable\r\n\r\n{body}"); + let error = parse_http_json_response(response.as_bytes()).unwrap_err(); + + assert!( + error.starts_with("Browser VM guest control HTTP 503:"), + "{error}" + ); + assert!(!is_retryable_guest_control_open_error(&error)); + assert!(!error.contains("private-value")); + } + } + + #[test] + fn guest_control_brief_error_omits_urls_secrets_and_call_logs() { + for cause in [ + "page.goto: net::ERR_NAME_NOT_RESOLVED at http://user:private-password@localhost/main?home_token=private-token\nCall log:\ncredential=private-credential", + "page.goto: net::ERR_NAME_NOT_RESOLVED\r\nDevTools listening on ws://127.0.0.1:9222/private-control-id", + ] { + let response = format!("HTTP/1.1 503 Error\r\n\r\n{}", json!({ "error": cause })); + let error = parse_http_json_response(response.as_bytes()).unwrap_err(); + + assert!(error.contains("net::ERR_NAME_NOT_RESOLVED"), "{error}"); + assert!(!error.contains("private-")); + assert!(!error.contains("://")); + assert!(!error.contains('\n')); + assert!(!is_retryable_guest_control_open_error(&error)); + } + for cause in [ + "TURN credential=private-value", + "auth_secret=private-value", + "transport_secret=private-value", + ] { + let response = format!("HTTP/1.1 503 Error\r\n\r\n{}", json!({ "error": cause })); + let error = parse_http_json_response(response.as_bytes()).unwrap_err(); + assert!(!error.contains("private-value")); + assert!(error.contains("private guest logs")); + } + let response = format!( + "HTTP/1.1 503 Error\r\n\r\n{}", + json!({ "error": "é".repeat(4096) }) + ); + let error = parse_http_json_response(response.as_bytes()).unwrap_err(); + assert!(error.chars().count() <= 550); + } + + #[test] + fn guest_control_retry_is_limited_to_connection_startup_errors() { + for error in [ + std::io::Error::from(ErrorKind::ConnectionReset).to_string(), + std::io::Error::from(ErrorKind::BrokenPipe).to_string(), + std::io::Error::from_raw_os_error(libc::ECONNRESET).to_string(), + std::io::Error::from_raw_os_error(libc::EPIPE).to_string(), + ] { + assert!(is_retryable_guest_control_open_error(&error), "{error}"); + } + for error in [ + "Browser VM guest control HTTP 503: Connection reset; Broken pipe", + "Browser VM guest control HTTP 500: failure logs=Broken pipe", + "Browser VM guest control returned an invalid HTTP response", + "Browser VM guest control response is not JSON: Broken pipe", + "Browser VM control HTTP response timed out", + ] { + assert!(!is_retryable_guest_control_open_error(error), "{error}"); + } + } + + #[test] + fn guest_control_private_log_response_remains_available() { + let logs = json!({ "schema": "elastos.browser.selkies-control.logs/v1", "logs": { "control": "private-log-fixture" } }); + let response = format!("HTTP/1.1 200 OK\r\n\r\n{logs}"); + assert_eq!(parse_http_json_response(response.as_bytes()).unwrap(), logs); + } + #[test] fn guest_control_response_timeout_has_clear_error() { let (client, _server) = UnixStream::pair().unwrap(); From 085d49e77f067507a4dee29a2eacc72d6b8e9e46 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 23:17:41 -0500 Subject: [PATCH 088/535] fix(browser): release completed transport bootstrap promptly --- scripts/browser-vm-vz-transport-bootstrap.mjs | 8 ++- ...browser-vm-vz-transport-bootstrap.test.mjs | 67 +++++++++++++++++++ 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/scripts/browser-vm-vz-transport-bootstrap.mjs b/scripts/browser-vm-vz-transport-bootstrap.mjs index 24018d0e9..ce4a256d6 100644 --- a/scripts/browser-vm-vz-transport-bootstrap.mjs +++ b/scripts/browser-vm-vz-transport-bootstrap.mjs @@ -422,7 +422,9 @@ function readDescriptor(socketPath) { const line = payload .subarray(0, payload.indexOf(0x0a)) .toString("utf8"); - resolve({ socket, request: JSON.parse(line) }); + const request = JSON.parse(line); + clearTimeout(timeout); + resolve({ socket, request }); } catch (error) { socket.destroy(); reject(error); @@ -473,7 +475,9 @@ async function main() { direct_network_probe_failed: true, }, }; - socket.end(`${JSON.stringify(receipt)}\n`); + // The descriptor reader is paused. Release both directions after the receipt + // flushes so the one-session relay and guest init can finish bootstrap. + socket.end(`${JSON.stringify(receipt)}\n`, () => socket.destroy()); } export { diff --git a/scripts/browser-vm-vz-transport-bootstrap.test.mjs b/scripts/browser-vm-vz-transport-bootstrap.test.mjs index 1ee254dd9..32bfffa1c 100644 --- a/scripts/browser-vm-vz-transport-bootstrap.test.mjs +++ b/scripts/browser-vm-vz-transport-bootstrap.test.mjs @@ -5,6 +5,7 @@ import net from "node:net"; import os from "node:os"; import path from "node:path"; import test from "node:test"; +import vm from "node:vm"; import { directNetworkErrorProvesUnavailable, @@ -209,3 +210,69 @@ test("bootstrap direct-network probe fails closed on a reachable target", async /indeterminate \(ECONNREFUSED\)/, ); }); + +test("bootstrap flushes its bound receipt and closes without waiting for the descriptor deadline", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "vz-bootstrap-close-")); + const socketPath = path.join(root, "relay.sock"); + const descriptor = fixture("f"); + const sockets = new Set(); + let data = ""; + let peerEnded; + const ended = new Promise(resolve => { peerEnded = resolve; }); + let clientClosed; + const closed = new Promise(resolve => { clientClosed = resolve; }); + // Match the relay/host: receive until EOF while keeping its write side open. + const server = net.createServer({ allowHalfOpen: true }, socket => { + sockets.add(socket); + socket.on("data", chunk => { data += chunk; }); + socket.on("end", peerEnded); + socket.write(`${JSON.stringify({ schema: "elastos.browser.vz-transport-bootstrap/v1", ...descriptor })}\n`); + }); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(socketPath, resolve); + }); + const source = fs.readFileSync(new URL("./browser-vm-vz-transport-bootstrap.mjs", import.meta.url), "utf8"); + const program = source.slice(source.indexOf("function readDescriptor("), source.indexOf("\nexport {")); + const deadlineTimers = new Set(); + const context = vm.createContext({ + Buffer, + JSON, + net: { createConnection(options) { + const socket = net.createConnection(options); + sockets.add(socket); + socket.once("close", clientClosed); + return socket; + } }, + MAX_DESCRIPTOR_BYTES: 64 * 1024, + REQUEST_SCHEMA: "elastos.browser.vz-transport-bootstrap/v1", + RECEIPT_SCHEMA: "elastos.browser.vz-transport-bootstrap-receipt/v1", + setTimeout(callback, delay) { const timer = setTimeout(callback, delay); deadlineTimers.add(timer); return timer; }, + clearTimeout(timer) { clearTimeout(timer); deadlineTimers.delete(timer); }, + readConfig: () => ({ relay_socket_path: socketPath, + authority_path: path.join(root, "authority.json"), ice_servers_path: path.join(root, "ice.json") }), + exactObjectKeys: (value, keys) => Object.keys(value).length === keys.length && keys.every(key => Object.hasOwn(value, key)), + validateAuthority, validateSecret, writeOwnerOnlyAtomic, + guestNetworkState: () => ({ interfaces: ["lo"], default_route_absent: true }), + proveDirectNetworkUnavailable: async () => true, + }); + let failureTimer; + try { + await vm.runInContext(`${program}\nmain()`, context); + await Promise.race([Promise.all([ended, closed]), new Promise((_, reject) => { + failureTimer = setTimeout(() => reject(new Error("bootstrap connection remains open after receipt write")), 500); + })]); + const receipt = JSON.parse(data); + assert.equal(receipt.schema, "elastos.browser.vz-transport-bootstrap-receipt/v1"); + assert.equal(receipt.binding_hash, descriptor.authority.binding_hash); + assert.equal(receipt.generation, descriptor.authority.generation); + assert.equal(receipt.terminal, true); + assert.equal(deadlineTimers.size, 0, "a completed descriptor has no pending read deadline"); + } finally { + clearTimeout(failureTimer); + for (const timer of deadlineTimers) clearTimeout(timer); + for (const socket of sockets) socket.destroy(); + await new Promise(resolve => server.close(resolve)); + fs.rmSync(root, { recursive: true, force: true }); + } +}); From 8c09a6db756df2ad4af33c7e16eca4502ef63ddd Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 23:31:15 -0500 Subject: [PATCH 089/535] fix(browser): read bootstrap receipts at their frame boundary --- .../src/bin/browser-vz-engine-supervisor.rs | 41 +++++++++++++++++-- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs index 681aaeb84..d8917cd63 100644 --- a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs +++ b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs @@ -2746,10 +2746,17 @@ async fn bootstrap_vz_transport( .write_all(&bytes) .and_then(|_| stream.flush()) .map_err(|err| format!("Browser VZ transport bootstrap write failed: {err}"))?; + let receipt = read_vz_transport_bootstrap_receipt(&mut stream)?; + validate_vz_transport_bootstrap_receipt(&receipt, &transport.authority)?; + Ok(receipt) +} + +fn read_vz_transport_bootstrap_receipt(stream: &mut impl Read) -> Result { let mut response = Vec::new(); - stream - .take(64 * 1024 + 1) - .read_to_end(&mut response) + // Bootstrap is one bounded JSON line. The peer may keep its write side + // open until we release the connection after validating that receipt. + BufReader::new(stream.take(64 * 1024 + 1)) + .read_until(b'\n', &mut response) .map_err(|err| format!("Browser VZ transport bootstrap read failed: {err}"))?; if response.len() > 64 * 1024 { return Err("Browser VZ transport bootstrap receipt is too large".to_string()); @@ -2761,7 +2768,6 @@ async fn bootstrap_vz_transport( .ok_or_else(|| "Browser VZ transport bootstrap receipt is empty".to_string())?, ) .map_err(|err| format!("Browser VZ transport bootstrap receipt is invalid JSON: {err}"))?; - validate_vz_transport_bootstrap_receipt(&receipt, &transport.authority)?; Ok(receipt) } @@ -4572,6 +4578,33 @@ mod tests { assert_eq!(runtime_to_guest, 4); } + #[test] + fn bootstrap_receipt_completes_while_the_peer_write_side_stays_open() { + let (mut client, mut peer) = UnixStream::pair().unwrap(); + client + .set_read_timeout(Some(Duration::from_millis(100))) + .unwrap(); + let receipt = json!({"schema": "elastos.browser.vz-transport-bootstrap-receipt/v1"}); + writeln!(peer, "{receipt}").unwrap(); + assert_eq!( + read_vz_transport_bootstrap_receipt(&mut client).unwrap(), + receipt + ); + } + + #[test] + fn bootstrap_receipt_read_rejects_empty_malformed_and_oversized_frames() { + for bytes in [b"".as_slice(), b"\n", b"{broken}\n"] { + assert!(read_vz_transport_bootstrap_receipt(&mut std::io::Cursor::new(bytes)).is_err()); + } + let oversized = vec![b'x'; 64 * 1024 + 1]; + assert!( + read_vz_transport_bootstrap_receipt(&mut std::io::Cursor::new(oversized)) + .unwrap_err() + .contains("too large") + ); + } + #[test] fn guest_control_completed_http_errors_are_terminal_and_keep_brief_cause() { for status in [503, 400, 403, 500] { From 6095381280ea23d16159007c4c06578723038897 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 23:39:41 -0500 Subject: [PATCH 090/535] test(browser): reject cleanup claims from an unowned bridge --- .../browser-selkies-control-service-smoke.sh | 45 ++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/scripts/browser-selkies-control-service-smoke.sh b/scripts/browser-selkies-control-service-smoke.sh index 31bda5479..26c8622c6 100755 --- a/scripts/browser-selkies-control-service-smoke.sh +++ b/scripts/browser-selkies-control-service-smoke.sh @@ -2,6 +2,13 @@ set -euo pipefail repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +control_only=0 +if [[ "$#" == 1 && "$1" == "--control-only" ]]; then + control_only=1 +elif [[ "$#" != 0 ]]; then + echo "usage: browser-selkies-control-service-smoke.sh [--control-only]" >&2 + exit 2 +fi tmp_dir="$(mktemp -d)" selkies_pid="" cdp_pid="" @@ -2344,10 +2351,46 @@ if (response.accepted !== true || response.direct_network !== false) throw new E if (inserted !== "Paste Text 123") throw new Error(`paste_text did not use CDP Input.insertText: ${inserted}`); ' "$paste_response" "$tmp_dir/fake-cdp-ready.json.inserted-text" +if [[ "$control_only" == 1 ]]; then + printf '%s\n' '{"schema":"elastos.browser.selkies-control-service-smoke/v1","ok":true,"scope":"control-service","adapter_cleanup_verified":false}' + exit 0 +fi + +# The shared control bridge owns page channels. This smoke owns Chromium, +# Selkies, and the proxy, so the bridge cannot certify their terminal cleanup. +# Keep the product display checks and require the adapter to reject that +# incomplete close response instead of inventing a supervisor receipt here. +preflight_status=0 scripts/browser-selkies-target-preflight.sh \ --out-dir "$tmp_dir/target-preflight" \ --control-socket "$tmp_dir/target-preflight.sock" \ --runtime-fetch-proxy-url "http://127.0.0.1:$runtime_proxy_port" \ --selkies-ws-url "ws://127.0.0.1:$selkies_port/signaling" \ --browser-cdp-endpoint "http://127.0.0.1:$cdp_port" \ - --ice-server "stun:stun.example.invalid:3478" >/dev/null + --ice-server "stun:stun.example.invalid:3478" \ + >"$tmp_dir/target-preflight.stdout" 2>"$tmp_dir/target-preflight.stderr" || preflight_status=$? +if ! "$node_bin" - "$preflight_status" "$tmp_dir/target-preflight.stderr" <<'NODE' +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const [status, stderrPath] = process.argv.slice(2); +const stderr = fs.readFileSync(stderrPath, "utf8"); +// The unchanged display smoke reports each failed response as JSON and throws +// its first failure after shutdown. Require close to be the only failing gate. +const responses = [...stderr.matchAll(/^\{[\s\S]*?^\}/gm)].map(match => JSON.parse(match[0])); +assert.equal(Number(status), 1, "shared bridge preflight must fail at terminal cleanup"); +assert.deepEqual(responses, [{ + status: "error", + code: "engine_close_indeterminate", + message: "Browser supervisor did not return an exact typed terminal cleanup receipt", +}]); +assert.deepEqual(stderr.match(/^(?:\w*Error):[^\n]*/gm), ["Error: adapter close_page failed"], + "launch, display validation, and shutdown must pass before accepting the expected close rejection"); +NODE +then + cat "$tmp_dir/target-preflight.stdout" "$tmp_dir/target-preflight.stderr" >&2 + exit 1 +fi +for fixture_pid in "$selkies_pid" "$cdp_pid" "$proxy_pid"; do + kill -0 "$fixture_pid" +done +printf '%s\n' '{"schema":"elastos.browser.selkies-control-service-smoke/v1","ok":true,"scope":"control-service-and-adapter-close-rejection","unowned_cleanup_rejected":true,"adapter_cleanup_verified":false}' From 98ab05a748771ed9e5c542f4cba30bfde5bcde69 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Mon, 7 Sep 2026 23:40:37 -0500 Subject: [PATCH 091/535] feat(browser): record bounded guest bootstrap stage timings --- scripts/browser-vm-vz-transport-bootstrap.mjs | 18 +++++++++++++++++- .../browser-vm-vz-transport-bootstrap.test.mjs | 11 +++++++++++ scripts/build/stage-browser-vm-target.sh | 4 ++++ 3 files changed, 32 insertions(+), 1 deletion(-) diff --git a/scripts/browser-vm-vz-transport-bootstrap.mjs b/scripts/browser-vm-vz-transport-bootstrap.mjs index ce4a256d6..17891eb4b 100644 --- a/scripts/browser-vm-vz-transport-bootstrap.mjs +++ b/scripts/browser-vm-vz-transport-bootstrap.mjs @@ -437,8 +437,17 @@ function readDescriptor(socketPath) { } async function main() { + const startedAt = performance.now(); + const markStage = (stage) => console.error(JSON.stringify({ + schema: "elastos.browser.vz-bootstrap-stage/v1", + stage, + at: new Date().toISOString(), + elapsed_ms: Math.round(performance.now() - startedAt), + })); const config = readConfig(); + markStage("descriptor_wait"); const { socket, request } = await readDescriptor(config.relay_socket_path); + markStage("descriptor_received"); if ( !exactObjectKeys(request, ["schema", "authority", "secret"]) || request.schema !== REQUEST_SCHEMA @@ -448,8 +457,11 @@ async function main() { const authority = validateAuthority(request.authority); const secret = validateSecret(authority, request.secret); const network = guestNetworkState(); + markStage("authority_validated"); await proveDirectNetworkUnavailable(authority); + markStage("direct_network_checked"); writeOwnerOnlyAtomic(config.authority_path, authority); + markStage("authority_written"); writeOwnerOnlyAtomic(config.ice_servers_path, [ { urls: [authority.turn.guest_url], @@ -457,6 +469,7 @@ async function main() { credential: secret.credential, }, ]); + markStage("ice_written"); const receipt = { schema: RECEIPT_SCHEMA, binding_hash: authority.binding_hash, @@ -477,7 +490,10 @@ async function main() { }; // The descriptor reader is paused. Release both directions after the receipt // flushes so the one-session relay and guest init can finish bootstrap. - socket.end(`${JSON.stringify(receipt)}\n`, () => socket.destroy()); + socket.end(`${JSON.stringify(receipt)}\n`, () => { + markStage("receipt_flushed"); + socket.destroy(); + }); } export { diff --git a/scripts/browser-vm-vz-transport-bootstrap.test.mjs b/scripts/browser-vm-vz-transport-bootstrap.test.mjs index 32bfffa1c..868163de5 100644 --- a/scripts/browser-vm-vz-transport-bootstrap.test.mjs +++ b/scripts/browser-vm-vz-transport-bootstrap.test.mjs @@ -235,9 +235,12 @@ test("bootstrap flushes its bound receipt and closes without waiting for the des const source = fs.readFileSync(new URL("./browser-vm-vz-transport-bootstrap.mjs", import.meta.url), "utf8"); const program = source.slice(source.indexOf("function readDescriptor("), source.indexOf("\nexport {")); const deadlineTimers = new Set(); + const stages = []; const context = vm.createContext({ Buffer, JSON, + performance, + console: { error: (line) => stages.push(JSON.parse(line)) }, net: { createConnection(options) { const socket = net.createConnection(options); sockets.add(socket); @@ -268,6 +271,14 @@ test("bootstrap flushes its bound receipt and closes without waiting for the des assert.equal(receipt.generation, descriptor.authority.generation); assert.equal(receipt.terminal, true); assert.equal(deadlineTimers.size, 0, "a completed descriptor has no pending read deadline"); + assert.deepEqual(stages.map(event => event.stage), ["descriptor_wait", "descriptor_received", + "authority_validated", "direct_network_checked", "authority_written", "ice_written", "receipt_flushed"]); + for (const event of stages) { + assert.deepEqual(Object.keys(event).sort(), ["at", "elapsed_ms", "schema", "stage"]); + assert.equal(event.schema, "elastos.browser.vz-bootstrap-stage/v1"); + assert.ok(Number.isInteger(event.elapsed_ms) && event.elapsed_ms >= 0); + assert.ok(Number.isFinite(Date.parse(event.at))); + } } finally { clearTimeout(failureTimer); for (const timer of deadlineTimers) clearTimeout(timer); diff --git a/scripts/build/stage-browser-vm-target.sh b/scripts/build/stage-browser-vm-target.sh index c6c5064bb..db92c7374 100755 --- a/scripts/build/stage-browser-vm-target.sh +++ b/scripts/build/stage-browser-vm-target.sh @@ -629,8 +629,12 @@ JSON echo "browser-vm-init: VZ transport bootstrap relay did not start" >&2 exit 1 } + rootfs_mark "VZ bootstrap helper starting at $(date -u +%FT%TZ)" + rootfs_mark "VZ bootstrap entropy_bits=$(cat /proc/sys/kernel/random/entropy_avail 2>/dev/null || true) rng=$(cat /sys/class/misc/hw_random/rng_current 2>/dev/null || true)" ELASTOS_BROWSER_VM_VZ_TRANSPORT_BOOTSTRAP_CONFIG='{"schema":"elastos.browser.vz-transport-bootstrap.config/v1","relay_socket_path":"/run/elastos/browser-vz-transport-bootstrap.sock","authority_path":"/run/elastos/browser-vz-transport-authority.json","ice_servers_path":"/run/elastos/browser-ice-servers.json"}' \ /opt/elastos/bin/node /opt/elastos/bin/browser-vm-vz-transport-bootstrap.mjs + rootfs_mark "VZ bootstrap helper exited at $(date -u +%FT%TZ)" + rootfs_mark "VZ bootstrap entropy_bits=$(cat /proc/sys/kernel/random/entropy_avail 2>/dev/null || true) rng=$(cat /sys/class/misc/hw_random/rng_current 2>/dev/null || true)" wait "$ELASTOS_BROWSER_VM_BOOTSTRAP_RELAY_PID" read_vz_authority_field() { /opt/elastos/bin/node -e ' From 9923983c5d5d970874d8ec35496e241b0f2d35b6 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 8 Sep 2026 00:07:09 -0500 Subject: [PATCH 092/535] test(browser): prove the installed Home journey and exact cleanup --- scripts/home-passkey-virtual-auth-smoke.mjs | 550 +++++++++++++++++-- scripts/lib/browser-journey-fixture.mjs | 144 +++++ scripts/lib/browser-journey-harness.test.mjs | 447 +++++++++++++++ 3 files changed, 1089 insertions(+), 52 deletions(-) create mode 100644 scripts/lib/browser-journey-fixture.mjs create mode 100644 scripts/lib/browser-journey-harness.test.mjs diff --git a/scripts/home-passkey-virtual-auth-smoke.mjs b/scripts/home-passkey-virtual-auth-smoke.mjs index 530f33acc..8aa811fc3 100644 --- a/scripts/home-passkey-virtual-auth-smoke.mjs +++ b/scripts/home-passkey-virtual-auth-smoke.mjs @@ -12,6 +12,7 @@ import { import { tmpdir } from "node:os"; import { join } from "node:path"; import { createRequire } from "node:module"; +import { randomUUID } from "node:crypto"; const require = createRequire(new URL("../elastos/tools/browser-playwright-engine/package.json", import.meta.url)); const { chromium } = require("playwright"); @@ -143,6 +144,11 @@ const BROWSER_UI_CLICK_TARGET_TIMEOUT_MS = parseBoundedIntegerEnv( ); const CHECK_BROWSER_EMBEDDED_UI_INPUT = process.env.HOME_VIRTUAL_AUTH_BROWSER_EMBEDDED_UI_INPUT === "1"; +const CHECK_BROWSER_CONTROLLED_JOURNEY = + process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_JOURNEY === "1"; +const REQUIRE_BROWSER_VZ_TRANSPORT = process.env.HOME_VIRTUAL_AUTH_BROWSER_REQUIRE_VZ_TRANSPORT === "1"; +const BROWSER_JOURNEY_FIXTURE_ORIGIN = process.env.HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ORIGIN || + "http://localhost:61511"; const CHECK_BROWSER_EMBEDDED_RECOVERY = process.env.HOME_VIRTUAL_AUTH_BROWSER_EMBEDDED_RECOVERY === "1"; const BROWSER_OPEN_URLS = parseBrowserOpenUrls(process.env.HOME_VIRTUAL_AUTH_BROWSER_OPEN_URLS); @@ -164,6 +170,9 @@ let smokeStage = "init"; function markStage(stage) { smokeStage = stage; + if (CHECK_BROWSER_CONTROLLED_JOURNEY) { + console.error(JSON.stringify({ stage, at: new Date().toISOString() })); + } } function readVirtualAuthenticatorCredentialStore() { @@ -311,7 +320,8 @@ function redactSensitive(value) { return Object.fromEntries( Object.entries(value).map(([key, entry]) => [ key, - ["credential", "auth_secret", "transport_secret"].includes(key) + ["credential", "auth_secret", "transport_secret"].includes(key) || + ["home_token", "homeToken"].includes(key) ? "[redacted]" : redactSensitive(entry), ]), @@ -1995,8 +2005,383 @@ async function holdBrowserUiForSetup(context, browserToken, route) { } } -async function checkBrowserEmbeddedUiInput(page) { +async function waitForEmbeddedBrowserPage(appFrame, failures, ignoredOpenIds = new Set()) { + const deadline = Date.now() + BROWSER_UI_PAGE_ID_TIMEOUT_MS; + while (Date.now() < deadline) { + const failure = failures.find(entry => entry.frame === appFrame && + !(entry.open_id && ignoredOpenIds.has(entry.open_id))); + if (failure) { + throw Object.assign(new Error("Embedded Browser startup returned a failed open settlement"), { + details: { stage: smokeStage, status: failure.status, settlement: failure.body }, + }); + } + const pageId = await appFrame.evaluate(() => window.__elastosBrowserCurrentPageId || ""); + if (pageId) return pageId; + await delay(200); + } + throw new Error("Embedded Browser UI did not publish the current page id before timeout"); +} + +async function waitForJourneyEvidence(read, predicate, label, timeoutMs = 30_000) { + const deadline = performance.now() + timeoutMs; + let last; + while (performance.now() < deadline) { + last = await read(); + if (predicate(last)) return last; + await delay(200); + } + throw Object.assign(new Error(`Controlled Browser journey timed out: ${label}`), { details: { last } }); +} + +async function closeControlledBrowserWindow(page, appFrame, windowLocator, token, baseline, { expectedPageId = null } = {}) { + markStage("browser:ui-close"); + // The opaque Home GUI survives Browser frame removal and supplies Origin: null. + const apiFrame = appFrame.parentFrame(); + const instance = new URL(appFrame.url()).searchParams.get("browser_instance") || ""; + const summaryPath = `/api/apps/browser/summary?browser_instance=${encodeURIComponent(instance)}`; + let before = await browserApi(apiFrame, token, summaryPath); + if (!before.body?.sessions?.recoverable_page) { + // Browser owns an in-flight open until its existing settlement finishes. + await appFrame.waitForFunction(() => document.body?.dataset.loading !== "true", null, + { timeout: BROWSER_UI_PAGE_ID_TIMEOUT_MS }).catch(() => null); + before = await browserApi(apiFrame, token, summaryPath); + } + const owner = before.body?.sessions?.recoverable_page; + assert(before.ok, "Browser close summary failed", before); + const started = performance.now(); + const evidence = { messages: [], frames: [], close_responses: [], dropped_messages: 0, dropped_frames: 0, dropped_responses: 0 }; + const safeText = value => redactSensitiveString(typeof value === "string" ? value : "") + .split(token || "\0").join("[redacted]").slice(0, 256); + const append = (list, dropped, value) => { + if (list.length === 32) { list.shift(); evidence[dropped]++; } + list.push(value); + }; + const frameEvent = (frame, event) => { + if (frame !== appFrame && frame !== apiFrame) return; + const url = new URL(frame.url()); + append(evidence.frames, "dropped_frames", { event, at_ms: Math.round(performance.now() - started), + frame: frame === appFrame ? "browser" : "home", url: safeText(`${url.origin}${url.pathname}`), + instance_matches: url.searchParams.get("browser_instance") === instance }); + }; + const navigated = frame => frameEvent(frame, "navigated"); + const detached = frame => frameEvent(frame, "detached"); + // Install in Home before clicking. Home survives removal/navigation of the Browser document. + const messages = await apiFrame.evaluateHandle(({ sourceFrame, instance, token }) => { + const source = sourceFrame.contentWindow; + const events = []; + const started = performance.now(); + let dropped = 0; + let pending = null; + let settle; + const terminal = new Promise(resolve => { settle = resolve; }); + const listener = event => { + const data = event.data; + if (event.source !== source || event.origin !== "null" || data?.homeToken !== token || + data?.type !== "elastos.browser.window-close.result/v1" || + data.browserInstance !== instance || !["pending", "error", "terminal"].includes(data.state)) return; + const safe = value => { + let text = typeof value === "string" ? value : ""; + for (const secret of [token, data.homeToken]) if (typeof secret === "string" && secret) text = text.split(secret).join("[redacted]"); + return text.replace(/https?:\/\/[^\s]+/gi, "[url]") + .replace(/home_token=[^&\s]+/gi, "home_token=[redacted]").slice(0, 256); + }; + if (events.length === 32) { events.shift(); dropped++; } + events.push({ at_ms: Math.round(performance.now() - started), requestId: safe(data.requestId), + state: data.state, reason: safe(data.reason), terminalKind: safe(data.terminalKind), + pageId: safe(data.pageId), cleanupId: safe(data.cleanupId), + generation: Number.isSafeInteger(data.generation) ? data.generation : null }); + const message = { request_id: data.requestId, state: data.state, terminal_kind: data.terminalKind, + page_id: data.pageId, cleanup_id: data.cleanupId, generation: data.generation }; + if (!pending && data.state === "pending" && data.pageId && data.cleanupId) pending = message; + if (data.state === "terminal") settle({ message, pending }); + }; + const timer = setTimeout(() => settle({ timed_out: true }), 60_000); + window.addEventListener("message", listener); + return { terminal, stop: () => { + clearTimeout(timer); + window.removeEventListener("message", listener); + settle({ stopped: true }); + return { events, dropped }; + } }; + }, { sourceFrame: await appFrame.frameElement(), instance, token }); + const closeResponses = []; + const runtimeOrigin = new URL(appFrame.url()).origin; + const effects = ["page_absent", "child_absent", "vm_absent", "route_absent", "socket_absent"]; + const transportEffects = ["transport_session_absent", "turn_process_absent", "turn_listener_absent", + "turn_relay_ports_absent", "ordinary_vsock_bridge_absent", "media_vsock_bridge_absent", + "bootstrap_vsock_bridge_absent", "hibernation_state_absent"]; + const closeResponse = async response => { + try { + const request = response.request(); + const url = new URL(response.url()); + const path = url.pathname.match(/^\/api\/apps\/browser\/pages\/([^/]+)\/close$/); + if (request.frame() !== appFrame || request.method() !== "POST" || url.origin !== runtimeOrigin || !path) return; + if (closeResponses.length === 32) { evidence.dropped_responses++; return; } + const body = request.postDataJSON(); + const record = { page_id: decodeURIComponent(path[1]), request: body, receipt: null, + ok: response.ok(), authority_matches: request.headers()["x-elastos-home-token"] === token }; + closeResponses.push(record); + const detail = { at_ms: Math.round(performance.now() - started), status: response.status(), + page_id: safeText(record.page_id), authority_matches: record.authority_matches, + request: { schema: safeText(body?.schema), cleanup_id: safeText(body?.cleanup_id), + browser_instance: safeText(body?.browser_instance) }, receipt: null }; + record.detail = detail; + evidence.close_responses.push(detail); + const receipt = await response.json().catch(() => null); + record.receipt = receipt; + if (receipt) detail.receipt = { + schema: safeText(receipt.schema), page_id: safeText(receipt.page_id), cleanup_id: safeText(receipt.cleanup_id), + browser_instance: safeText(receipt.browser_instance), closed: receipt.closed === true, + already_closed: receipt.already_closed === true, + cleanup: { schema: safeText(receipt.cleanup?.schema), ok: receipt.cleanup?.ok === true, + action: safeText(receipt.cleanup?.action) }, + terminal_effects: Object.fromEntries([...effects, ...transportEffects] + .filter(key => key in (receipt.terminal_effects || {})).map(key => [key, receipt.terminal_effects[key] === true])), + transport_proof_present: Boolean(receipt.transport_proof), + }; + } catch (error) { + evidence.response_error = safeText(error.message); + } + }; + page.on("response", closeResponse); + page.on("framenavigated", navigated); + page.on("framedetached", detached); + try { + assert(!expectedPageId || owner?.page_id === expectedPageId, + "Successful Browser journey requires fresh close of its last exercised page", { expectedPageId, owner }); + if (!owner) { + const messagePromise = messages.evaluate(observer => observer.terminal) + .then(result => result, error => ({ error })); + await windowLocator.getByRole("button", { name: /^(Close|Retry Browser close)$/ }).click(); + const terminal = await messagePromise; + if (terminal.error) throw terminal.error; + assert(!terminal.timed_out, "Browser startup close remained pending"); + const message = terminal.message; + let receipt; + if (message.terminal_kind === "already_absent") { + const pending = terminal.pending; + assert(instance && pending && typeof message.request_id === "string" && message.request_id && + Number.isSafeInteger(message.generation) && message.generation >= 0 && + pending.request_id === message.request_id && pending.page_id === message.page_id && + pending.cleanup_id === message.cleanup_id && pending.generation === message.generation, + "Browser prior-page close did not bind the pending UI handle"); + const observed = await waitForJourneyEvidence(() => closeResponses.find(record => + record.page_id === message.page_id && record.request?.cleanup_id === message.cleanup_id && + record.receipt?.schema === "elastos.browser.close-result/v1" && record.receipt.already_closed === true), + value => Boolean(value), "exact Runtime receipt for the retained Browser UI handle"); + receipt = observed.receipt; + assert(observed.ok && observed.authority_matches && observed.request.schema === "elastos.browser.close-request/v2" && + observed.request.browser_instance === instance && receipt.schema === "elastos.browser.close-result/v1" && + receipt.page_id === message.page_id && receipt.cleanup_id === message.cleanup_id && receipt.browser_instance === instance && + receipt.closed === false && receipt.already_closed === true && + receipt.cleanup?.schema === "elastos.browser.runtime-session-cleanup/v1" && receipt.cleanup.ok === true && + receipt.cleanup.action === "already_absent" && receipt.terminal_effects?.page_absent === true && + Object.values(receipt.terminal_effects).every(value => value === true) && !receipt.transport_proof, + "Browser prior-page close lacks an exact Runtime tombstone receipt"); + // Runtime's tombstone proves prior settlement; it does not repeat the Engine transport receipt. + receipt = observed.detail.receipt; + } else { + assert(message.terminal_kind === "no_page" && !message.page_id && !message.cleanup_id, + "Browser startup close did not confirm absent ownership", message); + } + await windowLocator.waitFor({ state: "detached", timeout: 15_000 }); + const after = await browserApi(apiFrame, token, summaryPath); + assert(after.ok && after.body?.sessions?.recoverable_page === null && + after.body.sessions.principal_sessions <= baseline.principal_sessions && + after.body.sessions.total_sessions <= baseline.total_sessions && + after.body.sessions.launching_sessions <= baseline.launching_sessions && + after.body.sessions.engine_cleanup_obligations <= baseline.engine_cleanup_obligations && + after.body.sessions.launch_reconciliation_obligations <= baseline.launch_reconciliation_obligations, + "Browser startup close retained Runtime ownership", after); + return { startup_close: message, receipt, window_detached: true, sessions_after_close: after.body.sessions, close_evidence: evidence }; + } + assert(owner.page_id && owner.cleanup?.schema === "elastos.browser.cleanup-handle/v1", + "Controlled Browser close requires the exact Runtime page and cleanup handle", before); + const messagePromise = messages.evaluate(observer => observer.terminal) + .then(result => result, error => ({ error })); + await windowLocator.getByRole("button", { name: /^(Close|Retry Browser close)$/ }).click(); + const terminal = await messagePromise; + if (terminal.error) throw terminal.error; + assert(!terminal.timed_out, "Browser UI close remained pending"); + const { message, pending } = terminal; + assert(instance && pending && message?.terminal_kind === "closed" && + typeof message.request_id === "string" && message.request_id && + Number.isSafeInteger(message.generation) && message.generation >= 0 && + pending.request_id === message.request_id && pending.generation === message.generation && + pending.page_id === owner.page_id && message.page_id === owner.page_id && + pending.cleanup_id === owner.cleanup.id && message.cleanup_id === owner.cleanup.id, + "Browser UI close did not bind its pending and terminal ownership", { message, pending }); + const observed = await waitForJourneyEvidence(() => closeResponses.find(record => + record.page_id === owner.page_id && record.request?.cleanup_id === owner.cleanup.id && + record.receipt?.schema === "elastos.browser.close-result/v1" && record.receipt.closed === true), + value => Boolean(value), "exact Runtime receipt for the exercised Browser page"); + const { receipt, request } = observed; + assert(observed.ok && observed.authority_matches && request?.schema === "elastos.browser.close-request/v2" && + request.cleanup_id === owner.cleanup.id && receipt.cleanup_id === owner.cleanup.id && + receipt.closed === true && receipt.already_closed !== true && + receipt.page_id === owner.page_id && request.browser_instance === instance && receipt.browser_instance === instance, + "UI close returned a mismatched Runtime cleanup receipt", { request, receipt }); + if (REQUIRE_BROWSER_VZ_TRANSPORT) { + assert(receipt.transport_proof?.schema === "elastos.browser.vz-transport-public-proof/v1" && + receipt.transport_proof.page_id === owner.page_id, + "UI close requires the selected VZ transport proof", receipt); + } + const requiredEffects = receipt.transport_proof ? [...effects, ...transportEffects] : effects; + assert(receipt.cleanup?.schema === "elastos.browser.runtime-session-cleanup/v1" && + receipt.cleanup.ok === true && requiredEffects.every(key => receipt.terminal_effects?.[key] === true), + "UI close did not confirm Runtime and Engine cleanup", receipt); + await windowLocator.waitFor({ state: "detached", timeout: 15_000 }); + const after = await waitForJourneyEvidence( + () => browserApi(apiFrame, token, summaryPath), + value => value.ok && value.body?.sessions?.schema === "elastos.browser.session-capacity/v1" && + value.body.sessions.principal_sessions === baseline.principal_sessions && + value.body.sessions.total_sessions === baseline.total_sessions && + value.body.sessions.launching_sessions === baseline.launching_sessions && + value.body.sessions.engine_cleanup_obligations <= baseline.engine_cleanup_obligations && + value.body.sessions.launch_reconciliation_obligations <= baseline.launch_reconciliation_obligations && + value.body.sessions.recoverable_page === null, + "Runtime sessions and cleanup obligations return to baseline"); + markStage("browser:cleanup-confirmed"); + return { receipt, window_detached: true, sessions_before_launch: baseline, sessions_after_close: after.body.sessions, close_evidence: evidence }; + } catch (error) { + error.details = { ...error.details, close_evidence: evidence }; + throw error; + } finally { + page.off("response", closeResponse); + page.off("framenavigated", navigated); + page.off("framedetached", detached); + const collected = await messages.evaluate(observer => observer.stop()).catch(error => ({ error: safeText(error.message) })); + evidence.messages = collected.events || []; + evidence.dropped_messages = collected.dropped || 0; + if (collected.error) evidence.message_error = collected.error; + await messages.dispose().catch(() => {}); + } +} + +async function runControlledBrowserJourney(page, appFrame, windowLocator, token, baseline, failures) { + const fixture = new URL(BROWSER_JOURNEY_FIXTURE_ORIGIN); + const run = randomUUID(); + const receiptUrl = `${fixture.origin}/receipt?run=${run}`; + const result = { schema: "elastos.browser.controlled-journey/v1", run, pages: [] }; + const readReceipt = async () => { + const response = await fetch(receiptUrl, { signal: AbortSignal.timeout(5_000) }); + const body = await response.json(); + if (response.status === 404 && body.error === "unknown run") { + return { schema: "elastos.browser.journey-receipt/v1", run, events: [] }; + } + assert(response.ok && body.schema === "elastos.browser.journey-receipt/v1" && body.run === run, + "Controlled fixture receipt failed", body); + return body; + }; + let failure = null; + try { + // Ordinary address entry is available after Browser settles its current open. + // This source disables the field while opening; the harness preserves that UI rule. + markStage("browser:controlled-address-ready"); + await appFrame.locator("#browser-url").waitFor({ state: "visible", timeout: 15_000 }); + await appFrame.waitForFunction(() => document.querySelector("#browser-url")?.disabled === false, + null, { timeout: BROWSER_UI_PAGE_ID_TIMEOUT_MS }); + const settled = failures.filter(entry => { + const outcome = entry.body?.error?.outcome || entry.body?.outcome; + return entry.frame === appFrame && entry.open_id && outcome?.schema === "elastos.browser.open-outcome/v1" && + ["terminal_pre_effect_failure", "terminal_post_effect_cleanup"].includes(outcome.state); + }); + const ignoredOpenIds = new Set(settled.map(entry => entry.open_id)); + result.prior_open_settlements = settled.map(({ open_id, body }) => ({ open_id, settlement: body })); + for (const name of ["main", "nav"]) { + markStage(`browser:controlled-${name}`); + const url = `${fixture.origin}/${name}?run=${run}`; + const navigationStarted = performance.now(); + await appFrame.locator("#browser-url").fill(url); + await appFrame.locator("#browser-url").press("Enter"); + const status = await waitForJourneyEvidence(async () => { + const id = await waitForEmbeddedBrowserPage(appFrame, failures, ignoredOpenIds); + const value = await browserApi(appFrame, token, `/api/apps/browser/pages/${encodeURIComponent(id)}/status`); + return { page_id: id, ok: value.ok, ...value.body }; + }, value => value.ok && value.schema === "elastos.browser.page-status/v1" && value.actual_url === url, + `Runtime navigates to ${name}`, BROWSER_UI_PAGE_ID_TIMEOUT_MS); + const statusReadyMs = Math.round(performance.now() - navigationStarted); + assert(status.direct_network === false && status.display_session?.media_transport === "runtime_relay" && + runtimeRelayIceContractOk(status.display_session), "Controlled Browser lost Runtime relay authority", status); + await appFrame.waitForFunction(expected => document.querySelector("#browser-url")?.value === expected, + url, { timeout: 15_000 }); + const receipt = await waitForJourneyEvidence(readReceipt, + value => value.events.some(event => event.type === "load" && event.page === name), `${name} page load`); + const load = receipt.events.find(event => event.type === "load" && event.page === name); + const ready = await waitForBrowserRemoteVideo(appFrame, { browserToken: token, pageId: status.page_id, + timeoutMs: BROWSER_REMOTE_VIDEO_TIMEOUT_MS }); + const decoded = await waitForJourneyEvidence(() => browserRemoteVideoMetrics(appFrame), + value => value.decoded_frames > ready.decoded_frames && value.decoded_frames > 0, + `${name} decoded WebRTC frame progress`, BROWSER_REMOTE_VIDEO_TIMEOUT_MS); + result.pages.push({ name, url, page_id: status.page_id, load, video: { ready, decoded }, + timing: { status_ready_ms: statusReadyMs, decoded_progress_ms: Math.round(performance.now() - navigationStarted) } }); + } + const current = result.pages.at(-1); + const rect = current.load.input_rect; + const point = await remoteVideoClickPositionForPagePoint(appFrame, + { x: rect.x + rect.width / 2, y: rect.y + rect.height / 2 }); + assert(point, "Controlled Browser input could not be mapped into the decoded video"); + markStage("browser:type"); + await appFrame.locator("#browser-remote-display").click({ position: { x: point.x, y: point.y } }); + const text = `Browser-${run.slice(0, 8)}`; + let typed = ""; + for (const character of text) { + await appFrame.locator("#browser-keyboard-capture").pressSequentially(character); + typed += character; + await waitForJourneyEvidence(readReceipt, + value => value.events.some(event => event.page === "nav" && event.type === "input" && event.value === typed), + "Engine page receives typed text"); + } + markStage("browser:scroll"); + await appFrame.locator("#browser-remote-display").hover(); + await page.mouse.wheel(0, 640); + const receipt = await waitForJourneyEvidence(readReceipt, + value => value.events.some(event => event.page === "nav" && event.type === "scroll" && + event.value === text && event.scroll_y >= current.load.scroll_y + 100), "Engine page scroll movement"); + const afterInput = await waitForJourneyEvidence(() => browserRemoteVideoMetrics(appFrame), + value => value.decoded_frames > current.video.decoded.decoded_frames, "decoded frames after input"); + result.input = { text, receipt, video_after_input: afterInput }; + } catch (error) { + error.details = { stage: smokeStage, ...error.details }; + failure = error; + } finally { + try { + result.close = await closeControlledBrowserWindow(page, appFrame, windowLocator, token, baseline, + { expectedPageId: failure ? null : result.pages.at(-1)?.page_id }); + } catch (error) { + if (failure) failure.details = { ...failure.details, cleanup_error: error.message, cleanup_details: error.details }; + else failure = error; + } + } + if (failure) { + failure.details = { ...failure.details, controlled_journey: result }; + throw failure; + } + return { page_id: result.pages.at(-1).page_id, display_mode: BROWSER_OPEN_DISPLAY_MODE, controlled_journey: result }; +} + +async function checkBrowserEmbeddedUiInput(page, baselineToken) { + let baseline = null; + if (baselineToken) { + const apiFrame = await homeGuiFrameForPage(page); + const summary = await browserApi(apiFrame, baselineToken, "/api/apps/browser/summary"); + assert(summary.ok && summary.body?.sessions?.schema === "elastos.browser.session-capacity/v1", + "Controlled Browser baseline summary failed", summary); + baseline = summary.body.sessions; + } const webrtcSignals = []; + const openFailures = []; + const captureOpenFailure = async response => { + const request = response.request(); + if (!/^\/api\/apps\/browser\/open(?:\/[^/]+)?$/.test(new URL(response.url()).pathname)) return; + const body = await response.json().catch(() => null); + if ((request.method() === "POST" && !response.ok()) || + (body?.schema === "elastos.browser.open-status/v1" && body.status === "failed")) { + openFailures.push({ frame: request.frame(), open_id: body?.open_id || "", + at: Date.now(), status: response.status(), body }); + if (openFailures.length > 8) openFailures.shift(); + } + }; const captureWebrtcResponse = async (response) => { const request = response.request(); if (request.method() !== "POST" || !response.url().includes("/webrtc")) { @@ -2022,32 +2407,32 @@ async function checkBrowserEmbeddedUiInput(page) { }); }; page.on("response", captureWebrtcResponse); - await page.goto(HOME_URL, { waitUntil: "domcontentloaded" }); - await waitForSignedHome(page); - const homeGuiFrame = await waitForCapsuleFrame(page, "home-gui"); - const browserShortcut = homeGuiFrame.locator( - '#desktop-shortcuts .desktop-shortcut[data-target="browser"]', - ); - await browserShortcut.waitFor({ state: "visible", timeout: 30_000 }); - await browserShortcut.dblclick(); - - const windowLocator = homeGuiFrame.locator('.window[data-target="browser"]:not(.hidden)').first(); - await windowLocator.waitFor({ state: "visible", timeout: 30_000 }); - await homeGuiFrame.waitForFunction(() => { - const node = [...document.querySelectorAll('.window[data-target="browser"]')] - .find((candidate) => !candidate.classList.contains("hidden")); - return node?.classList.contains("window-active") && - getComputedStyle(node.querySelector(".window-frame")).pointerEvents === "auto"; - }, null, { timeout: 10_000 }); - const frameHandle = await windowLocator.locator("iframe.window-frame").elementHandle(); - assert(frameHandle, "Home Browser window did not contain an iframe"); - const route = await frameHandle.getAttribute("src") || ""; - const browserToken = assertIsolatedLaunchRoute(route, "browser"); - const appFrame = await frameHandle.contentFrame(); - assert(appFrame, "Embedded Browser iframe did not expose a content frame"); - + page.on("response", captureOpenFailure); + markStage("browser:home-launch"); + let appFrame = null; + let windowLocator = null; + let browserToken = ""; let pageId = ""; + let controlledAttemptStarted = false; + let primaryFailure = null; try { + appFrame = await openDesktopAppWindow(page, "browser"); + const homeGuiFrame = await waitForCapsuleFrame(page, "home-gui"); + windowLocator = homeGuiFrame.locator('section.window[data-target="browser"]').last(); + await windowLocator.waitFor({ state: "visible", timeout: 30_000 }); + await homeGuiFrame.waitForFunction(() => { + const node = [...document.querySelectorAll('section.window[data-target="browser"]')].at(-1); + return node?.classList.contains("window-active") && + getComputedStyle(node.querySelector(".window-frame")).pointerEvents === "auto"; + }, null, { timeout: 10_000 }); + const frameHandle = await windowLocator.locator("iframe.window-frame").elementHandle(); + assert(frameHandle, "Home Browser window did not contain an iframe"); + const route = await frameHandle.getAttribute("src") || ""; + browserToken = assertIsolatedLaunchRoute(route, "browser"); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) { + controlledAttemptStarted = true; + return await runControlledBrowserJourney(page, appFrame, windowLocator, browserToken, baseline, openFailures); + } await appFrame.evaluate(() => { window.__elastosBrowserSmokeClicks = []; const panel = document.querySelector("#browser-render-panel"); @@ -2060,15 +2445,10 @@ async function checkBrowserEmbeddedUiInput(page) { }); }, { capture: true }); }); - pageId = await appFrame.waitForFunction( - () => window.__elastosBrowserCurrentPageId || "", - null, - { timeout: BROWSER_UI_PAGE_ID_TIMEOUT_MS }, - ).then((handle) => handle.jsonValue()).catch(async (error) => { - throw Object.assign(new Error("Embedded Browser UI did not publish the current page id before timeout"), { - details: await embeddedBrowserDebugState(windowLocator, appFrame), - cause: error, - }); + markStage("browser:page-acquisition"); + pageId = await waitForEmbeddedBrowserPage(appFrame, openFailures).catch(async error => { + error.details = { ...error.details, ui: await embeddedBrowserDebugState(windowLocator, appFrame) }; + throw error; }); assert(pageId, "Embedded Browser UI did not publish the current page id"); const panelBox = await appFrame.locator("#browser-render-panel").boundingBox(); @@ -2078,7 +2458,7 @@ async function checkBrowserEmbeddedUiInput(page) { const initialGeometry = await embeddedBrowserGeometry(windowLocator, appFrame); assertEmbeddedBrowserAspect(initialGeometry); if (BROWSER_OPEN_DISPLAY_MODE === "webrtc_remote_display") { - const initialStatus = await checkBrowserPageStatus(page, browserToken, pageId); + const initialStatus = await checkBrowserPageStatus(appFrame, browserToken, pageId); const displaySession = initialStatus.display_session || {}; assert( displaySession.media_transport === "runtime_relay", @@ -2114,7 +2494,7 @@ async function checkBrowserEmbeddedUiInput(page) { let videoClickY = Math.max(1, Math.min(videoBox.height - 1, BROWSER_INPUT_CLICK_Y)); if (BROWSER_UI_CLICK_HREF_RE) { const targetProof = await waitForBrowserHrefClickTarget( - page, + appFrame, browserToken, pageId, BROWSER_UI_CLICK_HREF_RE, @@ -2171,7 +2551,7 @@ async function checkBrowserEmbeddedUiInput(page) { BROWSER_UI_CLICK_NAV_TIMEOUT_MS, ).catch(async (error) => { const runtimeStatus = await checkBrowserPageStatus( - page, + appFrame, browserToken, pageId, ).catch((statusError) => ({ @@ -2186,7 +2566,7 @@ async function checkBrowserEmbeddedUiInput(page) { }; throw error; }); - const status = await checkBrowserPageStatus(page, browserToken, pageId); + const status = await checkBrowserPageStatus(appFrame, browserToken, pageId); clickInput = normalizeRemoteDisplayClickInputEvidence( clickInput, status, @@ -2221,7 +2601,7 @@ async function checkBrowserEmbeddedUiInput(page) { assert(inputBody.accepted === true, "Embedded Browser WebRTC navigation was not accepted", inputBody); assert(inputBody.direct_network === false, "Embedded Browser WebRTC navigation reported direct network", inputBody); const navStatus = await waitForBrowserPageStatus( - page, + appFrame, browserToken, pageId, (status) => status.actual_url === BROWSER_UI_NAV_URL, @@ -2281,7 +2661,7 @@ async function checkBrowserEmbeddedUiInput(page) { if (CHECK_BROWSER_EMBEDDED_RECOVERY) { const oldPageId = pageId; const closed = await browserApi( - page, + appFrame, browserToken, `/api/apps/browser/pages/${encodeURIComponent(oldPageId)}/close`, { method: "POST", body: {} }, @@ -2394,11 +2774,36 @@ async function checkBrowserEmbeddedUiInput(page) { }, dom_clicks: clicks.slice(-3), }; + } catch (error) { + primaryFailure = error; + console.error(JSON.stringify(redactSensitive({ + stage: smokeStage, + event: "journey-failed-before-cleanup", + error: error.message, + stack: error.stack, + }))); + throw error; } finally { page.off("response", captureWebrtcResponse); - if (pageId && browserToken) { + if (primaryFailure && !controlledAttemptStarted) { + try { + if (!appFrame || !windowLocator || !browserToken) { + const gui = await homeGuiFrameForPage(page); + windowLocator = gui.locator('section.window[data-target="browser"]').last(); + const frame = await windowLocator.locator("iframe.window-frame").elementHandle(); + appFrame = frame ? await frame.contentFrame() : null; + browserToken = assertIsolatedLaunchRoute(appFrame?.url() || "", "browser"); + } + const cleanup = await closeControlledBrowserWindow(page, appFrame, windowLocator, browserToken, baseline); + primaryFailure.details = { ...primaryFailure.details, startup_cleanup: cleanup }; + } catch (error) { + primaryFailure.details = { ...primaryFailure.details, cleanup_error: error.message, cleanup_details: error.details }; + } + } + page.off("response", captureOpenFailure); + if (!CHECK_BROWSER_CONTROLLED_JOURNEY && !primaryFailure && pageId && browserToken) { const closed = await browserApi( - page, + appFrame, browserToken, `/api/apps/browser/pages/${encodeURIComponent(pageId)}/close`, { method: "POST", body: {} }, @@ -2659,7 +3064,11 @@ async function homeGuiFrameForPage(page) { } async function openDesktopAppWindow(page, target) { - await page.goto(HOME_URL, { waitUntil: "domcontentloaded" }); + const current = new URL(page.url()); + const home = new URL(HOME_URL); + if (current.origin !== home.origin || current.pathname !== home.pathname) { + await page.goto(HOME_URL, { waitUntil: "domcontentloaded" }); + } await waitForSignedHome(page); const homeGuiFrame = await waitForCapsuleFrame(page, "home-gui"); const setupLater = homeGuiFrame.getByRole("button", { name: "Later", exact: true }); @@ -2671,7 +3080,14 @@ async function openDesktopAppWindow(page, target) { await setupLater.click(); } const shelfItem = homeGuiFrame.locator(`#taskbar-targets [data-target="${target}"]`).first(); - if (await shelfItem.isVisible()) { + const existingWindow = homeGuiFrame.locator(`section.window[data-target="${target}"]`).last(); + const alreadyActive = await existingWindow.isVisible() && + await existingWindow.evaluate(node => node.classList.contains("window-active")); + // A Shelf click minimizes an active window. Its title can also be hidden in + // fullscreen, so reuse active content without clicking the hidden chrome. + if (alreadyActive) { + // The requested window is already the active user surface. + } else if (await shelfItem.isVisible()) { await shelfItem.click(); } else { await homeGuiFrame.locator("#launcher-toggle").click(); @@ -3188,7 +3604,13 @@ async function checkBrowserLaunchGrant(page, homeToken) { launched.body.browser_ui_input = await checkBrowserUiInput(page.context(), browserToken, route); } if (CHECK_BROWSER_EMBEDDED_UI_INPUT) { - launched.body.browser_embedded_ui_input = await checkBrowserEmbeddedUiInput(page); + launched.body.browser_embedded_ui_input = await checkBrowserEmbeddedUiInput(page, browserToken); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) { + console.error(JSON.stringify(redactSensitive({ + stage: "browser:controlled-journey-result", + result: launched.body.browser_embedded_ui_input, + }))); + } } if (OPEN_BROWSER) { assert( @@ -3420,7 +3842,8 @@ async function checkBrowserLaunchGrant(page, homeToken) { close_results: closeResults, }; } else if (CHECK_BROWSER_SUMMARY) { - const summary = await browserApi(page, browserToken, "/api/apps/browser/summary"); + const summaryFrame = CHECK_BROWSER_CONTROLLED_JOURNEY ? await homeGuiFrameForPage(page) : page; + const summary = await browserApi(summaryFrame, browserToken, "/api/apps/browser/summary"); assert(summary.ok, "Browser summary failed", summary); assert( summary.body?.sessions?.schema === "elastos.browser.session-capacity/v1", @@ -3630,6 +4053,20 @@ async function revokeCurrentPasskey(page, proofBindingId, homeToken) { } async function main() { + assert(!CHECK_BROWSER_CONTROLLED_JOURNEY || (INCLUDE_BROWSER && CHECK_BROWSER_EMBEDDED_UI_INPUT && + !OPEN_BROWSER && !CHECK_BROWSER_UI_INPUT && !CHECK_BROWSER_UI_SETUP && !CHECK_BROWSER_VIEWER_PREFLIGHT), + "Controlled journey requires BROWSER=1 and BROWSER_EMBEDDED_UI_INPUT=1, with other Browser runs disabled"); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) { + const fixture = new URL(BROWSER_JOURNEY_FIXTURE_ORIGIN); + assert(isLoopbackUrl(fixture.href) && fixture.protocol === "http:" && + !fixture.username && !fixture.password && fixture.pathname === "/" && !fixture.search && !fixture.hash, + "Browser fixture requires a loopback HTTP origin with its exact authorized port"); + markStage("browser:fixture-preflight"); + const response = await fetch(`${fixture.origin}/health`, { signal: AbortSignal.timeout(5_000) }); + const health = await response.json(); + assert(response.ok && health.schema === "elastos.browser.journey-fixture/v1" && health.ok === true, + "Controlled Browser fixture is unavailable", health); + } if (!ALLOW_REMOTE) { assert( isLoopbackUrl(HOME_URL), @@ -3643,6 +4080,7 @@ async function main() { ); } + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:viewer-launch"); const context = await chromium.launchPersistentContext(PROFILE_DIR, { headless: HEADLESS, executablePath: process.env.ELASTOS_BROWSER_EXECUTABLE || undefined, @@ -3671,15 +4109,22 @@ async function main() { return cleanupResult; } try { + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:virtual-authenticator"); virtualAuthenticator = await setupVirtualAuthenticator(context, page); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:navigate"); await page.goto(HOME_URL, { waitUntil: "domcontentloaded" }); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:sign-in"); created = await ensureSignedWithVirtualPasskey(page); homeToken = created.homeToken; + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:passkey-read"); passkey = await currentPasskey(page, homeToken); assert(passkey?.proof_binding_id, "signed virtual passkey was not visible through the passkey list", passkey); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:sign-out"); await signOut(page, homeToken); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:sign-in-again"); homeToken = await signBackIn(page); + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:credential-store"); const afterSignIn = await currentPasskey(page, homeToken); assert( afterSignIn?.proof_binding_id === passkey.proof_binding_id, @@ -3690,6 +4135,7 @@ async function main() { ? await persistVirtualAuthenticatorCredentials(virtualAuthenticator) : { skipped: true, reason: "created credential will be cleaned up" }; + if (CHECK_BROWSER_CONTROLLED_JOURNEY) markStage("home:public-copy"); const homePublicCopy = await checkHomePublicCopy(page); const system = CHECK_SYSTEM ? await launchSystem(page, homeToken) : null; const shellSwitch = CHECK_SHELL_SWITCH @@ -3748,16 +4194,16 @@ async function main() { const cleanup = await cleanupCreatedPasskey(); if (cleanup && cleanup.ok === false) { console.error("virtual test passkey cleanup failed after smoke error"); - console.error(JSON.stringify(cleanup, null, 2)); + console.error(JSON.stringify(redactSensitive(cleanup), null, 2)); } } catch (cleanupError) { console.error("virtual test passkey cleanup threw after smoke error"); - console.error(cleanupError.message || cleanupError); + console.error(redactSensitiveString(cleanupError.message || cleanupError)); } console.error("FAIL home-passkey-virtual-auth-smoke"); - console.error(error.message || error); + console.error(redactSensitiveString(error.message || error)); if (error.stack) { - console.error(error.stack); + console.error(redactSensitiveString(error.stack)); } if (error.details) { console.error(JSON.stringify(redactSensitive(error.details), null, 2)); @@ -3765,7 +4211,7 @@ async function main() { const state = page ? await homeState(page).catch(() => null) : null; if (state) { state.stage = smokeStage; - console.error(JSON.stringify(state, null, 2)); + console.error(JSON.stringify(redactSensitive(state), null, 2)); } } process.exitCode = 1; diff --git a/scripts/lib/browser-journey-fixture.mjs b/scripts/lib/browser-journey-fixture.mjs new file mode 100644 index 000000000..8b5690cb7 --- /dev/null +++ b/scripts/lib/browser-journey-fixture.mjs @@ -0,0 +1,144 @@ +#!/usr/bin/env node +// Parent-owned fixture: node scripts/lib/browser-journey-fixture.mjs [--port 61511] +// Authorize only localhost: in the test Runtime's allowed_private_targets. +import http from "node:http"; +import { pathToFileURL } from "node:url"; + +const MAX_RUNS = 16; +const MAX_EVENTS = 128; +const TTL_MS = 10 * 60_000; +const RUN_ID = /^[a-zA-Z0-9_-]{8,64}$/; + +function fixturePage(page, run) { + return ` +Browser journey ${page} + +
+

Browser journey ${page}

+ + + Open navigation page +
+

Scroll down to move this page.

+

End of controlled scroll content

+
+ +`; +} + +export function createBrowserJourneyFixture() { + const runs = new Map(); + return http.createServer({ requestTimeout: 5_000, headersTimeout: 5_000 }, async (req, res) => { + const json = (status, body) => { + res.writeHead(status, { "content-type": "application/json", "cache-control": "no-store" }); + res.end(JSON.stringify(body)); + }; + try { + const url = new URL(req.url, "http://localhost"); + if (req.method === "GET" && url.pathname === "/health") { + json(200, { schema: "elastos.browser.journey-fixture/v1", ok: true }); + return; + } + const run = url.searchParams.get("run") || ""; + if (!RUN_ID.test(run)) { json(400, { error: "invalid run id" }); return; } + for (const [id, record] of runs) { + if (Date.now() - record.created_at >= TTL_MS) runs.delete(id); + } + if (req.method === "GET" && ["/main", "/nav"].includes(url.pathname)) { + if (!runs.has(run)) { + if (runs.size >= MAX_RUNS) { json(429, { error: "fixture run capacity" }); return; } + runs.set(run, { created_at: Date.now(), events: [] }); + } + res.writeHead(200, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" }); + res.end(fixturePage(url.pathname.slice(1), run)); + return; + } + const record = runs.get(run); + if (!record) { json(404, { error: "unknown run" }); return; } + if (req.method === "GET" && url.pathname === "/receipt") { + json(200, { schema: "elastos.browser.journey-receipt/v1", run, events: record.events }); + return; + } + if (req.method !== "POST" || url.pathname !== "/events") { + json(404, { error: "unknown route" }); return; + } + const chunks = []; + let bytes = 0; + for await (const chunk of req) { + bytes += chunk.length; + if (bytes > 4096) { json(413, { error: "event too large" }); return; } + chunks.push(chunk); + } + const event = JSON.parse(Buffer.concat(chunks).toString("utf8")); + const rect = event?.input_rect; + if (!event || !["load", "input", "scroll"].includes(event.type) || + !["main", "nav"].includes(event.page) || typeof event.value !== "string" || + event.value.length > 96 || !rect || + ![event.scroll_x, event.scroll_y, rect.x, rect.y, rect.width, rect.height] + .every(value => Number.isFinite(value) && Math.abs(value) <= 100_000)) { + json(400, { error: "invalid event" }); return; + } + if (record.events.length >= MAX_EVENTS) { json(429, { error: "fixture event capacity" }); return; } + record.events.push({ sequence: record.events.length + 1, received_at: Date.now(), + type: event.type, page: event.page, value: event.value, + scroll_x: event.scroll_x, scroll_y: event.scroll_y, + input_rect: { x: rect.x, y: rect.y, width: rect.width, height: rect.height } }); + json(200, { ok: true }); + } catch { + if (!res.headersSent) json(400, { error: "invalid fixture request" }); + } + }); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const args = process.argv.slice(2); + const port = args.length === 0 ? 61511 : args.length === 2 && args[0] === "--port" ? Number(args[1]) : NaN; + if (!Number.isInteger(port) || port < 1024 || port > 65535) { + throw new Error("Usage: node scripts/lib/browser-journey-fixture.mjs [--port 61511]"); + } + const server = createBrowserJourneyFixture(); + server.listen(port, "127.0.0.1", () => console.log(JSON.stringify({ + schema: "elastos.browser.journey-fixture/v1", origin: `http://localhost:${port}`, + retention: { max_runs: MAX_RUNS, max_events_per_run: MAX_EVENTS, ttl_ms: TTL_MS }, + }))); + for (const signal of ["SIGINT", "SIGTERM"]) process.once(signal, () => { + server.close(); + server.closeAllConnections(); + }); +} diff --git a/scripts/lib/browser-journey-harness.test.mjs b/scripts/lib/browser-journey-harness.test.mjs new file mode 100644 index 000000000..8204b561c --- /dev/null +++ b/scripts/lib/browser-journey-harness.test.mjs @@ -0,0 +1,447 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import test from "node:test"; +import vm from "node:vm"; +import { createBrowserJourneyFixture } from "./browser-journey-fixture.mjs"; + +const source = readFileSync(new URL("../home-passkey-virtual-auth-smoke.mjs", import.meta.url), "utf8"); +function harnessFunction(name, globals = {}) { + const start = source.search(new RegExp(`(?:async )?function ${name}\\(`)); + assert.ok(start >= 0, name); + const next = source.slice(start + 1).search(/\n(?:async )?function /); + const declaration = source.slice(start, start + 1 + next); + return vm.runInNewContext(`(${declaration})`, { URL, Date, performance, ...globals }); +} + +test("controlled fixture isolates runs, records bounded events and rejects malformed requests", async () => { + const server = createBrowserJourneyFixture(); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + const base = `http://127.0.0.1:${server.address().port}`; + const run = "test-run-123"; + const event = { type: "input", page: "nav", value: "Browser-é", scroll_x: 0, scroll_y: 640, + input_rect: { x: 32, y: 150, width: 480, height: 60 } }; + const post = body => fetch(`${base}/events?run=${run}`, { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body), + }); + try { + assert.equal((await fetch(`${base}/health`).then(r => r.json())).ok, true); + assert.equal((await fetch(`${base}/receipt?run=${run}`)).status, 404); + assert.equal((await fetch(`${base}/main?run= + + @@ -62,6 +64,10 @@ Media + @@ -73,6 +79,7 @@

Discover

+ diff --git a/capsules/marketplace/browser/marketplace.js b/capsules/marketplace/browser/marketplace.js index bf7a4c2e1..ffbedc8fc 100644 --- a/capsules/marketplace/browser/marketplace.js +++ b/capsules/marketplace/browser/marketplace.js @@ -51,6 +51,11 @@ { id: "shells", label: "Home views", icon: "system" }, ]; + const models = window.ElastosModelManagement.create({ + root: document.querySelector("[data-model-management]"), capsule: "marketplace", token: homeToken, + buttonClass: "store-pill", + }); + const icons = { package: '', play: '', @@ -64,6 +69,7 @@ }; const CAPSULE_ICON_ROUTE = /^\/apps\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_./-]+\.png$/; + document.querySelector("[data-model-icon]").innerHTML = icons.package; boot(); @@ -451,6 +457,7 @@ } function normalizeDestination(id) { + if (id === "models") return "models"; if (id === "media") return "media"; if (id === "installed") return "installed"; if (categories.some((category) => category.id === id)) return id; @@ -458,6 +465,7 @@ } function destinationTitle(id) { + if (id === "models") return "Models"; if (id === "media") return "Media"; if (id === "installed") return "Installed"; const category = categories.find((entry) => entry.id === id); @@ -467,6 +475,7 @@ function selectDestination(id, options = {}) { state.destination = normalizeDestination(id); + models.setVisible(state.destination === "models"); document.querySelectorAll("[data-destination]").forEach((node) => { const selected = node.dataset.destination === state.destination; node.classList.toggle("selected", selected); @@ -499,6 +508,12 @@ } function renderSurfaceState() { + if (state.destination === "models") { + els.loadingState.classList.add("hidden"); + els.loadError.classList.add("hidden"); + els.storeSections.classList.add("hidden"); + return true; + } const surface = surfaceState(); els.loadingState.classList.toggle("hidden", !surface.loading); if (surface.loading) { diff --git a/capsules/marketplace/browser/model-management.css b/capsules/marketplace/browser/model-management.css new file mode 100644 index 000000000..62a16ef7e --- /dev/null +++ b/capsules/marketplace/browser/model-management.css @@ -0,0 +1,9 @@ +.model-management[hidden] { display: none; } +.model-management { min-width: 0; color: var(--ink); } +.model-row { padding: 16px 0; border-top: 1px solid var(--line); min-width: 0; } +.model-row h2 { font-size: 18px; margin: 0 0 12px; } +.model-row p { overflow-wrap: anywhere; line-height: 1.5; margin: 8px 0; } +.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 16px; } +.model-keep { display: inline-flex; align-items: center; gap: 8px; } +.model-keep input { accent-color: var(--accent); } +.model-row progress { width: 100%; max-width: 440px; accent-color: var(--accent); } diff --git a/capsules/marketplace/browser/model-management.js b/capsules/marketplace/browser/model-management.js new file mode 100644 index 000000000..808355b29 --- /dev/null +++ b/capsules/marketplace/browser/model-management.js @@ -0,0 +1,235 @@ +/* Shared model presentation. Runtime owns admission, execution and retention. */ +(() => { + const CID = /^bafybei[a-z2-7]{52}$/; + const STATES = new Set(["reserved", "preparing", "verifying", "admission_pending", "admitted", "failed", "uncertain", "cancelled", "expired"]); + const active = p => p && ["reserved", "preparing", "verifying", "admission_pending", "uncertain"].includes(p.state); + const text = (v, max = 256) => typeof v === "string" && v.length > 0 && v.length <= max && !/[\u0000-\u001f]/.test(v); + function check(ok) { if (!ok) throw new Error("Invalid model response"); } + function parseRuntime(r, cid) { + check(r && [r.admitted, r.kept, r.dispatch_ready].every(v => typeof v === "boolean")); + check(r.dispatch_ready ? r.admitted && /^model:[0-9a-f]{64}$/.test(r.offer_id) : r.offer_id === null); + const p = r.preparation; + if (p !== null) { + check(p && p.cid === cid && text(p.operation_id, 160) && STATES.has(p.state)); + check([p.total_bytes, p.completed_bytes].every(n => Number.isSafeInteger(n) && n >= 0) && p.completed_bytes <= p.total_bytes); + check(typeof p.cancel_requested === "boolean" && typeof p.activation_pending === "boolean" && typeof p.admitted === "boolean"); + check(p.admitted === (p.state === "admitted") && r.admitted === p.admitted); + } else check(!r.admitted && !r.kept && !r.dispatch_ready); + return r; + } + function element(tag, value, className) { + const node = document.createElement(tag); + if (value) node.textContent = value; + if (className) node.className = className; + return node; + } + // Typed operation outputs are flat. Only catalog rows nest model_runtime. + function operationRuntime(output, cid, retainedPreparation) { + const preparation = retainedPreparation || Object.fromEntries([ + "operation_id", "cid", "state", "total_bytes", "completed_bytes", "cancel_requested", "admitted", "activation_pending", + ].map(key => [key, output[key]])); + return parseRuntime({ admitted: output.admitted, kept: output.kept, + dispatch_ready: output.dispatch_ready, offer_id: output.offer_id, preparation }, cid); + } + window.ElastosModelManagement = { + create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary" }) { + let visible = false, closed = false, generation = 0, busy = false, timer, reader; + let model = null, methods = new Map(), message = "", loading = false, polls = 0; + let unresolvedUse = null, reconcileRequired = false; + let pendingFocus = null; + root.classList.add("model-management"); + root.tabIndex = -1; + const show = () => visible && !closed && !document.hidden; + function stopRead() { clearTimeout(timer); reader?.abort(); reader = null; } + async function request(path, body, signal) { + const response = await fetch(path, { method: body ? "POST" : "GET", headers: { + "x-elastos-home-token": token, ...(body ? { "content-type": "application/json" } : {}), + }, ...(body ? { body: JSON.stringify(body) } : {}), signal }); + if (!response.ok) throw new Error("Model request failed"); + const raw = await response.text(); + check(raw.length <= 2 * 1024 * 1024); + return JSON.parse(raw); + } + async function invoke(operation, input, signal, requestId) { + const method = methods.get(operation); + check(method); + const body = { capsule, interface: method.interface, method: method.id, + request_id: requestId || `model-${crypto.randomUUID()}`, input }; + const result = await request("/api/capsules/interfaces/invoke", body, signal); + check(result.schema === "elastos.capsules.invoke-result/v1" && result.status === "ok" + && ["capsule", "interface", "method", "request_id"].every(key => result[key] === body[key])); + return result.output; + } + function controller() { + const value = new AbortController(); + const timeout = setTimeout(() => value.abort(), 35000); + return { signal: value.signal, abort: () => value.abort(), done: () => clearTimeout(timeout) }; + } + function parseCatalog(catalog) { + check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); + check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); + if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); + const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); + check(entries.length <= 1 && (!entries.length || catalog.model_catalog_state === "verified")); + if (!entries.length) return null; + const entry = entries[0]; + check(entry.role === "content" && entry.installed === false && entry.launchable === false + && CID.test(entry.cid) && text(entry.title) && text(entry.publisher_did, 256) + && entry.publisher_did.startsWith("did:") && entry.signature_state === "catalog-signature-verified" + && Number.isSafeInteger(entry.content_size_bytes) && entry.content_size_bytes > 0); + parseRuntime(entry.model_runtime, entry.cid); + return entry; + } + function bindMethods(registry) { + check(Array.isArray(registry.interfaces) && registry.interfaces.length <= 1024); + const found = new Map(); + for (const entry of registry.interfaces.filter(e => e.capsule === capsule)) { + check(text(entry.interface?.id, 160) && /^[A-Za-z0-9_.:-]+$/.test(entry.interface.id)); + for (const method of entry.interface?.methods || []) { + if (method.resource !== "elastos://capsules/*" || !["use", "status", "cancel", "retention"].includes(method.operation)) continue; + check(!found.has(method.operation)); + const bindings = entry.bindings?.filter(b => b.method === method.id); + check(method.id === `content.${method.operation}` && bindings?.length === 1 + && bindings[0].executable === true && method.approval === "runtime_policy" + && method.risk === (method.operation === "status" ? "read" : "write")); + found.set(method.operation, { id: method.id, interface: entry.interface.id }); + } + } + check(found.size === 4); + methods = found; + } + async function load() { + if (!show() || busy) return; + stopRead(); const epoch = ++generation; const io = controller(); reader = io; + loading = true; message = ""; render(); + try { + const [catalog, interfaces] = await Promise.all([ + request("/api/capsules/catalog", null, io.signal), request("/api/capsules/interfaces", null, io.signal), + ]); + if (epoch !== generation || !show()) return; + bindMethods(interfaces); model = parseCatalog(catalog); polls = 0; + reconcileRequired = false; + if (model?.cid !== unresolvedUse?.cid || model?.model_runtime.preparation) unresolvedUse = null; + } catch { + if (epoch !== generation || !show()) return; + model = null; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; + } finally { + io.done(); + if (epoch === generation) { loading = false; render(); schedule(); } + } + } + function schedule() { + clearTimeout(timer); + if (!show() || busy || message || !active(model?.model_runtime.preparation)) return; + if (polls >= 120) { message = "Preparation is still pending. Refresh to check its status."; render(); return; } + timer = setTimeout(poll, 1500); + } + async function poll() { + if (!show() || busy || !model) return; + const epoch = generation, cid = model.cid, id = model.model_runtime.preparation.operation_id; + const io = controller(); reader = io; polls++; + try { + const result = await invoke("status", { operation_id: id }, io.signal); + if (epoch !== generation || !show() || model?.cid !== cid) return; + check(result.cid === cid && result.operation_id === id); + const r = operationRuntime(result, cid); + check(r.preparation?.operation_id === id); + model.model_runtime = r; + } catch { + if (epoch !== generation || !show()) return; + message = "Status is unavailable. Refresh to check before trying again."; + } finally { io.done(); if (epoch === generation) { render(); schedule(); } } + } + async function act(operation, keep) { + if (busy || loading || !show() || !model || reconcileRequired) return; + stopRead(); const epoch = ++generation, cid = model.cid; + const id = model.model_runtime.preparation?.operation_id; + busy = true; message = ""; render(); + const io = controller(); + try { + const input = operation === "retention" ? { cid, keep } : operation === "cancel" ? { operation_id: id } : { cid }; + if (operation === "use" && !unresolvedUse) unresolvedUse = { cid, id: `model-${crypto.randomUUID()}` }; + const result = await invoke(operation, input, io.signal, operation === "use" ? unresolvedUse.id : undefined); + if (epoch !== generation || !show() || model?.cid !== cid) return; + check(result.cid === cid); + if (operation === "retention") { + check(result.kept === keep && result.admitted === true); + // Keep is confirmed only by the current caller's Runtime response. + const candidate = operationRuntime(result, cid, model.model_runtime.preparation); + check(candidate.kept === keep); + model.model_runtime = candidate; + } else { + const r = operationRuntime(result, cid); + check(r.preparation?.operation_id === result.operation_id && (operation !== "cancel" || result.operation_id === id)); + model.model_runtime = r; polls = 0; + if (operation === "use") unresolvedUse = null; + } + } catch { + if (epoch === generation && show()) { + reconcileRequired = true; + message = "Model action could not be confirmed. Refresh to check its status."; + } + } finally { + io.done(); busy = false; + if (epoch === generation) { render(); schedule(); } + else if (show()) void load(); + } + } + function button(label, run, disabled = false) { + const node = element("button", label, buttonClass); + node.type = "button"; node.disabled = disabled || busy || loading; node.addEventListener("click", run); return node; + } + function render() { + const focused = root.contains(document.activeElement) ? document.activeElement?.dataset.modelControl || pendingFocus : null; + pendingFocus = focused; + const restoreFocus = () => { + if (!focused) return; + const target = root.querySelector(`[data-model-control="${focused}"]`); + if (target && !target.disabled) { target.focus(); pendingFocus = null; } + else root.focus(); + }; + root.replaceChildren(); + const refresh = button("Refresh models", () => void load(), loading); + refresh.dataset.modelControl = "refresh"; root.append(refresh); + const status = element("p", loading ? "Loading models…" : message); + status.setAttribute("role", "status"); root.append(status); + if (!model) { + if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); + restoreFocus(); + return; + } + const r = model.model_runtime, p = r.preparation; + const row = element("article", "", "model-row"); + row.append(element("h2", model.title), element("p", `Verified publisher: ${model.publisher_did}`), + element("p", `Content ID: ${model.cid}`, "model-identity"), element("p", `${model.content_size_bytes.toLocaleString()} bytes`)); + const label = message ? "Current status unavailable" : r.dispatch_ready ? "Ready to use" : r.admitted ? "Prepared. The model offer is unavailable." + : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : "Preparing…") + : p ? ({ cancelled: "Preparation cancelled.", failed: "Preparation failed.", expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Not prepared"; + row.append(element("p", label)); + if (active(p)) { + const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; + progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${p.completed_bytes.toLocaleString()} of ${p.total_bytes.toLocaleString()} bytes`)); + } + const controls = element("div", "", "model-controls"); + const use = button(p || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); + use.dataset.modelControl = "use"; + if (!active(p) && !r.dispatch_ready) controls.append(use); + if (active(p)) { const cancel = button("Cancel preparation", () => void act("cancel"), p.cancel_requested || reconcileRequired); cancel.dataset.modelControl = "cancel"; controls.append(cancel); } + const labelNode = element("label", "", "model-keep"), toggle = element("input"); + toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !r.admitted || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; + toggle.addEventListener("change", () => void act("retention", toggle.checked)); + labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); + row.append(controls); root.append(row); + restoreFocus(); + } + function setVisible(value) { + if (visible === value) return; + visible = value; root.hidden = !value; ++generation; stopRead(); + if (show()) void load(); + } + document.addEventListener("visibilitychange", () => { ++generation; stopRead(); if (show()) void load(); }); + window.addEventListener("pagehide", () => { closed = true; ++generation; stopRead(); }); + return { setVisible, refresh: load }; + }, + }; +})(); diff --git a/capsules/marketplace/capsule.json b/capsules/marketplace/capsule.json index d2cbc74fd..4de71d00e 100644 --- a/capsules/marketplace/capsule.json +++ b/capsules/marketplace/capsule.json @@ -46,6 +46,33 @@ "resource": "elastos://marketplace/install", "operation": "install" }, + { + "id": "content.use", + "description": "Prepare a verified model", + "risk": "write", + "approval": "runtime_policy", + "audit": "summary", + "resource": "elastos://capsules/*", + "operation": "use" + }, + { + "id": "content.status", + "description": "Check model preparation", + "risk": "read", + "approval": "runtime_policy", + "audit": "summary", + "resource": "elastos://capsules/*", + "operation": "status" + }, + { + "id": "content.cancel", + "description": "Cancel model preparation", + "risk": "write", + "approval": "runtime_policy", + "audit": "summary", + "resource": "elastos://capsules/*", + "operation": "cancel" + }, { "id": "content.retention", "description": "Keep or release a model on this device", diff --git a/capsules/system/browser/index.html b/capsules/system/browser/index.html index f8231bd1b..c30448626 100644 --- a/capsules/system/browser/index.html +++ b/capsules/system/browser/index.html @@ -5,6 +5,8 @@ System · ElastOS + + @@ -56,6 +58,10 @@

Open from Home

Security + +
+`; + +test("real Home menu focuses captured Browser through overlap; launcher binds foreground rather than DOM-last", async () => { + const server = createServer((req, res) => { + res.setHeader("content-type", "text/html"); + if (req.url.startsWith("/apps/home-gui/")) res.end(shell); + else if (req.url.startsWith("/apps/browser/")) res.end("Fixture Browser"); + else res.end(''); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + const origin = `http://127.0.0.1:${server.address().port}`; + let browser; + try { + browser = await chromium.launch({ headless: true, + ...(process.env.ELASTOS_BROWSER_EXECUTABLE ? { executablePath: process.env.ELASTOS_BROWSER_EXECUTABLE } : {}), + args: ["--disable-background-networking", "--disable-component-update", "--no-first-run"] }); + const page = await browser.newPage({ viewport: { width: 900, height: 700 } }); + const errors = []; page.on("pageerror", error => errors.push(error.message)); + const globals = { URL, URLSearchParams, Date, HOME_URL: `${origin}/apps/home/`, + delay: ms => new Promise(resolve => setTimeout(resolve, ms)), assert: (value, message) => assert.ok(value, message) }; + for (const name of ["launchTokenFromRoute", "assertIsolatedLaunchRoute", "assertBrowserWindowIdentity", "captureBrowserWindowIdentity", + "focusCapturedBrowserWindow", "clickBrowserWindowClose", "waitForBrowserWindowDetached"]) { + globals[name] = vm.runInNewContext(`(${declaration(main, name)})`, globals); + } + for (const empty of [false, true]) { + await page.goto(`${origin}/apps/home/${empty ? "?empty=1" : ""}`); + const gui = page.frames().find(frame => frame.url().includes("/apps/home-gui/")); + await gui.locator("#shelf").waitFor(); + const open = vm.runInNewContext(`(${declaration(main, "openDesktopAppWindow")})`, { + ...globals, waitForSignedHome: async () => {}, waitForCapsuleFrame: async () => gui, + }); + let picked; + const frame = await open(page, "browser", async selected => { + picked = selected; + // Restore takes foreground after capture, before later readiness waits. + await gui.evaluate(() => window.restoreC()); + }); + assert.equal(frame, picked); + assert.equal(new URL(frame.url()).searchParams.get("browser_instance"), "browser--2"); + const identity = await globals.captureBrowserWindowIdentity(frame); + assert.equal(identity.windowId, "browser--2"); + assert.equal(identity.token, "token-browser--2"); + await gui.evaluate(() => window.restoreC()); + const button = await identity.section.$('[data-action="close"]'); + await assert.rejects(button.click({ trial: true, timeout: 250 }), /intercepts pointer events/); + await assert.rejects(globals.clickBrowserWindowClose(identity, frame, "wrong-token"), /authority identity/); + assert.deepEqual(await gui.evaluate(() => evidence.closes), []); + await globals.clickBrowserWindowClose(identity, frame, identity.token); + await globals.waitForBrowserWindowDetached(identity); + assert.equal(await gui.locator('[data-window-id="browser--3"]').count(), 1); + assert.equal(await gui.locator('[data-window-id="browser--1"]').count(), empty ? 0 : 1); + const evidence = await gui.evaluate(() => window.evidence); + assert.deepEqual(evidence.closes, [{ id: "browser--2", trusted: true }]); + assert.equal(evidence.actions.length, 2); + assert.ok(evidence.actions.every(row => row.action === "focus-window:browser--2" && row.trusted)); + assert.deepEqual(evidence.contextClicks, [true, true]); + await button.dispose(); + } + assert.deepEqual(errors, []); + } finally { + await browser?.close(); server.closeAllConnections(); await new Promise(resolve => server.close(resolve)); + } +}); diff --git a/scripts/home-passkey-virtual-auth-smoke.mjs b/scripts/home-passkey-virtual-auth-smoke.mjs index ee58a8cfc..4b8237616 100644 --- a/scripts/home-passkey-virtual-auth-smoke.mjs +++ b/scripts/home-passkey-virtual-auth-smoke.mjs @@ -2095,6 +2095,7 @@ async function assertBrowserWindowIdentity(identity, appFrame, token) { } async function clickBrowserWindowClose(identity, appFrame, token) { + await focusCapturedBrowserWindow(identity, appFrame, token); await assertBrowserWindowIdentity(identity, appFrame, token); // ElementHandle.click cannot retarget another window during auto-wait. const button = await identity.section.$('[data-action="close"]'); @@ -2103,6 +2104,24 @@ async function clickBrowserWindowClose(identity, appFrame, token) { try { await button.click(); } finally { await button.dispose(); } } +async function focusCapturedBrowserWindow(identity, appFrame, token) { + await assertBrowserWindowIdentity(identity, appFrame, token); + const actionable = await identity.section.evaluate(node => { + const button = node.querySelector('[data-action="close"]'); + const rect = button?.getBoundingClientRect(); + return node.classList.contains("window-active") && rect?.width > 0 && rect?.height > 0 && + button.contains(node.ownerDocument.elementFromPoint(rect.x + rect.width / 2, rect.y + rect.height / 2)); + }); + if (actionable) return; + // Home's per-window Shelf action raises this window without toggling another Browser. + await identity.parent.locator('#taskbar-targets [data-target="browser"]').first().click({ button: "right", timeout: 5_000 }); + await identity.parent.locator(`#desktop-context-menu [data-context-action="focus-window:${identity.windowId}"]`) + .click({ timeout: 5_000 }); + await identity.parent.waitForFunction(node => node.isConnected && node.classList.contains("window-active"), + identity.section, { timeout: 5_000 }); + await assertBrowserWindowIdentity(identity, appFrame, token); +} + async function waitForBrowserWindowDetached(identity) { await identity.parent.waitForFunction(node => !node.isConnected, identity.section, { timeout: 15_000 }); @@ -3087,6 +3106,7 @@ async function checkBrowserEmbeddedUiInput(page, baselineToken) { windowIdentity = await captureBrowserWindowIdentity(appFrame); windowLocator = windowIdentity.locator; browserToken = windowIdentity.token; + await focusCapturedBrowserWindow(windowIdentity, appFrame, browserToken); await windowLocator.waitFor({ state: "visible", timeout: 30_000 }); await windowIdentity.parent.waitForFunction(node => { return node.isConnected && node.classList.contains("window-active") && @@ -3749,6 +3769,44 @@ async function openDesktopAppWindow(page, target, onFrame = null) { if (setupVisible) { await setupLater.click(); } + if (target === "browser") { + // Match Home's foreground choice once. DOM order can differ from z-order, + // and a later restored window must not replace the chosen frame. + const selected = (await homeGuiFrame.evaluateHandle(() => { + const windows = [...document.querySelectorAll('section.window[data-target="browser"]')]; + const active = windows.filter(node => node.classList.contains("window-active")); + if (active.length > 1) throw new Error("Home has ambiguous active Browser windows"); + const visible = windows.filter(node => !node.classList.contains("hidden")); + return active[0] || (visible.length ? visible : windows) + .sort((a, b) => Number(b.style.zIndex || 0) - Number(a.style.zIndex || 0))[0] || null; + })).asElement(); + let handle; + if (selected) { + handle = await selected.$("iframe.window-frame"); + } else { + const shelf = homeGuiFrame.locator('#taskbar-targets [data-target="browser"]').first(); + if (await shelf.isVisible()) await shelf.click(); + else { + await homeGuiFrame.locator("#launcher-toggle").click(); + await homeGuiFrame.locator('#launcher-grid [data-target="browser"]').first().click(); + } + // There was no Browser at selection. Ambiguous concurrent creations fail + // the strict locator instead of assigning another window's authority. + const created = homeGuiFrame.locator('section.window[data-target="browser"] iframe.window-frame'); + await created.waitFor({ state: "attached", timeout: 20_000 }); + handle = await created.elementHandle(); + } + const appFrame = handle ? await handle.contentFrame() : null; + assert(appFrame, "Selected Home Browser window has no content frame"); + if (onFrame) await onFrame(appFrame); + const deadline = Date.now() + 20_000; + while (Date.now() < deadline && !appFrame.url().includes("/apps/browser/")) await delay(100); + assert(appFrame.url().includes("/apps/browser/"), "Selected Home Browser window did not load its capsule document"); + const identity = await captureBrowserWindowIdentity(appFrame); + await focusCapturedBrowserWindow(identity, appFrame, identity.token); + await handle.waitForElementState("visible", { timeout: 20_000 }); + return appFrame; + } const shelfItem = homeGuiFrame.locator(`#taskbar-targets [data-target="${target}"]`).first(); const existingWindow = homeGuiFrame.locator(`section.window[data-target="${target}"]`).last(); const alreadyActive = await existingWindow.isVisible() && @@ -3765,12 +3823,7 @@ async function openDesktopAppWindow(page, target, onFrame = null) { await card.waitFor({ state: "visible", timeout: 10_000 }); await card.click(); } - // The desktop restores persisted windows at boot and restore can steal - // focus from the window the launcher just opened, so bind to the newest - // window for the target rather than whichever one holds the active class. - const windowFrameEl = homeGuiFrame - .locator(`section.window[data-target="${target}"] iframe.window-frame`) - .last(); + const windowFrameEl = homeGuiFrame.locator(`section.window[data-target="${target}"] iframe.window-frame`).last(); await windowFrameEl.waitFor({ state: onFrame ? "attached" : "visible", timeout: 20_000 }); const handle = await windowFrameEl.elementHandle(); const appFrame = handle ? await handle.contentFrame() : null; diff --git a/scripts/lib/browser-journey-harness.test.mjs b/scripts/lib/browser-journey-harness.test.mjs index 79386d9f2..a0dd55cc5 100644 --- a/scripts/lib/browser-journey-harness.test.mjs +++ b/scripts/lib/browser-journey-harness.test.mjs @@ -144,34 +144,28 @@ test("error-state redaction covers frame fragments and direct token fields", () assert.match(source, /JSON\.stringify\(redactSensitive\(state\), null, 2\)/); }); -test("launcher reuses a visible restored window without toggling its Shelf item", async () => { +test("launcher reuses the selected Browser through exact focus without toggling its Shelf item", async () => { const actions = []; const frame = { url: () => "http://localhost/apps/browser/" }; - const frameElement = { waitFor: async () => {}, elementHandle: async () => ({ contentFrame: async () => frame }) }; - let active = true; - const existing = { isVisible: async () => true, - evaluate: async callback => callback({ classList: { contains: name => name === "window-active" && active } }), - }; + const handle = { contentFrame: async () => frame, waitForElementState: async () => {} }; + const selected = { $: async () => handle }; const gui = { getByRole: () => ({ waitFor: async () => { const error = new Error(); error.name = "TimeoutError"; throw error; } }), - locator: selector => { - if (selector.includes("#taskbar-targets")) return { first: () => ({ - isVisible: async () => true, click: async () => actions.push("toggle-shelf"), - }) }; - if (selector.includes("iframe.window-frame")) return { last: () => frameElement }; - return { last: () => existing }; - }, + evaluateHandle: async () => ({ asElement: () => selected }), + locator: () => { throw new Error("Existing Browser must not toggle generic Shelf"); }, }; + const identity = { token: "selected-token" }; const open = harnessFunction("openDesktopAppWindow", { HOME_URL: "http://localhost/apps/home/", waitForSignedHome: async () => {}, waitForCapsuleFrame: async () => gui, assert: (condition, message) => assert.ok(condition, message), + captureBrowserWindowIdentity: async actual => { assert.equal(actual, frame); return identity; }, + focusCapturedBrowserWindow: async (actual, selectedFrame, token) => { + assert.equal(actual, identity); assert.equal(selectedFrame, frame); assert.equal(token, identity.token); actions.push("focus-exact"); + }, }); const page = { url: () => "http://localhost/apps/home/", goto: async () => actions.push("reload-home") }; - assert.equal(await open(page, "browser"), frame); - assert.deepEqual(actions, []); - active = false; - assert.equal(await open(page, "browser"), frame); - assert.deepEqual(actions, ["toggle-shelf"]); + assert.equal(await open(page, "browser", actual => { assert.equal(actual, frame); actions.push("captured"); }), frame); + assert.deepEqual(actions, ["captured", "focus-exact"]); }); function closeHarnessSurface() { @@ -193,7 +187,10 @@ function closeHarnessSurface() { }; const frame = { url: () => "http://localhost/apps/browser/?browser_instance=instance-one#home_token=browser-token", parentFrame: () => gui, frameElement: async () => iframe, waitForFunction: async () => {} }; - const node = { isConnected: true, dataset: { windowId: "browser--2" }, querySelector: () => iframe }; + const closeNode = { getBoundingClientRect: () => ({ x: 0, y: 0, width: 20, height: 20 }), contains: node => node === closeNode }; + const node = { isConnected: true, dataset: { windowId: "browser--2" }, + classList: { contains: name => name === "window-active" }, ownerDocument: { elementFromPoint: () => closeNode }, + querySelector: selector => selector.includes("close") ? closeNode : iframe }; const iframe = { contentWindow: source, isConnected: true, closest: () => node, getAttribute: async () => frame.url(), evaluateHandle: async callback => ({ asElement: () => { assert.equal(callback(iframe), node); return section; @@ -201,6 +198,7 @@ function closeHarnessSurface() { const section = { node, getAttribute: async () => node.dataset.windowId, evaluate: async (callback, args) => callback(node, args), $: async selector => { + if (selector === "iframe.window-frame") return iframe; assert.equal(selector, '[data-action="close"]'); return { getAttribute: async () => "Close", dispose: async () => {}, click: async () => { actions.push("ui-close"); await surface.click(); @@ -266,6 +264,7 @@ function browserIdentityGlobals() { globals.launchTokenFromRoute = harnessFunction("launchTokenFromRoute", globals); globals.assertIsolatedLaunchRoute = harnessFunction("assertIsolatedLaunchRoute", globals); globals.assertBrowserWindowIdentity = harnessFunction("assertBrowserWindowIdentity", globals); + globals.focusCapturedBrowserWindow = harnessFunction("focusCapturedBrowserWindow", globals); return globals; } @@ -353,16 +352,20 @@ test("launcher visibility failure retains exact Frame B for caller cleanup while surface.gui.locator = selector => { if (selector.includes("data-window-id=")) return pinnedLocator(selector); if (selector.includes("#taskbar-targets")) return { first: () => ({ isVisible: async () => true }) }; - if (selector.includes("iframe.window-frame")) return { last: () => ({ + if (selector.includes("iframe.window-frame")) return { waitFor: async options => { assert.equal(options.state, "attached"); }, elementHandle: async () => surface.iframe, - }) }; - return { last: () => ({ isVisible: async () => true, + }; + return { isVisible: async () => true, evaluate: async callback => callback({ classList: { contains: () => true } }), - }) }; + }; }; + const evaluateMessages = surface.gui.evaluateHandle; + surface.gui.evaluateHandle = async (callback, args) => callback.toString().includes("document.querySelectorAll") + ? { asElement: () => surface.section } : evaluateMessages(callback, args); const globals = browserIdentityGlobals(); const open = harnessFunction("openDesktopAppWindow", { ...globals, + captureBrowserWindowIdentity: harnessFunction("captureBrowserWindowIdentity", globals), waitForSignedHome: async () => {}, waitForCapsuleFrame: async () => surface.gui, }); const empty = { schema: "elastos.browser.session-capacity/v1", status: "configured", recoverable_page: null, From 74d5b2724154dce892ad93bf754d72b20b1c4c37 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 8 Sep 2026 17:52:58 -0500 Subject: [PATCH 210/535] feat(browser): adapt scoped reference fill for Playwright clients --- .../lib/browser-operator-client-playwright.md | 136 +++++++++ .../browser-operator-client-playwright.mjs | 192 ++++++++++++ ...rowser-operator-client-playwright.test.mjs | 287 ++++++++++++++++++ 3 files changed, 615 insertions(+) create mode 100644 scripts/lib/browser-operator-client-playwright.md create mode 100644 scripts/lib/browser-operator-client-playwright.mjs create mode 100644 scripts/lib/browser-operator-client-playwright.test.mjs diff --git a/scripts/lib/browser-operator-client-playwright.md b/scripts/lib/browser-operator-client-playwright.md new file mode 100644 index 000000000..c561c3932 --- /dev/null +++ b/scripts/lib/browser-operator-client-playwright.md @@ -0,0 +1,136 @@ +# Playwright reference fill client + +This adapter lets an unchanged, pinned Playwright client attach to one existing +Runtime page and replace or clear a field through an approved element reference. +Runtime owns the operator session, inspection grant, writer lease, quotas, +revocation and native effects. The adapter translates a small Playwright wire +profile through the [existing operator client](browser-operator-client.md). + +The exact dependency, source revision, registry integrity and extracted package +hash are recorded in `PLAYWRIGHT_OPERATOR_CLIENT` in the adjacent module. Tests +execute the unchanged SDK and check every file against that package hash. This +is a reference-fill compatibility subset. Full Browser acceptance remains in +[B04](../../docs/BROWSER_ACCEPTANCE.md#b04). + +## Invitation, approval and connection + +The owner shares the existing Browser page invitation. The operator attaches its +own Runtime session, creates `createBrowserOperatorClient` with that invitation, +and calls `requestAdmission({ actions: ["fill"], reason })`. The owner approves +inspection and field replacement/clearing in Browser. `status()` must report an +active admission before the Playwright handshake can finish. An existing active +admission is also usable; it is not evidence of invitation or approval by itself. + +The operator host creates `createPlaywrightOperatorAdapter({ client, accessKey })` +and owns its WebSocket listener. Use a separate random access key for this one +adapter; Runtime session credentials stay in the native operator client. Bind +local listeners to loopback, limit messages to 16384 bytes, disable compression, +and retain at most one accepted connection. A remote listener needs the host's +authenticated encrypted transport and connection limits. + +For a `ws`-compatible server, forward each connection as follows. The host also +owns listener shutdown and rejects connections on unrelated routes. + +```js +server.on("connection", (socket, request) => { + let peer; + try { + peer = adapter.connect({ + headers: request.headers, + send: text => socket.send(text), + close: () => socket.close(), + }); + } catch { socket.close(1008, "operator connection rejected"); return; } + socket.on("message", (data, binary) => { + void peer.receive(binary ? null : data.toString()); + }); + socket.on("close", () => { void peer.disconnect(); }); + socket.on("error", () => { void peer.disconnect(); }); +}); +``` + +After approval, the actual pinned SDK workflow is: + +```js +const browser = await chromium.connect(endpoint, { + headers: { + authorization: `Bearer ${accessKey}`, + "x-elastos-playwright-client": PLAYWRIGHT_OPERATOR_CLIENT.version, + }, + timeout: 3000, +}); +const [context] = browser.contexts(); +const [page] = context.pages(); +let snapshot = await adapter.inspect(); +const field = snapshot.nodes.find(node => node.role === "textbox" && node.name === "Message"); +if (!field) throw new Error("Expected invited field is absent"); +await page.locator(field.selector).fill("replacement", { timeout: 1500 }); +snapshot = await adapter.inspect(); +// Use the reference from this new inspection, selected by the caller. +const current = snapshot.nodes.find(node => node.role === "textbox" && node.name === "Message"); +if (!current) throw new Error("Expected invited field is absent"); +await page.locator(current.selector).clear({ timeout: 1500 }); +const observed = await adapter.inspect(); +await browser.close(); +const release = await adapter.closed; +// Check release.detached. If false, retain its code/request_id for reconciliation. +``` + +The caller chooses the intended node from approved inspection. This example's +name lookup is application code, not Playwright locator resolution. Inspection +values keep the Runtime snapshot's bounds and sensitive-field policy. + +## Exact limits + +The accepted wire commands are JavaScript `initialize` and strict main-frame +`fill` with an exact selector returned by `adapter.inspect()`. The SDK's real +`Locator.clear()` sends `fill("")`. References expire after 30 seconds from the +start of inspection; a new snapshot replaces the adapter's reference set. +Runtime checks document identity and authority again for each effect. + +Fill retains Runtime's supported text inputs and textarea, 1024 UTF-8 byte limit, +and control-character restriction. It performs one native dispatch and returns +its completed receipt. It does not implement Playwright's general actionability +or automatic waiting. Explicit `force: true`, CSS/role selectors, selector chains, +click, navigation, evaluation, `inputValue`, screenshots, events, network access, +new pages/contexts and page/context close return capability or reference errors. +The required SDK support objects expose no command authority. + +Cached SDK getters can still return local object state: one context and page, +an empty URL/name, and the adapter profile as `browser.version()`. These are +transport projections, not Engine version, document state or load evidence. +SDK-only waits and local state setters retain upstream behavior; the adapter +does not advertise their conformance. + +Timeouts cap each operation at 10 seconds, including a zero SDK timeout. The +existing Runtime client's per-request deadline also applies. Concurrent work +fails with `adapter_busy`; the adapter keeps no pending command queue. Unknown +wire methods fail explicitly; malformed, oversized or replayed messages end the +operator connection. Client network exposure is rejected during authentication. + +An uncertain write returns `operator_reconciliation_required` and its Runtime +request ID. Further writes need a new admission; inspection/reconciliation is +observation only. A failed write or lost response is never replayed by the adapter. + +Connected `browser.close()` closes the SDK transport and triggers Runtime +operator detach. Upstream marks its local Page objects closed on disconnect; +Runtime preserves the owner's page. Await `adapter.closed` separately to verify +release. A pending native effect or failed detach remains explicitly unconfirmed. +The host must call `adapter.disconnect()` when shutting down its listener too. + +## Source conformance test + +Provide an unchanged extracted `playwright-core` npm package matching the module's +pin. Browser binaries and a provider process are unnecessary. + +```sh +BROWSER_OPERATOR_PLAYWRIGHT_CORE=/path/to/playwright-core \ + node --test scripts/lib/browser-operator-client-playwright.test.mjs +``` + +Tests use the actual SDK connection, initializer validators, Browser, Page, Frame +and Locator implementations. A small Runtime API fixture records native contract +calls; ephemeral loopback WebSockets are closed after each case. This proves the +client wire translation and failure fences. Installed DOM effects, owner UX, +remote lifecycle, full Playwright behavior and the shared B04 workflow still need +their respective product evidence. diff --git a/scripts/lib/browser-operator-client-playwright.mjs b/scripts/lib/browser-operator-client-playwright.mjs new file mode 100644 index 000000000..18c092774 --- /dev/null +++ b/scripts/lib/browser-operator-client-playwright.mjs @@ -0,0 +1,192 @@ +import { timingSafeEqual } from "node:crypto"; +import { BrowserOperatorError } from "./browser-operator-client.mjs"; + +export const PLAYWRIGHT_OPERATOR_CLIENT = Object.freeze({ + package: "playwright-core", version: "1.59.1", + revision: "d466ac5358cae058cdc75d2ae3ab3ad220042730", + profile: "runtime-reference-fill-v1", + tarball: "https://registry.npmjs.org/playwright-core/-/playwright-core-1.59.1.tgz", + integrity: "sha512-HBV/RJg81z5BiiZ9yPzIiClYV/QMsDCKUyogwH9p3MCP6IYjUFu/MActgYAvK0oWyV9NlwM3GLBjADyWgydVyg==", + treeSha256: "b7195ef48f7c5fd966810f8f7a9e8e27d6bddfb6391c30c5e614e2e351fa7e2c", +}); + +const requireValue = (ok, code) => { if (!ok) throw new BrowserOperatorError(code); }; +const object = value => value !== null && typeof value === "object" && !Array.isArray(value); +const only = (value, keys) => object(value) && Object.keys(value).every(key => keys.includes(key)); +const channel = guid => ({ guid }); +const unsupported = () => { throw new BrowserOperatorError("capability_unsupported", 501); }; + +// The caller owns the authenticated WebSocket transport. This adapter owns one +// operator connection to one existing Runtime page, with no provider transport. +export function createPlaywrightOperatorAdapter({ client, accessKey, now = () => performance.now() }) { + requireValue(client && typeof client.pageId === "string" && + ["status", "inspect", "input", "detach"].every(key => typeof client[key] === "function") && + typeof accessKey === "string" && /^[a-zA-Z0-9_-]{16,128}$/.test(accessKey), "adapter_configuration_invalid"); + let peer, initialized = false, lastId = 0, active = null, stopped = false; + let bindings = new Map(), expires = 0, disconnectPromise, lastUncertain, writeUncertain = false; + let resolveClosed; + const closed = new Promise(resolve => { resolveClosed = resolve; }); + const lifetime = new AbortController(); + const guids = new Set(); + + function authenticate(headers) { + const h = new Headers(headers); + const actual = Buffer.from(h.get("authorization") || ""); + const expected = Buffer.from(`Bearer ${accessKey}`); + requireValue(actual.length === expected.length && timingSafeEqual(actual, expected) && + !h.has("cookie") && !h.has("x-elastos-home-token"), "adapter_authority_required"); + requireValue(h.get("x-elastos-playwright-client") === PLAYWRIGHT_OPERATOR_CLIENT.version && + h.get("x-playwright-browser") === "chromium", "adapter_client_version_unsupported"); + requireValue(!h.get("x-playwright-proxy"), "capability_unsupported"); + } + function send(message) { + if (stopped) return; + try { peer.send(JSON.stringify(message)); } + catch { void disconnect(); } + } + function create(parent, type, guid, initializer = {}) { + guids.add(guid); + send({ guid: parent, method: "__create__", params: { type, guid, initializer } }); + } + function graph() { + // These initializers are required by the unchanged pinned SDK. Every command + // on the support objects is rejected. No executable or website URL is claimed. + for (const name of ["chromium", "firefox", "webkit"]) + create("", "BrowserType", name, { name, executablePath: "" }); + create("", "Android", "android"); + create("", "Electron", "electron"); + create("", "Browser", "browser", { name: "chromium", browserName: "chromium", version: "runtime-reference-fill-v1" }); + create("browser", "Tracing", "tracing"); + create("browser", "Debugger", "debugger"); + create("browser", "APIRequestContext", "request", { tracing: channel("tracing") }); + create("browser", "BrowserContext", "context", { debugger: channel("debugger"), + requestContext: channel("request"), tracing: channel("tracing"), options: {} }); + create("context", "Frame", "frame", { url: "", name: "", loadStates: [] }); + create("context", "Page", "page", { mainFrame: channel("frame"), isClosed: false }); + send({ guid: "page", method: "__adopt__", params: { guid: "frame" } }); + send({ guid: "context", method: "page", params: { page: channel("page") } }); + send({ guid: "browser", method: "context", params: { context: channel("context") } }); + create("", "Playwright", "playwright", { chromium: channel("chromium"), firefox: channel("firefox"), + webkit: channel("webkit"), android: channel("android"), electron: channel("electron"), + preLaunchedBrowser: channel("browser") }); + } + async function run(work, timeout = 10000) { + requireValue(!stopped, "operator_detached"); + requireValue(!active, "adapter_busy"); + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeout); + const signal = AbortSignal.any([controller.signal, lifetime.signal]); + const pending = Promise.resolve().then(() => { + signal.throwIfAborted(); + return work(signal); + }); + active = pending; + try { + const result = await pending; + requireValue(!signal.aborted, "operator_deadline_exceeded"); + return result; + } finally { clearTimeout(timer); active = null; } + } + async function inspect() { + requireValue(initialized, "adapter_initialization_required"); + return run(async signal => { + bindings.clear(); expires = 0; + const started = now(); + const snapshot = await client.inspect({ signal }); + requireValue(!signal.aborted && !stopped, "operator_detached"); + const next = new Map(); + const nodes = snapshot.nodes.map(node => { + const selector = `runtime-ref=${snapshot.document_generation}:${node.ref}`; + next.set(selector, { document_generation: snapshot.document_generation, ref: node.ref }); + return { ...node, selector }; + }); + bindings = next; expires = started + 30000; + return { ...snapshot, nodes }; + }); + } + async function dispatch(message) { + const { guid, method, params } = message; + if (guid === "" && method === "initialize") { + requireValue(!initialized, "adapter_already_initialized"); + requireValue(only(params, ["sdkLanguage"]) && params.sdkLanguage === "javascript", "capability_unsupported"); + return run(async signal => { + const status = await client.status({ signal }); + requireValue(status.status === "active", "operator_admission_inactive"); + requireValue(!stopped && !signal.aborted, "operator_detached"); + graph(); initialized = true; + return { playwright: channel("playwright") }; + }); + } + requireValue(initialized && guids.has(guid), "adapter_object_invalid"); + if (guid !== "frame" || method !== "fill") unsupported(); + requireValue(only(params, ["selector", "value", "strict", "timeout", "force"]) && params.strict === true && + (params.force === undefined || params.force === false) && typeof params.selector === "string" && + Number.isFinite(params.timeout) && params.timeout >= 0, "capability_unsupported"); + const binding = bindings.get(params.selector); + requireValue(binding && now() < expires, "stale_inspection"); + requireValue(!writeUncertain, "operator_reconciliation_required"); + return run(async signal => { + try { + const result = await client.input({ ...binding, action: "fill", text: params.value }, { signal }); + // A late response is observation, not permission to retry this write. + if (signal.aborted || result.outcome !== "completed") + throw new BrowserOperatorError("operator_reconciliation_required", 409, { request_id: result.request_id }); + return {}; + } catch (error) { + if (error.code === "operator_reconciliation_required" || signal.aborted) writeUncertain = true; + if (/^[a-f0-9]{32}$/.test(error.details?.request_id || "")) lastUncertain = error.details.request_id; + throw error; + } + }, Math.min(params.timeout || 10000, 10000)); + } + async function receive(raw) { + if (stopped) return; + let message; + try { + requireValue(typeof raw === "string" && Buffer.byteLength(raw) <= 16384, "adapter_message_invalid"); + message = JSON.parse(raw); + requireValue(only(message, ["id", "guid", "method", "params", "metadata"]) && + Number.isSafeInteger(message.id) && message.id > lastId && typeof message.guid === "string" && + message.guid.length <= 128 && typeof message.method === "string" && message.method.length <= 80 && + (message.params === undefined || object(message.params)), "adapter_message_invalid"); + lastId = message.id; + } catch { void disconnect(); return; } + try { send({ id: message.id, result: await dispatch(message) }); } + catch (error) { + const code = error instanceof BrowserOperatorError ? error.code : "adapter_failed"; + const requestId = error instanceof BrowserOperatorError && error.details?.request_id; + if (typeof requestId === "string" && /^[a-f0-9]{32}$/.test(requestId)) lastUncertain = requestId; + send({ id: message.id, error: { error: { name: "Error", + message: lastUncertain && code === "operator_reconciliation_required" ? `${code}: request_id=${lastUncertain}` : code } } }); + } + } + function disconnect() { + if (disconnectPromise) return disconnectPromise; + stopped = true; lifetime.abort(); bindings.clear(); + disconnectPromise = (async () => { + // Yield before releasing so reentrant transport-close callbacks see the + // same promise. Closing a socket alone never confirms Runtime release. + await Promise.resolve(); + await active?.catch(() => {}); + let result; + try { result = await client.detach(); } + catch (error) { + result = { detached: false, page_closed: false, requires_reconciliation: true, + code: error instanceof BrowserOperatorError ? error.code : "adapter_detach_unconfirmed", + ...(lastUncertain ? { request_id: lastUncertain } : {}) }; + } + resolveClosed(result); + return result; + })(); + try { peer?.close(); } catch { /* Runtime release still runs after transport failure. */ } + return disconnectPromise; + } + function connect({ headers, send: sendWire, close }) { + authenticate(headers); + requireValue(!peer && !stopped && typeof sendWire === "function" && typeof close === "function", "adapter_connection_unavailable"); + peer = { send: sendWire, close }; + return Object.freeze({ receive, disconnect }); + } + return Object.freeze({ connect, inspect, disconnect, closed, pageId: client.pageId, + profile: PLAYWRIGHT_OPERATOR_CLIENT.profile }); +} diff --git a/scripts/lib/browser-operator-client-playwright.test.mjs b/scripts/lib/browser-operator-client-playwright.test.mjs new file mode 100644 index 000000000..c55afed1b --- /dev/null +++ b/scripts/lib/browser-operator-client-playwright.test.mjs @@ -0,0 +1,287 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { createRequire } from "node:module"; +import { readFile, readdir } from "node:fs/promises"; +import { resolve, join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { createHash } from "node:crypto"; +import { once } from "node:events"; +import { createBrowserOperatorClient } from "./browser-operator-client.mjs"; +import { createPlaywrightOperatorAdapter, PLAYWRIGHT_OPERATOR_CLIENT } from "./browser-operator-client-playwright.mjs"; + +// Explicit dependency input: the unchanged npm package, without browser binaries. +assert.ok(process.env.BROWSER_OPERATOR_PLAYWRIGHT_CORE, "Set BROWSER_OPERATOR_PLAYWRIGHT_CORE to the pinned playwright-core package directory"); +const sdkPath = resolve(process.env.BROWSER_OPERATOR_PLAYWRIGHT_CORE); +const metadata = JSON.parse(await readFile(join(sdkPath, "package.json"), "utf8")); +assert.equal(metadata.name, PLAYWRIGHT_OPERATOR_CLIENT.package); +assert.equal(metadata.version, PLAYWRIGHT_OPERATOR_CLIENT.version); +const { chromium } = await import(pathToFileURL(join(sdkPath, "index.mjs"))); +const requireSDK = createRequire(join(sdkPath, "package.json")); +const { wsServer: WebSocketServer, ws: WebSocket } = requireSDK("./lib/utilsBundle.js"); +const accessKey = "playwright_source_fixture_key"; +const generation = "a".repeat(32), snapshotId = "b".repeat(32); +const pageId = "invited-runtime-page", admissionId = "invited-admission"; +const headers = { authorization: `Bearer ${accessKey}`, "x-elastos-playwright-client": "1.59.1" }; +const rawHeaders = { ...headers, "x-playwright-browser": "chromium" }; + +test("every SDK file matches the unchanged registry tarball tree", async () => { + const paths = []; + async function walk(directory, prefix = "") { + for (const entry of await readdir(directory, { withFileTypes: true })) { + const relative = `${prefix}${entry.name}`; + if (entry.isDirectory()) await walk(join(directory, entry.name), `${relative}/`); + else { assert.equal(entry.isFile(), true); paths.push(relative); } + } + } + await walk(sdkPath); + const hash = createHash("sha256"); + for (const path of paths.sort()) hash.update(`${path}\0${createHash("sha256").update(await readFile(join(sdkPath, path))).digest("hex")}\n`); + assert.equal(hash.digest("hex"), PLAYWRIGHT_OPERATOR_CLIENT.treeSha256); +}); + +function fixture() { + const state = { value: "initial", status: "active", generation, snapshotId, calls: [], effects: [], + pageOpen: true, writer: true, detachCount: 0, loseInputReply: false, detachUncertain: false }; + const admission = () => ({ schema: "elastos.browser.operator-admission/v1", request_id: admissionId, + status: state.status, capability: "fixture-write-grant", inspection_capability: "fixture-read-grant", + writer_acquired: state.writer }); + const fetchImpl = async (url, options) => { + options.signal.throwIfAborted(); + assert.equal(options.headers.Authorization, "Bearer operator-session"); + assert.equal(options.headers["x-elastos-home-token"], undefined); + assert.equal(new URL(url).origin, "https://runtime.invalid"); + const path = new URL(url).pathname; + const prefix = `/api/apps/browser/pages/${pageId}`; + state.calls.push({ path, method: options.method }); + if (path === `${prefix}/operator-requests/${admissionId}`) { + if (state.beforeStatus) await state.beforeStatus(options.signal); + return Response.json(admission()); + } + if (path === `${prefix}/operator-requests/${admissionId}/detach`) { + state.detachCount++; + if (state.detachUncertain) return Response.json({ code: "operator_native_effect_pending" }, { status: 409 }); + state.status = "revoked"; state.writer = false; + return Response.json(admission()); + } + if (state.status !== "active") return Response.json({ code: "operator_revoked" }, { status: 403 }); + if (path === `${prefix}/operator-requests/${admissionId}/inspect`) { + assert.equal(options.headers["x-elastos-capability"], "fixture-read-grant"); + return Response.json({ schema: "elastos.browser.inspect-result/v1", page_id: pageId, + snapshot_id: state.snapshotId, document_generation: state.generation, truncated: false, next_cursor: null, + nodes: [{ ref: `${state.snapshotId}:0`, role: "textbox", name: "Message", description: "", value: state.value }] }); + } + if (path === `${prefix}/input`) { + assert.equal(options.headers["x-elastos-capability"], "fixture-write-grant"); + const { event } = JSON.parse(options.body); + assert.equal(event.admission_id, admissionId); + assert.equal(event.action, "fill"); + if (event.document_generation !== state.generation || event.ref !== `${state.snapshotId}:0`) + return Response.json({ code: "stale_inspection" }, { status: 409 }); + state.effects.push(event); state.value = event.text; + if (state.inputStarted) state.inputStarted(); + if (state.loseInputReply) return new Promise((_, reject) => { + options.signal.addEventListener("abort", () => reject(options.signal.reason), { once: true }); + }); + return Response.json({ schema: "elastos.browser.ref-input-result/v1", page_id: pageId, + admission_id: admissionId, document_generation: event.document_generation, request_id: event.request_id, + accepted: true, outcome: "completed" }); + } + assert.fail(`unexpected Runtime route ${path}`); + }; + const client = createBrowserOperatorClient({ origin: "https://runtime.invalid", sessionToken: "operator-session", + pageId, admissionId, fetchImpl, timeoutMs: 500 }); + return { state, client }; +} + +async function setup(t, options = {}) { + const { state, client } = fixture(); + let clock = 0; + const adapter = createPlaywrightOperatorAdapter({ client, accessKey, now: () => clock }); + const wire = [], rejectionCodes = []; + const server = new WebSocketServer({ host: "127.0.0.1", port: 0, maxPayload: 16384, perMessageDeflate: false }); + server.on("connection", (socket, request) => { + let connection; + try { + connection = adapter.connect({ headers: request.headers, + send: raw => { wire.push(JSON.parse(raw)); socket.send(raw); }, close: () => socket.close() }); + } catch (error) { rejectionCodes.push(error.code); socket.close(1008, error.code); return; } + socket.on("message", (raw, binary) => { void connection.receive(binary ? null : raw.toString()); }); + socket.on("close", () => { void connection.disconnect(); }); + socket.on("error", () => { void connection.disconnect(); }); + }); + await once(server, "listening"); + const endpoint = `ws://127.0.0.1:${server.address().port}`; + let browser; + t.after(async () => { + await browser?.close(); + await adapter.disconnect(); + for (const socket of server.clients) socket.terminate(); + await new Promise(resolve => server.close(resolve)); + }); + if (options.connect !== false) browser = await chromium.connect(endpoint, { headers, timeout: 2000 }); + return { state, client, adapter, wire, rejectionCodes, endpoint, browser, + page: browser?.contexts()[0]?.pages()[0], advance: ms => { clock += ms; } }; +} + +test("unchanged pinned SDK connects, fills, clears and disconnects with the Runtime page preserved", async t => { + const f = await setup(t); + assert.equal(f.browser.constructor.name, "Browser"); + assert.equal(f.browser.contexts().length, 1); + assert.equal(f.browser.contexts()[0].pages().length, 1); + assert.equal(f.page.constructor.name, "Page"); + assert.equal(f.page.mainFrame().constructor.name, "Frame"); + assert.equal(f.adapter.pageId, pageId); + const snapshot = await f.adapter.inspect(); + const locator = f.page.locator(snapshot.nodes[0].selector); + assert.equal(locator.constructor.name, "Locator"); + await locator.fill("replacement"); + assert.equal((await f.adapter.inspect()).nodes[0].value, "replacement"); + await locator.clear(); + assert.equal((await f.adapter.inspect()).nodes[0].value, ""); + assert.deepEqual(f.state.effects.map(e => e.text), ["replacement", ""]); + assert.equal(new Set(f.state.effects.map(e => e.request_id)).size, 2); + await f.browser.close(); + assert.deepEqual(await f.adapter.closed, { detached: true, page_closed: false }); + assert.equal(f.state.pageOpen, true); + assert.equal(f.state.writer, false); + assert.equal(f.state.detachCount, 1); + assert.doesNotMatch(JSON.stringify(f.wire), /operator-session|fixture-write-grant|fixture-read-grant/); +}); + +test("ordinary selectors and selector compositions never reach Runtime input", async t => { + const f = await setup(t); + const selector = (await f.adapter.inspect()).nodes[0].selector; + for (const locator of [f.page.locator("input"), f.page.getByRole("textbox", { name: "Message" }), + f.page.locator(selector).first(), f.page.locator(selector).filter({ hasText: "Message" }), + f.page.locator(selector.replace(generation, "c".repeat(32)))]) + await assert.rejects(() => locator.fill("x"), /stale_inspection/); + assert.equal(f.state.effects.length, 0); +}); + +test("inspection refresh, expiry and Runtime generation checks fence old references", async t => { + const f = await setup(t); + const old = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + f.state.snapshotId = "c".repeat(32); + const current = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + await assert.rejects(() => old.fill("old"), /stale_inspection/); + f.advance(30000); + await assert.rejects(() => current.fill("expired"), /stale_inspection/); + const next = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + f.state.generation = "d".repeat(32); + await assert.rejects(() => next.fill("wrong document"), /operator_reconciliation_required: request_id=[a-f0-9]{32}/); + await assert.rejects(() => next.clear(), /operator_reconciliation_required/); + assert.equal(f.state.effects.length, 0); +}); + +test("owner revocation rejects the next real-client fill and inspection", async t => { + const f = await setup(t); + const locator = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + f.state.status = "revoked"; + await assert.rejects(() => locator.fill("revoked"), /operator_revoked/); + await assert.rejects(() => f.adapter.inspect(), /operator_revoked/); + assert.equal(f.state.effects.length, 0); +}); + +test("real-client timeout and concurrent request cannot replay an uncertain native effect", async t => { + const f = await setup(t); + const locator = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + let started; + const inputStarted = new Promise(resolve => { started = resolve; }); + f.state.inputStarted = started; f.state.loseInputReply = true; + const first = assert.rejects(() => locator.fill("one effect", { timeout: 60 }), /operator_reconciliation_required: request_id=[a-f0-9]{32}/); + await inputStarted; + await assert.rejects(() => locator.clear(), /adapter_busy/); + await first; + await assert.rejects(() => locator.fill("retry"), /operator_reconciliation_required/); + assert.equal(f.state.effects.length, 1); +}); + +test("disconnect aborts an outstanding write, releases once and exposes unconfirmed release", async t => { + const f = await setup(t); + const locator = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + let started; + const inputStarted = new Promise(resolve => { started = resolve; }); + f.state.inputStarted = started; f.state.loseInputReply = true; f.state.detachUncertain = true; + const first = assert.rejects(() => locator.fill("one effect"), /closed/); + await inputStarted; + await f.browser.close(); await first; + const release = await f.adapter.closed; + assert.equal(release.detached, false); + assert.equal(release.requires_reconciliation, true); + assert.equal(release.request_id, f.state.effects[0].request_id); + assert.equal(release.code, "operator_native_effect_pending"); + assert.deepEqual(await f.adapter.disconnect(), release); + assert.equal(f.state.detachCount, 1); + assert.equal(f.state.effects.length, 1); + assert.equal(f.state.pageOpen, true); + await assert.rejects(() => f.adapter.inspect(), /operator_detached/); +}); + +test("deadline during grant acquisition prevents input dispatch", async t => { + const f = await setup(t); + const locator = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + f.state.beforeStatus = signal => new Promise((_, reject) => { + signal.addEventListener("abort", () => reject(signal.reason), { once: true }); + }); + await assert.rejects(() => locator.fill("too late", { timeout: 20 }), /operator_transport_failed/); + f.state.beforeStatus = null; + assert.equal(f.state.effects.length, 0); +}); + +test("actual SDK rejects foreign authority, version and network exposure before admission reads", async t => { + const f = await setup(t, { connect: false }); + for (const options of [ + { headers: { ...headers, authorization: "Bearer wrong-key" } }, + { headers: { ...headers, "x-elastos-home-token": "owner-credential" } }, + { headers: { ...headers, cookie: "owner-cookie" } }, + { headers: { ...headers, "x-elastos-playwright-client": "1.60.0" } }, + { headers, exposeNetwork: "*" }, + ]) await assert.rejects(() => chromium.connect(f.endpoint, { ...options, timeout: 1000 })); + assert.deepEqual(f.rejectionCodes, ["adapter_authority_required", "adapter_authority_required", "adapter_authority_required", + "adapter_client_version_unsupported", "capability_unsupported"]); + assert.equal(f.state.calls.length, 0); +}); + +test("a second real SDK connection cannot take over the active adapter", async t => { + const f = await setup(t); + await assert.rejects(() => chromium.connect(f.endpoint, { headers, timeout: 1000 })); + assert.deepEqual(f.rejectionCodes, ["adapter_connection_unavailable"]); + const locator = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + await locator.fill("first operator remains"); + assert.equal(f.state.effects.length, 1); +}); + +test("raw duplicate IDs and malformed messages close only the operator transport", async t => { + for (const mutation of [message => message, () => "invalid json", () => "x".repeat(16385)]) { + const f = await setup(t, { connect: false }); + const socket = new WebSocket(f.endpoint, { headers: rawHeaders }); + await once(socket, "open"); + const initialized = new Promise(resolve => socket.on("message", raw => { + const message = JSON.parse(raw.toString()); + if (message.id === 1) resolve(message); + })); + const initialize = JSON.stringify({ id: 1, guid: "", method: "initialize", params: { sdkLanguage: "javascript" } }); + socket.send(initialize); assert.ok((await initialized).result); + const disconnected = once(socket, "close"); + socket.send(mutation(initialize)); + await disconnected; + assert.equal((await f.adapter.closed).detached, true); + assert.equal(f.state.pageOpen, true); + assert.equal(f.state.effects.length, 0); + } +}); + +test("unsupported SDK methods fail explicitly before Runtime dispatch", async t => { + const f = await setup(t); + const locator = f.page.locator((await f.adapter.inspect()).nodes[0].selector); + const calls = f.state.calls.length; + for (const operation of [ + () => locator.inputValue(), () => locator.click(), () => locator.isVisible(), () => locator.fill("x", { force: true }), + () => f.page.evaluate(() => document.title), () => f.page.goto("https://example.invalid"), + () => f.browser.newContext(), () => f.browser.contexts()[0].newPage(), + () => f.page.screenshot(), () => f.page.context().newCDPSession(f.page), + () => f.page.request.get("https://example.invalid"), () => f.page.close(), () => f.page.context().close(), + ]) await assert.rejects(operation, /capability_unsupported/); + assert.equal(f.state.calls.length, calls); + assert.equal(f.state.pageOpen, true); +}); From c17776f5e6a0c9511356a16249f477e7705fd6de Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 8 Sep 2026 18:15:03 -0500 Subject: [PATCH 211/535] test(browser): bind qualification to selected services and actual VM identity --- scripts/browser-qualification-audit.mjs | 70 ++++++- scripts/browser-qualification-runner.mjs | 37 ++-- .../lib/browser-qualification-observer.mjs | 14 +- scripts/lib/browser-qualification.test.mjs | 191 ++++++++++++++++-- 4 files changed, 273 insertions(+), 39 deletions(-) diff --git a/scripts/browser-qualification-audit.mjs b/scripts/browser-qualification-audit.mjs index 3e78715e4..a6ec4d215 100644 --- a/scripts/browser-qualification-audit.mjs +++ b/scripts/browser-qualification-audit.mjs @@ -6,6 +6,7 @@ import { resolve, dirname, basename } from "node:path"; import { pathToFileURL } from "node:url"; import { isDeepStrictEqual } from "node:util"; import { controlledTonePresent } from "./lib/browser-journey-audio.mjs"; +import { browserJourneyTargetConfig } from "./lib/browser-journey-target.mjs"; export const QUALIFICATION_SCHEMA = "elastos.browser.qualification/v1"; export const MODES = ["lifecycle", "cold", "warm", "media", "mixed"]; @@ -23,7 +24,8 @@ export function candidateFingerprint(plan, inputs) { "candidate_frozen_input_mismatch:" + a.role); requireEvidence(inputs.some(f => f.role === "operator_coordinates" && f.path === plan.runtime.operator_coords) && ["scripts/browser-qualification-runner.mjs", "scripts/browser-qualification-audit.mjs", - "scripts/lib/browser-qualification-observer.mjs", "scripts/home-passkey-virtual-auth-smoke.mjs"].every(role => + "scripts/lib/browser-qualification-observer.mjs", "scripts/lib/browser-journey-target.mjs", + "scripts/home-passkey-virtual-auth-smoke.mjs"].every(role => inputs.some(f => f.role === role && hash(f.sha256))), "frozen_harness_inputs_required"); return sha256(JSON.stringify({ plan, inputs })); } @@ -195,6 +197,52 @@ export function deriveMedia(rows, { page_id, run, engine_id, exit_id, mode = "me samples: rows.length, interactions: actions.length, hidden_cycles: hidden, idle_cycles: idle, max_sample_gap_ms: gap, video, audio }; } +export function qualificationTarget(runtime) { + requireEvidence(runtime.allow_remote_fixture === undefined || typeof runtime.allow_remote_fixture === "boolean", + "remote_fixture_option_invalid"); + requireEvidence(runtime.fixture_admin_origin === undefined || + typeof runtime.fixture_admin_origin === "string", + "fixture_admin_origin_invalid"); + requireEvidence(typeof runtime.fixture_origin === "string" && runtime.fixture_origin.length > 0, + "fixture_origin_required"); + for (const key of ["engine_id", "exit_id"]) requireEvidence(typeof runtime[key] === "string" && + (runtime[key] === "" || /^[A-Za-z0-9:_-]{1,128}$/.test(runtime[key])), "selected_service_identity_required"); + requireEvidence(runtime.remote_exit_id === undefined || runtime.remote_exit_id === runtime.exit_id, + "conflicting_legacy_exit_selection"); + return browserJourneyTargetConfig({ + HOME_VIRTUAL_AUTH_BROWSER_ENGINE_ID: runtime.engine_id, + HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ORIGIN: runtime.fixture_origin, + HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ADMIN_ORIGIN: runtime.fixture_admin_origin, + HOME_VIRTUAL_AUTH_BROWSER_ALLOW_REMOTE_FIXTURE: runtime.allow_remote_fixture === true ? "1" : "0", + }); +} +export function qualificationVmIdentity(proof, pageId) { + requireEvidence(proof?.schema === "elastos.browser.vz-transport-public-proof/v1" && proof.page_id === pageId && + typeof pageId === "string" && /^page:vz-[a-f0-9]{64}$/.test(pageId) && + /^browser-vm-[a-f0-9]{64}$/.test(proof.vm_id) && + /^sha256:[a-f0-9]{64}$/.test(proof.generation) && /^sha256:[a-f0-9]{64}$/.test(proof.binding_hash), + "launch_transport_identity_required"); + return Object.fromEntries(["schema", "page_id", "vm_id", "generation", "binding_hash"].map(k => [k, proof[k]])); +} +export function validateLocalLaunchSample(sample, pageId, plan, before) { + const control = sample?.control, owner = plan.runtime.resource_roots.find(p => p.role === "vm_control"); + requireEvidence(sample?.identity_verified === true && finite(sample.rss_bytes) && sample.rss_bytes >= 0 && + sample.rss_bytes <= plan.limits.rss_bytes && finite(sample.cpu_percent) && sample.cpu_percent >= 0 && + sample.cpu_percent <= plan.limits.cpu_percent && Array.isArray(sample.process_ids) && + plan.runtime.resource_roots.every(p => sample.process_ids.includes(p.pid + ":" + p.start)), + "launch_resource_identity_required"); + requireEvidence(owner && control?.schema === "elastos.browser.vm-control-service.status/v1" && control.ok === true && + control.pid === owner.pid && before?.pid === owner.pid && + control.control_service?.schema === "elastos.browser.vm-control-service.identity/v1" && + typeof control.control_service.service_id === "string" && /^service:[a-f0-9]{64}$/.test(control.control_service.service_id) && + isDeepStrictEqual(control.control_service, before.control_service) && + hash(control.config_fingerprint) && control.config_fingerprint === before.config_fingerprint && + control.direct_network === false && control.network_mode === "runtime_net_only" && + control.active_pages > 0 && control.page_ids?.includes(pageId) && + control.lifecycle?.schema === "elastos.browser.lifecycle-status/v1" && + control.lifecycle.sessions?.some(s => s.phase === "ACTIVE_SESSION" && s.warm_vm === false && + s.page_id === "sha256:" + sha256(pageId).slice(0, 16)), "launch_control_identity_required"); +} export function validatePlan(plan) { requireEvidence(plan?.schema === "elastos.browser.qualification-plan/v1" && MODES.includes(plan.mode), "plan_schema_or_mode"); requireEvidence(plan.mode !== "warm", "warm_conditioning_unsupported"); @@ -215,9 +263,8 @@ export function validatePlan(plan) { "signed_home_reuse_option_invalid"); requireEvidence(typeof runtime?.viewer_version === "string" && runtime.viewer_version.length > 0 && runtime.viewer_version.length < 120 && typeof runtime.headed === "boolean", "viewer_identity_required"); - requireEvidence(typeof runtime.engine_id === "string" && typeof runtime.exit_id === "string" && - runtime.engine_id.length <= 256 && runtime.exit_id.length <= 256, "selected_service_identity_required"); - for (const name of ["base_url", "fixture_origin"]) { + qualificationTarget(runtime); + for (const name of ["base_url"]) { const url = new URL(runtime?.[name]); requireEvidence(url.protocol === "http:" && ["localhost", "127.0.0.1", "[::1]"].includes(url.hostname) && !url.username && !url.password && url.pathname === "/" && !url.search && !url.hash, "task_loopback_origin_required"); @@ -234,6 +281,8 @@ export function validatePlan(plan) { "resource_root_identity_required"); requireEvidence(runtime.resource_roots.filter(r => r.role === "runtime").length === 1, "runtime_process_role_required"); + requireEvidence(runtime.resource_roots.filter(r => r.role === "vm_control").length === 1, + "vm_control_process_role_required"); requireEvidence(plan.require_operator === true, "combined_operator_journey_required"); requireEvidence(["local", "wan"].includes(plan.network?.profile) && finite(plan.network.control_rtt_ms) && plan.network.control_rtt_ms >= 0, "network_profile_required"); @@ -312,8 +361,13 @@ export function validateAttempt(attempt, plan) { q.launch.clock === "host_monotonic" && finite(q.launch.usable_frame_ms) && q.launch.usable_frame_ms > 0, "monotonic_launch_witness_required"); requireEvidence(q.services?.engine_id === plan.runtime.engine_id && q.services?.exit_id === plan.runtime.exit_id && - attempt.observed_page_ids?.includes(attempt.journey.page_id) && attempt.active_vm_keys?.length > 0, - "selected_service_or_local_vm_witness_missing"); + attempt.observed_page_ids?.includes(attempt.journey.page_id), "selected_service_or_local_vm_witness_missing"); + requireEvidence(isDeepStrictEqual(qualificationVmIdentity(q.vm_identity, attempt.journey.page_id), + qualificationVmIdentity(j.close.receipt.transport_proof, attempt.journey.page_id)), "launch_close_vm_identity_mismatch"); + validateLocalLaunchSample(attempt.local_launch_sample, attempt.journey.page_id, plan, attempt.before); + requireEvidence(attempt.after.pid === attempt.before.pid && + attempt.after.config_fingerprint === attempt.before.config_fingerprint && + isDeepStrictEqual(attempt.after.control_service, attempt.before.control_service), "closed_control_identity_changed"); if (["media", "mixed"].includes(plan.mode)) { const o = q.observation; requireEvidence(o?.ok === true && o.duration_ms >= plan.duration_ms && @@ -366,12 +420,13 @@ export function auditQualification(receiptPath) { Number.isInteger(row.attempt) && row.attempt >= 1 && row.attempt <= 100 && ["resources", "media"].includes(row.kind), "sample_sequence_or_kind"); lastAt = row.at_ms; - const rows = observed.get(row.attempt) || { resources: 0, media: [], parent_times: [] }; + const rows = observed.get(row.attempt) || { resources: 0, resource_hashes: new Set(), media: [], parent_times: [] }; if (row.kind === "resources") { requireEvidence(row.value?.identity_verified === true && finite(row.value.rss_bytes) && row.value.rss_bytes <= plan.limits.rss_bytes && finite(row.value.cpu_percent) && row.value.cpu_percent <= plan.limits.cpu_percent, "resource_sample_invalid"); rows.resources++; + rows.resource_hashes.add(sha256(JSON.stringify(row.value))); } else { rows.media.push(row.value); rows.parent_times.push(row.at_ms); } observed.set(row.attempt, rows); } @@ -390,6 +445,7 @@ export function auditQualification(receiptPath) { requireEvidence(a.index === index + 1 && a.candidate_fingerprint === r.candidate_fingerprint && !ids.has(j.run), "candidate_or_run_id_mismatch"); const rows = observed.get(a.index); requireEvidence(rows?.resources > 0, "resource_observation_missing"); + requireEvidence(rows.resource_hashes.has(sha256(JSON.stringify(a.local_launch_sample))), "launch_sample_not_in_raw_evidence"); requireEvidence(a.log && basename(a.log.file) === a.log.file && hash(a.log.sha256), "journey_log_required"); const logPath = resolve(dirname(receiptPath), a.log.file); requireEvidence(statSync(logPath).size <= 1024 * 1024 && sha256(readFileSync(logPath)) === a.log.sha256, diff --git a/scripts/browser-qualification-runner.mjs b/scripts/browser-qualification-runner.mjs index ca045e561..4beb0bc49 100644 --- a/scripts/browser-qualification-runner.mjs +++ b/scripts/browser-qualification-runner.mjs @@ -9,7 +9,8 @@ import { fileURLToPath, pathToFileURL } from "node:url"; import { promisify } from "node:util"; import { validatePlan, validateAttempt, cleanControl, sha256, boundedJson, requireEvidence, QUALIFICATION_SCHEMA } from "./browser-qualification-audit.mjs"; -import { candidateFingerprint, validateInstallation, validateLiveIdentity } from "./browser-qualification-audit.mjs"; +import { candidateFingerprint, validateInstallation, validateLiveIdentity, qualificationTarget } from "./browser-qualification-audit.mjs"; +import { readBrowserJourneyHealth } from "./lib/browser-journey-target.mjs"; const execute = promisify(execFile); const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); @@ -23,7 +24,8 @@ export function planTemplate() { candidate: { platform: "darwin-arm64", source_commit: "", source_tree: "", artifacts: ["runtime", "browser_ui", "engine_adapter", "image_manifest", "rootfs", "kernel", "initrd", "host_helper", "relay", "components", "installation_review"].map(role => ({ role, path: "", sha256: "" })) }, - runtime: { base_url: "", browser_ui_url: "", fixture_origin: "", profile: "", control_socket: "", operator_coords: "", + runtime: { base_url: "", browser_ui_url: "", fixture_origin: "", fixture_admin_origin: "", allow_remote_fixture: false, + profile: "", control_socket: "", operator_coords: "", browser_executable: "", node_path: "", viewer_version: "", headed: true, reuse_signed_home: false, engine_id: "", exit_id: "", resource_roots: [{ role: "runtime", pid: null, start: "" }, { role: "vm_control", pid: null, start: "" }, { role: "exit_relay", pid: null, start: "" }] }, @@ -155,6 +157,7 @@ export function journeyEnvironment(plan, index) { const env = { ...process.env }; for (const key of Object.keys(env)) if (key.startsWith("HOME_VIRTUAL_AUTH_")) delete env[key]; const p = plan.runtime; + const target = qualificationTarget(p); Object.assign(env, { ELASTOS_BASE_URL: p.base_url, HOME_URL: p.base_url.replace(/\/$/, "") + "/apps/home/", HOME_VIRTUAL_AUTH_NAME: "Browser qualification controlled Mac", @@ -168,7 +171,11 @@ export function journeyEnvironment(plan, index) { HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_RECOVERY: "0", HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_TURN_TEST_HOME: "", HOME_VIRTUAL_AUTH_BROWSER_REQUIRE_VZ_TRANSPORT: "1", - HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ORIGIN: p.fixture_origin, + HOME_VIRTUAL_AUTH_BROWSER_ENGINE_ID: target.engineId, + HOME_VIRTUAL_AUTH_BROWSER_REMOTE_EXIT_ID: p.exit_id, + HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ORIGIN: target.origin, + HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ADMIN_ORIGIN: target.adminOrigin, + HOME_VIRTUAL_AUTH_BROWSER_ALLOW_REMOTE_FIXTURE: target.allowRemote ? "1" : "0", HOME_VIRTUAL_AUTH_PROFILE: p.profile, HOME_VIRTUAL_AUTH_CLEANUP: "0", HOME_VIRTUAL_AUTH_PRESERVE_PROFILE: "1", HOME_VIRTUAL_AUTH_REUSE_SIGNED_HOME: p.reuse_signed_home === true ? "1" : "0", @@ -181,9 +188,13 @@ export function journeyEnvironment(plan, index) { if (p.browser_executable) env.ELASTOS_BROWSER_EXECUTABLE = p.browser_executable; if (p.node_path) env.NODE_PATH = p.node_path; if (p.headed === true) env.HOME_VIRTUAL_AUTH_HEADED = "1"; - if (p.remote_exit_id) env.HOME_VIRTUAL_AUTH_BROWSER_REMOTE_EXIT_ID = p.remote_exit_id; return env; } +export async function qualificationFixtureHealth(plan, fetchImpl = fetch) { + const health = await readBrowserJourneyHealth(qualificationTarget(plan.runtime), fetchImpl); + requireEvidence(health.qualification === "bounded-v1", "qualification_fixture_hook_missing"); + return health; +} export async function childProcessTable() { const { stdout } = await execute("/bin/ps", ["-axo", "pid=,ppid=,pgid=,lstart="], { timeout: 3000, maxBuffer: 512 * 1024, env: { ...process.env, LC_ALL: "C" } }); @@ -359,7 +370,7 @@ export async function runQualification(planPath, output, { signal = new AbortCon for (const path of ["scripts/home-passkey-virtual-auth-smoke.mjs", "scripts/lib/browser-journey-fixture.mjs", "scripts/lib/browser-journey-operator.mjs", "scripts/lib/browser-journey-audio.mjs", "scripts/lib/browser-journey-recovery.mjs", - "scripts/lib/browser-open-failure.mjs", + "scripts/lib/browser-open-failure.mjs", "scripts/lib/browser-journey-target.mjs", "scripts/lib/browser-journey-viewer-reload.mjs", "scripts/lib/browser-journey-turn-interruption.mjs", "scripts/lib/browser-qualification-observer.mjs", "scripts/browser-qualification-runner.mjs", "scripts/browser-qualification-audit.mjs"]) { @@ -375,9 +386,7 @@ export async function runQualification(planPath, output, { signal = new AbortCon writeFileSync(join(output, "installation-review.json"), installationBytes, { flag: "wx", mode: 0o600 }); used += installationBytes.length; receipt.live_before = await liveCandidate(plan, frozen); - const health = await fetch(plan.runtime.fixture_origin + "/health", { signal: AbortSignal.timeout(3000) }).then(r => r.json()); - requireEvidence(health.schema === "elastos.browser.journey-fixture/v1" && health.qualification === "bounded-v1", - "qualification_fixture_hook_missing"); + await qualificationFixtureHealth(plan); sampleFd = openSync(join(output, "samples.jsonl"), "wx", 0o600); const start = performance.now(); const append = (kind, attempt, value) => { @@ -396,12 +405,12 @@ export async function runQualification(planPath, output, { signal = new AbortCon signal.addEventListener("abort", abort, { once: true }); let stopSampling = false, sampleTask, logFd, logHash = createHash("sha256"), logBytes = 0; const observedChild = {}; + const launchSamples = new Map(); try { attempt.before = await controlStatus(plan.runtime.control_socket); cleanControl(attempt.before, plan.mode === "warm"); const baseline = await resources(plan.runtime.resource_roots, tracked); attempt.resource_peak = { ...baseline }; - attempt.active_vm_keys = []; attempt.observed_page_ids = []; logFd = openSync(join(output, "attempt-" + i + ".log"), "wx", 0o600); sampleTask = (async () => { @@ -417,11 +426,12 @@ export async function runQualification(planPath, output, { signal = new AbortCon requireEvidence(attempt.observed_page_ids.length < 16, "page_identity_bound"); attempt.observed_page_ids.push(id); } - for (const s of control.lifecycle?.sessions || []) if (s.vm_key_hash && !attempt.active_vm_keys.includes(s.vm_key_hash)) { - requireEvidence(attempt.active_vm_keys.length < 16, "vm_identity_bound"); - attempt.active_vm_keys.push(s.vm_key_hash); + const sample = { ...value, control }; + for (const id of control.page_ids || []) if (!launchSamples.has(id)) { + requireEvidence(launchSamples.size < 16, "launch_sample_bound"); + launchSamples.set(id, sample); } - append("resources", i, { ...value, control }); + append("resources", i, sample); await new Promise(resolve => { const timer = setTimeout(done, 5000); function done() { clearTimeout(timer); controller.signal.removeEventListener("abort", done); resolve(); } controller.signal.addEventListener("abort", done, { once: true }); @@ -434,6 +444,7 @@ export async function runQualification(planPath, output, { signal = new AbortCon writeSync(logFd, chunk); logHash.update(chunk); logBytes += chunk.length; used += chunk.length; }, sample: value => append("media", i, value) }); Object.assign(attempt, child); + attempt.local_launch_sample = launchSamples.get(child.journey?.page_id) || null; stopSampling = true; controller.abort(); await sampleTask; requireEvidence(!attempt.sampling_failure && !child.failure, attempt.sampling_failure || child.failure); attempt.after = await controlStatus(plan.runtime.control_socket); cleanControl(attempt.after); diff --git a/scripts/lib/browser-qualification-observer.mjs b/scripts/lib/browser-qualification-observer.mjs index 60ff90cb2..4b4ee667c 100644 --- a/scripts/lib/browser-qualification-observer.mjs +++ b/scripts/lib/browser-qualification-observer.mjs @@ -1,5 +1,5 @@ // Observation only: product media remains the Browser's WebRTC receiver. -import { requireEvidence, MODES, deriveMedia, TERMINAL_EFFECTS } from "../browser-qualification-audit.mjs"; +import { requireEvidence, MODES, deriveMedia, TERMINAL_EFFECTS, qualificationVmIdentity } from "../browser-qualification-audit.mjs"; export function qualificationOptions(env) { if (!env.HOME_VIRTUAL_AUTH_BROWSER_QUALIFICATION_MODE) return null; @@ -374,6 +374,7 @@ export async function createQualificationHarness(context, page, options, now = ( attempts.find(a => a.request === res.request()) : attempts.find(a => a.open_id === decodeURIComponent(path.split("/").at(-1))); if (!opened) return; + if (frame !== opened.request.frame() || new URL(res.url()).origin !== new URL(opened.request.url()).origin) return; if (path.endsWith("/open") ? res.request() !== opened.request : !opened.open_id || decodeURIComponent(path.split("/").at(-1)) !== opened.open_id) return; const body = await res.json().catch(() => null); @@ -382,7 +383,13 @@ export async function createQualificationHarness(context, page, options, now = ( body?.schema === "elastos.browser.open-status/v1" && body.status === "completed" ? body.result : null; if (body?.status === "failed" || res.status?.() >= 400) opened.outcome = "failed"; if (result?.engine_page?.page_id) { - opened.outcome = "completed"; opened.page_id = result.engine_page.page_id; match(frame); + opened.outcome = "completed"; opened.page_id = result.engine_page.page_id; + const vmIdentity = result.engine_page.transport_proof ? + qualificationVmIdentity(result.engine_page.transport_proof, opened.page_id) : null; + requireEvidence(!opened.vm_identity || JSON.stringify(opened.vm_identity) === JSON.stringify(vmIdentity), + "launch_transport_identity_changed"); + opened.vm_identity = vmIdentity; + match(frame); } }; const response = res => { void observeResponse(res).catch(() => controller.abort()); }; @@ -396,16 +403,19 @@ export async function createQualificationHarness(context, page, options, now = ( cancellation?.setRuntimeCleanup(() => cleanupQualificationOpens(attempts)); const openSnapshot = pageId => ({ schema: "elastos.browser.qualification-observation/v1", mode: options.mode, launch: launches.get(pageId) || null, observation: null, + vm_identity: attempts.find(a => a.page_id === pageId)?.vm_identity || null, open_attempts: attempts.map(a => ({ open_id: a.open_id, page_id: a.page_id, outcome: a.outcome, started_ms: a.start - attempts[0].start })) }); return { snapshot: openSnapshot, async observe({ appFrame, pageId, readReceipt, readStatus, interact }) { + requireEvidence(!controller.signal.aborted, "qualification_cancelled"); const evidence = openSnapshot(pageId); if (attempts.length !== 1 || attempts[0].outcome !== "completed") { throw Object.assign(new Error("qualification_open_retry_or_failure"), { qualification: evidence }); } if (["cold", "warm"].includes(options.mode)) requireEvidence(evidence.launch, "qualification_launch_not_observed"); + qualificationVmIdentity(evidence.vm_identity, pageId); evidence.services = await observationCall(() => appFrame.evaluate(() => ({ engine_id: document.querySelector("#browser-engine")?.value, exit_id: document.querySelector("#browser-exit")?.value, diff --git a/scripts/lib/browser-qualification.test.mjs b/scripts/lib/browser-qualification.test.mjs index 3a924d774..9b36189de 100644 --- a/scripts/lib/browser-qualification.test.mjs +++ b/scripts/lib/browser-qualification.test.mjs @@ -3,16 +3,29 @@ import test from "node:test"; import vm from "node:vm"; import { PassThrough } from "node:stream"; import { EventEmitter } from "node:events"; +import { createServer } from "node:http"; import { mkdtemp, writeFile, readFile, rm, rename, utimes, realpath } from "node:fs/promises"; import { writeFileSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { validatePlan, validateJourney, validateAttempt, auditQualification, sha256, - TERMINAL_EFFECTS, percentile, cleanControl, candidateFingerprint, validateInstallation, validateLiveIdentity, deriveMedia } from "../browser-qualification-audit.mjs"; -import { freezeArtifacts, checkArtifacts, journeyEnvironment, parseProcesses, ownedProcesses, runJourneyChild, executableInode } from "../browser-qualification-runner.mjs"; + TERMINAL_EFFECTS, percentile, cleanControl, candidateFingerprint, validateInstallation, validateLiveIdentity, deriveMedia, qualificationTarget, qualificationVmIdentity, validateLocalLaunchSample } from "../browser-qualification-audit.mjs"; +import { freezeArtifacts, checkArtifacts, journeyEnvironment, parseProcesses, ownedProcesses, runJourneyChild, executableInode, qualificationFixtureHealth } from "../browser-qualification-runner.mjs"; import { qualificationOptions, installQualificationReceiver, observationCall, createQualificationHarness, createQualificationCancellation, qualificationInteraction, cleanupQualificationOpens } from "./browser-qualification-observer.mjs"; +import { readBrowserJourneyReceipt } from "./browser-journey-target.mjs"; import { createBrowserJourneyFixture } from "./browser-journey-fixture.mjs"; +const PAGE_ID = "page:vz-" + "d".repeat(64); +function vmProof() { + return { schema: "elastos.browser.vz-transport-public-proof/v1", page_id: PAGE_ID, + vm_id: "browser-vm-" + "e".repeat(64), generation: "sha256:" + "f".repeat(64), binding_hash: "sha256:" + "a".repeat(64) }; +} +function launchSample() { + const c = control(); c.active_pages = 1; c.page_ids = [PAGE_ID]; + c.lifecycle.sessions = [{ phase: "ACTIVE_SESSION", warm_vm: false, vm_key_hash: null, + page_id: "sha256:" + sha256(PAGE_ID).slice(0, 16) }]; + return { identity_verified: true, rss_bytes: 128, cpu_percent: 10, process_ids: ["10:birth-a", "20:birth-b"], control: c }; +} function plan(mode = "lifecycle") { const p = { schema: "elastos.browser.qualification-plan/v1", mode, count: 100, duration_ms: 1_800_000, require_operator: true, candidate: { platform: "darwin-arm64", source_commit: "a".repeat(40), @@ -22,7 +35,7 @@ function plan(mode = "lifecycle") { runtime: { base_url: "http://localhost:61510", browser_ui_url: "http://localhost:61510/apps/browser/browser.js", fixture_origin: "http://localhost:61511", profile: "/fixture/profile", control_socket: "/fixture/control.sock", operator_coords: "/fixture/gateway-runtime-coords.json", viewer_version: "fixture-chromium", headed: true, engine_id: "local", exit_id: "runtime-default", - resource_roots: [{ role: "runtime", pid: 10, start: "birth-a" }, { pid: 20, start: "birth-b" }] }, + resource_roots: [{ role: "runtime", pid: 10, start: "birth-a" }, { role: "vm_control", pid: 20, start: "birth-b" }] }, network: { profile: "local", control_rtt_ms: 1 }, limits: { rss_bytes: 1024 * 1024, cpu_percent: 400, fps: 30, width: 1920, height: 1080 } }; p.candidate.artifacts.find(a => a.role === "installation_review").sha256 = sha256(JSON.stringify(installation(p))); @@ -38,7 +51,7 @@ function installation(p) { function frozenInputs(p) { return [...p.candidate.artifacts, { role: "operator_coordinates", path: p.runtime.operator_coords, sha256: "e".repeat(64) }, ...["scripts/browser-qualification-runner.mjs", "scripts/browser-qualification-audit.mjs", - "scripts/lib/browser-qualification-observer.mjs", "scripts/home-passkey-virtual-auth-smoke.mjs"].map(role => + "scripts/lib/browser-qualification-observer.mjs", "scripts/lib/browser-journey-target.mjs", "scripts/home-passkey-virtual-auth-smoke.mjs"].map(role => ({ role, path: "/fixture/" + role, sha256: "e".repeat(64) }))].map(a => ({ ...a, identity: "1:2:3:4:5" })); } function liveIdentity(p) { @@ -61,7 +74,7 @@ const tone = () => ({ ok: true, receiver_unchanged: true, receiver_muted: false, track_state: "live", context_state: "running", probe_context_closed: true, samples: Array.from({ length: 40 }, (_, i) => ({ at_ms: 500 + i * 50, rms: 0.03, peak_hz: 445.3 })) }); function journey(id = "fixture-run-0001") { - const pageId = "page:vz-fixture"; + const pageId = PAGE_ID; return { page_id: pageId, display_mode: "webrtc_remote_display", controlled_journey: { schema: "elastos.browser.controlled-journey/v1", run: id, pages: ["main", "nav"].map(name => ({ name, page_id: pageId, load: { type: "load" }, @@ -71,25 +84,27 @@ function journey(id = "fixture-run-0001") { inspection: { pages: [{}, {}], after_navigation: { status: 409, body: { code: "stale_inspection" } } }, viewer_reload: { ok: true }, operator: { ok: true }, qualification: { schema: "elastos.browser.qualification-observation/v1", - open_attempts: [{ outcome: "completed" }], + open_attempts: [{ outcome: "completed" }], vm_identity: vmProof(), services: { engine_id: "local", exit_id: "runtime-default" }, launch: { page_id: pageId, clock: "host_monotonic", usable_frame_ms: 1000 } }, close: { window_detached: true, receipt: { schema: "elastos.browser.close-result/v1", closed: true, page_id: pageId, cleanup_id: "cleanup-fixture", cleanup: { ok: true, action: "released_exact_runtime_browser_ownership" }, - transport_proof: { page_id: pageId }, terminal_effects: Object.fromEntries(TERMINAL_EFFECTS.map(k => [k, true])) }, + transport_proof: vmProof(), terminal_effects: Object.fromEntries(TERMINAL_EFFECTS.map(k => [k, true])) }, sessions_after_close: { active_sessions: 0, principal_sessions: 0, total_sessions: 0, launching_sessions: 0, engine_cleanup_obligations: 0, launch_reconciliation_obligations: 0, recoverable_page: null } } } }; } function control() { - return { schema: "elastos.browser.vm-control-service.status/v1", ok: true, direct_network: false, + return { schema: "elastos.browser.vm-control-service.status/v1", ok: true, pid: 20, + config_fingerprint: "c".repeat(64), control_service: { schema: "elastos.browser.vm-control-service.identity/v1", + service_id: "service:" + "b".repeat(64) }, direct_network: false, network_mode: "runtime_net_only", active_pages: 0, active_vms: 0, warm_vms: 0, pending_launches: 0, - page_ids: [], active_stream_ids: [], pending_stream_ids: [], lifecycle: { sessions: [] } }; + page_ids: [], active_stream_ids: [], pending_stream_ids: [], lifecycle: { schema: "elastos.browser.lifecycle-status/v1", sessions: [] } }; } function attempt(index = 1, mode = "lifecycle") { return { schema: "elastos.browser.qualification-attempt/v1", index, mode, ok: true, child_exit: 0, candidate_fingerprint: "f".repeat(64), candidate_unchanged: true, before: control(), after: control(), - observed_page_ids: ["page:vz-fixture"], active_vm_keys: ["vm-fixture"], + observed_page_ids: [PAGE_ID], local_launch_sample: launchSample(), viewer: { version: "fixture-chromium", headed: true }, resource_peak: { identity_verified: true, rss_bytes: 128, cpu_percent: 10 }, journey: journey("fixture-run-" + index) }; @@ -123,6 +138,86 @@ test("private installed journey flags select the embedded flow and remove confli else process.env.HOME_VIRTUAL_AUTH_BROWSER_OPEN = old; } }); +test("qualification exports exact canonical services and fixture options while default plans stay local", () => { + const keys = ["HOME_VIRTUAL_AUTH_BROWSER_ENGINE_ID", "HOME_VIRTUAL_AUTH_BROWSER_REMOTE_EXIT_ID", + "HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ADMIN_ORIGIN", "HOME_VIRTUAL_AUTH_BROWSER_ALLOW_REMOTE_FIXTURE"]; + const prior = keys.map(k => process.env[k]); + keys.forEach(k => { process.env[k] = "unrelated"; }); + try { + const p = plan(); p.runtime.engine_id = p.runtime.exit_id = ""; + const defaults = journeyEnvironment(p, 1); + assert.deepEqual(keys.map(k => defaults[k]), ["", "", p.runtime.fixture_origin, "0"]); + p.runtime.engine_id = "remote-engine-approved"; p.runtime.exit_id = "remote-carrier-approved"; + p.runtime.fixture_origin = "https://fixture.example"; + p.runtime.fixture_admin_origin = "http://127.0.0.1:61511"; p.runtime.allow_remote_fixture = true; + validatePlan(p); + const env = journeyEnvironment(p, 1); + assert.deepEqual(keys.map(k => env[k]), [p.runtime.engine_id, p.runtime.exit_id, p.runtime.fixture_admin_origin, "1"]); + assert.equal(env.HOME_VIRTUAL_AUTH_BROWSER_FIXTURE_ORIGIN, p.runtime.fixture_origin); + p.runtime.remote_exit_id = p.runtime.exit_id; validatePlan(p); + p.runtime.remote_exit_id = "other"; + assert.throws(() => validatePlan(p), /conflicting_legacy_exit/); + assert.throws(() => journeyEnvironment(p, 1), /conflicting_legacy_exit/); + } finally { keys.forEach((k, i) => { if (prior[i] === undefined) delete process.env[k]; else process.env[k] = prior[i]; }); } + for (const change of [r => { r.engine_id = "choice with spaces"; }, r => { r.exit_id = " trimmed "; }, + r => { r.engine_id = "x".repeat(129); }, r => { r.fixture_origin = "https://fixture.example"; }, + r => { r.fixture_admin_origin = "https://fixture.example"; }, r => { r.allow_remote_fixture = "true"; }, + r => { r.fixture_admin_origin = null; }]) { + const p = plan(); change(p.runtime); assert.throws(() => validatePlan(p)); + } + for (const key of ["fixture_origin", "fixture_admin_origin"]) for (const origin of [ + "https://user:password@fixture.example", "https://fixture.example/path", "https://fixture.example/?query=1", + "https://fixture.example/#fragment", "https://fixture.example/a/..", " https://fixture.example"]) { + const p = plan(); p.runtime.allow_remote_fixture = true; p.runtime[key] = origin; + assert.throws(() => validatePlan(p), /exact HTTP/); + } +}); +test("qualification preflight uses the admin server and keeps health, redirect and run binding checks", async () => { + const fixture = createBrowserJourneyFixture(); + await new Promise(r => fixture.listen(0, "127.0.0.1", r)); + const admin = "http://127.0.0.1:" + fixture.address().port; + const redirect = createServer((_req, res) => { res.writeHead(302, { Location: admin + "/health" }); res.end(); }); + await new Promise(r => redirect.listen(0, "127.0.0.1", r)); + try { + const p = plan(); Object.assign(p.runtime, { fixture_origin: "https://product.example", + fixture_admin_origin: admin, allow_remote_fixture: true }); + assert.equal((await qualificationFixtureHealth(p)).qualification, "bounded-v1"); + await fetch(admin + "/main?run=qualification-route-test&qualification=1"); + const receipt = await readBrowserJourneyReceipt(qualificationTarget(p.runtime), "qualification-route-test"); + assert.equal(receipt.run, "qualification-route-test"); assert.equal(receipt.observation, "bounded-v1"); + for (const [ok, body] of [[false, { schema: "elastos.browser.journey-fixture/v1", ok: true, qualification: "bounded-v1" }], + [true, { schema: "wrong", ok: true, qualification: "bounded-v1" }], + [true, { schema: "elastos.browser.journey-fixture/v1", ok: false, qualification: "bounded-v1" }], + [true, { schema: "elastos.browser.journey-fixture/v1", ok: true }]]) { + await assert.rejects(qualificationFixtureHealth(p, async (url, options) => { + assert.equal(url, admin + "/health"); assert.equal(options.credentials, "omit"); + assert.equal(options.redirect, "error"); assert.ok(options.signal); + return { ok, json: async () => body }; + })); + } + await assert.rejects(readBrowserJourneyReceipt(qualificationTarget(p.runtime), "qualification-route-test", { + fetchImpl: async () => ({ ok: true, json: async () => ({ ...receipt, run: "another-run" }) }), + }), /receipt failed/); + p.runtime.fixture_admin_origin = "http://127.0.0.1:" + redirect.address().port; + await assert.rejects(qualificationFixtureHealth(p), /fetch failed/); + } finally { + fixture.closeAllConnections(); redirect.closeAllConnections(); + await Promise.all([new Promise(r => fixture.close(r)), new Promise(r => redirect.close(r))]); + } +}); +test("the canonical target helper is required in frozen inputs and its replacement invalidates the campaign", async () => { + const p = plan(), inputs = frozenInputs(p), role = "scripts/lib/browser-journey-target.mjs"; + assert.throws(() => candidateFingerprint(p, inputs.filter(f => f.role !== role)), /frozen_harness_inputs/); + const original = candidateFingerprint(p, inputs); inputs.find(f => f.role === role).sha256 = "a".repeat(64); + assert.notEqual(candidateFingerprint(p, inputs), original); + const dir = await realpath(await mkdtemp(join(tmpdir(), "qualification-target-freeze-"))); + try { + const path = join(dir, "target.mjs"); await writeFile(path, "old"); + const frozen = await freezeArtifacts([{ role, path, sha256: sha256("old") }]); + await writeFile(join(dir, "replacement"), "old"); await rename(join(dir, "replacement"), path); + await assert.rejects(checkArtifacts(frozen), /candidate_changed/); + } finally { await rm(dir, { recursive: true, force: true }); } +}); test("current close receipt omits already_closed; every terminal effect remains mandatory", () => { validateJourney(journey()); for (const key of TERMINAL_EFFECTS) { @@ -140,7 +235,7 @@ test("current close receipt omits already_closed; every terminal effect remains test("attempts bind exact viewer, services, observed local VM and monotonic launch", () => { validateAttempt(attempt(), plan()); for (const change of [a => { a.candidate_unchanged = false; }, a => { a.viewer.version = "other"; }, - a => { a.observed_page_ids = []; }, a => { a.active_vm_keys = []; }, + a => { a.observed_page_ids = []; }, a => { a.local_launch_sample = null; }, a => { a.journey.controlled_journey.qualification.services.exit_id = "other"; }, a => { a.resource_peak.rss_bytes = 9e9; }]) { const a = attempt(); change(a); assert.throws(() => validateAttempt(a, plan())); @@ -149,6 +244,32 @@ test("attempts bind exact viewer, services, observed local VM and monotonic laun cold.journey.controlled_journey.qualification.launch.clock = "wall_clock"; assert.throws(() => validateAttempt(cold, plan("cold")), /monotonic/); }); +test("transport-authority launches require matching real VM proof and sampled ownership even with no reuse key", () => { + const a = attempt(); + assert.equal(a.local_launch_sample.control.active_vms, 0); + assert.equal(a.local_launch_sample.control.lifecycle.sessions[0].vm_key_hash, null); + validateAttempt(a, plan()); + for (const key of ["vm_id", "generation", "binding_hash", "page_id"]) { + const changed = structuredClone(a); + changed.journey.controlled_journey.close.receipt.transport_proof[key] = + key === "vm_id" ? "browser-vm-" + "1".repeat(64) : key === "page_id" ? "page:vz-" + "1".repeat(64) : "sha256:" + "1".repeat(64); + assert.throws(() => validateAttempt(changed, plan()), key === "page_id" ? /thirteen_effect/ : /identity/); + } + for (const change of [a => { a.journey.controlled_journey.qualification.vm_identity = null; }, + a => { a.journey.controlled_journey.qualification.vm_identity.vm_id = "invented-reuse-key"; }, + a => { a.local_launch_sample = null; a.active_vm_keys = ["legacy-key"]; }, + a => { a.local_launch_sample.process_ids = ["10:birth-a", "20:replacement-birth"]; }, + a => { a.local_launch_sample.control.pid = 21; }, + a => { a.local_launch_sample.control.config_fingerprint = "9".repeat(64); }, + a => { a.local_launch_sample.control.control_service.service_id = "service:" + "9".repeat(64); }, + a => { a.local_launch_sample.control.page_ids = ["another-page"]; }, + a => { a.local_launch_sample.control.lifecycle.sessions[0].page_id = "sha256:another-page"; }, + a => { a.local_launch_sample.control.lifecycle.sessions[0].phase = "STARTING_VM"; }, + a => { a.local_launch_sample.control.lifecycle.sessions[0].warm_vm = true; }, + a => { a.after.control_service.service_id = "service:" + "9".repeat(64); }]) { + const changed = structuredClone(a); change(changed); assert.throws(() => validateAttempt(changed, plan())); + } +}); test("warm conditioning stays unsupported, including an idle warm VM beside a cold allocation", () => { const a = attempt(1, "warm"); a.before.active_vms = a.before.warm_vms = 1; @@ -215,20 +336,51 @@ test("launch clock excludes Home auth and binds first frame before completed ope const open = req("/api/apps/browser/open"); page.emit("request", open); await respond(open, { open_id: "open-current" }); clock = 37_250; - callback({ frame }, { page_id: "page:vz-fixture", width: 1920, height: 1080 }); - const result = { schema: "elastos.browser.open-result/v1", engine_page: { page_id: "page:vz-fixture" } }; + callback({ frame }, { page_id: PAGE_ID, width: 1920, height: 1080 }); + const result = { schema: "elastos.browser.open-result/v1", engine_page: { page_id: PAGE_ID, transport_proof: vmProof() } }; await respond(req("/api/apps/browser/open/open-old", "GET"), { schema: "elastos.browser.open-status/v1", status: "completed", result }); - await assert.rejects(harness.observe({ appFrame: frame, pageId: "page:vz-fixture" }), /open_retry_or_failure/); + await assert.rejects(harness.observe({ appFrame: frame, pageId: PAGE_ID }), /open_retry_or_failure/); clock = 38_000; await respond(req("/api/apps/browser/open/open-current", "GET"), { schema: "elastos.browser.open-status/v1", status: "completed", result }); - const evidence = await harness.observe({ appFrame: frame, pageId: "page:vz-fixture" }); + const evidence = await harness.observe({ appFrame: frame, pageId: PAGE_ID }); assert.equal(evidence.launch.usable_frame_ms, 250); assert.equal(evidence.launch.clock, "host_monotonic"); + assert.deepEqual(evidence.vm_identity, vmProof()); } finally { await harness.stop(); } assert.equal(page.listenerCount("request"), 0); assert.equal(page.listenerCount("response"), 0); }); +test("launch observer captures only the owning frame's exact transport proof and rejects missing or changed identity", async () => { + for (const variant of ["valid", "missing", "wrong-page", "changed"]) { + const page = new EventEmitter(), frame = { evaluate: async () => ({ engine_id: "local", exit_id: "runtime-default" }) }; + const context = { exposeBinding: async () => {}, addInitScript: async () => {} }; + const h = await createQualificationHarness(context, page, { mode: "lifecycle", duration_ms: 0 }); + const req = { method: () => "POST", url: () => "http://localhost/api/apps/browser/open", frame: () => frame }; + const respond = async (request, body) => { + page.emit("response", { request: () => request, url: request.url, json: async () => body }); + await new Promise(setImmediate); + }; + try { + page.emit("request", req); await respond(req, { open_id: "exact-open" }); + const poll = { method: () => "GET", url: () => "http://localhost/api/apps/browser/open/exact-open", frame: () => frame }; + const proof = vmProof(); if (variant === "wrong-page") proof.page_id = "page:vz-" + "1".repeat(64); + const body = { schema: "elastos.browser.open-status/v1", status: "completed", result: { + schema: "elastos.browser.open-result/v1", engine_page: { page_id: PAGE_ID, + ...(variant === "missing" ? {} : { transport_proof: proof }) } } }; + await respond({ ...poll, frame: () => ({}) }, body); + await respond({ ...poll, url: () => "http://other-origin/api/apps/browser/open/exact-open" }, body); + assert.equal(h.snapshot(PAGE_ID).vm_identity, null); + await respond(poll, body); + if (variant === "changed") { + body.result.engine_page.transport_proof.generation = "sha256:" + "1".repeat(64); + await respond(poll, body); + } + if (variant === "valid") assert.deepEqual((await h.observe({ appFrame: frame, pageId: PAGE_ID })).vm_identity, vmProof()); + else await assert.rejects(h.observe({ appFrame: frame, pageId: PAGE_ID }), /identity|cancelled/); + } finally { await h.stop(); } + } +}); test("qualification fixture retains an active eight-hour run with a 128-event ring and monotonic cursor", async () => { let clock = 1_000_000; const server = createBrowserJourneyFixture({ now: () => clock }); @@ -266,7 +418,7 @@ test("qualification audit binds all 100 unique attempts, raw logs and samples; p const bytes = JSON.stringify(a); writeFileSync(join(dir, file), bytes); refs.push({ file, sha256: sha256(bytes) }); resources.push(JSON.stringify({ sequence: i, at_ms: i * 60_000, attempt: i, kind: "resources", - value: { identity_verified: true, rss_bytes: 128, cpu_percent: 10 } })); + value: launchSample() })); } const sampleBytes = resources.join("\n") + "\n"; writeFileSync(join(dir, "samples.jsonl"), sampleBytes); const planBytes = JSON.stringify(p); writeFileSync(join(dir, "plan.json"), planBytes); @@ -278,6 +430,11 @@ test("qualification audit binds all 100 unique attempts, raw logs and samples; p sealBundle(dir, receipt); save(receipt); const accepted = auditQualification(path); assert.equal(accepted.ok, true); assert.equal(accepted.product_accepted, false); + const firstPath = join(dir, refs[0].file), original = readFileSync(firstPath); + const first = JSON.parse(original); first.local_launch_sample.rss_bytes++; + const changed = JSON.stringify(first); writeFileSync(firstPath, changed); refs[0].sha256 = sha256(changed); save(receipt); + assert.throws(() => auditQualification(path), /launch_sample_not_in_raw_evidence/); + writeFileSync(firstPath, original); refs[0].sha256 = sha256(original); save(receipt); save({ ...receipt, frozen_inputs: undefined }); assert.throws(() => auditQualification(path), /frozen_inputs/); const relabeled = structuredClone(receipt); @@ -714,7 +871,7 @@ test("raw thirty-minute media and interaction trace drives its summary and retai const save = rows => { const attemptBytes = JSON.stringify(a); writeFileSync(join(dir, "attempt-1.json"), attemptBytes); const samples = [JSON.stringify({ sequence: 1, at_ms: 0, attempt: 1, kind: "resources", - value: { identity_verified: true, rss_bytes: 128, cpu_percent: 10 } }), ...rows.map(r => JSON.stringify(r))].join("\n") + "\n"; + value: launchSample() }), ...rows.map(r => JSON.stringify(r))].join("\n") + "\n"; writeFileSync(join(dir, "samples.jsonl"), samples); const r = { schema: "elastos.browser.qualification/v1", plan: p, plan_sha256: sha256(planBytes), completed: true, candidate_unchanged: true, cancelled: false, From 946b994a650d8b92a0e79a76a9ca69196cbd374f Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 8 Sep 2026 18:19:45 -0500 Subject: [PATCH 212/535] fix(services): name Engine authority in Inbox approval requests --- .../src/api/gateway_home_system.rs | 22 +++++-- .../src/api/gateway_tests/home_system.rs | 60 +++++++++++++++++++ 2 files changed, 76 insertions(+), 6 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index a6bc8b0a5..9d0b6a318 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -3716,6 +3716,8 @@ pub(super) fn append_home_service_access_notifications( if existing_ids.contains(&id) { continue; } + let engine_request = request.service_kind == crate::carrier::ENGINE_SERVICE_KIND + && request.service_uri == crate::carrier::ENGINE_SERVICE_URI; notifications.unread_count += 1; notifications.attention_count += 1; notifications.entries.push(HomeNotificationEntrySummary { @@ -3723,13 +3725,21 @@ pub(super) fn append_home_service_access_notifications( source_app: SERVICES_CAPSULE_ID.to_string(), kind: "service_access_request".to_string(), title: format!( - "{} requests your Browser Exit Node", - request.requester_display_name - ), - body: format!( - "{} wants to use {}. Approval records your intent; Browser access still requires an installed remote Exit grant.", - request.requester_display_name, request.service_display_name + "{} requests your {}", + request.requester_display_name, + if engine_request { "Browser Engine" } else { "Browser Exit Node" } ), + body: if engine_request { + format!( + "{} wants to use {}. Approval allows the requested Browser Engine operations through your Runtime.", + request.requester_display_name, request.service_display_name + ) + } else { + format!( + "{} wants to use {}. Approval records your intent; Browser access still requires an installed remote Exit grant.", + request.requester_display_name, request.service_display_name + ) + }, action_ref: Some(HomeNotificationActionSummary { app: SERVICES_CAPSULE_ID.to_string(), action_id: format!("service-approve-request:{}", request.request_id), diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs b/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs index 0e77b6618..5e7b91241 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs @@ -8455,6 +8455,66 @@ fn services_engine_offer(fixture: &ServicesContactFixture) -> String { ) } +#[tokio::test] +async fn test_services_inbox_request_copy_matches_engine_or_exit_and_exact_approval() { + for (engine, service_name, offer_name, approval_text) in [ + ( + true, + "Browser Engine", + "Bob's Browser Engine", + "Approval allows the requested Browser Engine operations through your Runtime.", + ), + ( + false, + "Browser Exit Node", + "Bob's Browser Exit", + "Approval records your intent; Browser access still requires an installed remote Exit grant.", + ), + ] { + let left = tempfile::tempdir().unwrap(); + let right = tempfile::tempdir().unwrap(); + let bus = Arc::new(TokioMutex::new(FakePeerBus::default())); + let (trusted_key, _) = generate_keypair(); + let network = configured_discovery_network_profile_for_test(&trusted_key, "services-contacts"); + let alice = services_contact_fixture(left.path(), "Alice", bus.clone(), network.clone()).await; + let bob = services_contact_fixture(right.path(), "Bob", bus, network).await; + accept_services_contact_pair(&alice, &bob); + let action = if engine { + services_contact_pending_engine_request(left.path(), right.path(), &alice, &bob).await + } else { + services_contact_pending_request(left.path(), right.path(), &alice, &bob).await + }; + let token = app_token_for_authority(right.path(), INBOX_CAPSULE_ID, &bob.authority); + let (status, inbox) = + home_test_get_json(&bob.app, "/api/apps/inbox/summary", &token, "null").await; + assert_eq!(status, StatusCode::OK); + let entries = inbox["notifications"]["entries"] + .as_array() + .unwrap() + .iter() + .filter(|entry| entry["kind"] == "service_access_request") + .collect::>(); + assert_eq!(entries.len(), 1, "{inbox}"); + let entry = entries[0]; + assert_eq!(entry["title"], format!("Alice requests your {service_name}")); + assert_eq!( + entry["body"], + format!("Alice wants to use {offer_name}. {approval_text}") + ); + assert_eq!(entry["source_app"], SERVICES_CAPSULE_ID); + assert_eq!(entry["action_ref"]["app"], SERVICES_CAPSULE_ID); + assert_eq!(entry["action_ref"]["action_id"], action); + assert_eq!( + entry["id"], + format!( + "service-access-request:{}", + action.strip_prefix("service-approve-request:").unwrap() + ) + ); + assert_eq!(entry["read"], false); + } +} + async fn services_contact_pending_engine_request( left: &std::path::Path, right: &std::path::Path, From 690170bcefc07e0821951d5a24bcb4b7a705f015 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 8 Sep 2026 18:19:46 -0500 Subject: [PATCH 213/535] fix(browser): identify remote execution and owner retirement stages --- .../src/api/gateway_browser_remote.rs | 19 ++++-- .../elastos-server/src/carrier_engine.rs | 60 +++++++++++++++++-- 2 files changed, 68 insertions(+), 11 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs b/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs index 65c6519db..cb6b31dc9 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs @@ -918,20 +918,29 @@ async fn supervise(page: Arc) { return; } let stale = page.last_seen.lock().await.elapsed() >= Duration::from_secs(50); - let current = if stale { - false + let retirement_reason = if stale { + Some("owner_heartbeat_expired") } else { let root = page.gateway.data_dir.clone(); let network = page.network.clone(); let owner = page.owner.clone(); - tokio::time::timeout(Duration::from_secs(1), tokio::task::spawn_blocking(move || { + let observation = tokio::time::timeout(Duration::from_secs(1), tokio::task::spawn_blocking(move || { let source = owner.requester_endpoint.parse::().ok()?; let current = crate::api::gateway::authorize_home_service_engine(&root, &network, &source, &json!({"op":"page_status","principal_id":owner.requester_principal_id,"grant_id":owner.grant_id}), now_ts()).ok()?; (current.revision == owner.grant_revision && current.execution_allowed).then_some(()) - })).await.is_ok_and(|result| result.is_ok_and(|result| result.is_some())) + })).await; + match observation { + Ok(Ok(Some(()))) => None, + Ok(Ok(None)) => Some("authority_rejected_or_changed"), + Ok(Err(_)) => Some("authority_worker_failed"), + Err(_) => Some("authority_observation_deadline"), + } }; - if !current { + if let Some(reason) = retirement_reason { + if !page.closing.load(Ordering::Acquire) { + tracing::warn!(reason, "Remote Engine owner retired"); + } page.closing.store(true, Ordering::Release); } if page.closing.load(Ordering::Acquire) { diff --git a/elastos/crates/elastos-server/src/carrier_engine.rs b/elastos/crates/elastos-server/src/carrier_engine.rs index c00f661c6..96a4666b8 100644 --- a/elastos/crates/elastos-server/src/carrier_engine.rs +++ b/elastos/crates/elastos-server/src/carrier_engine.rs @@ -159,6 +159,7 @@ pub(super) async fn invoke( ) -> Value { let operation = data["operation"].as_str().unwrap_or_default(); if !matches!(operation, "status" | "readiness") || data["request"].get("page_id").is_some() { + let mut stage = "execution_envelope"; let outcome = async { anyhow::ensure!( serde_json::to_vec(data)?.len() <= 256 * 1024 @@ -168,6 +169,7 @@ pub(super) async fn invoke( && super::browser_engine_binding::EXECUTION_OPERATIONS.contains(&operation), "Engine execution envelope invalid" ); + stage = "invocation_contract"; super::validate_carrier_provider_invocation( "browser", "browser-engine", @@ -177,6 +179,7 @@ pub(super) async fn invoke( ) .map_err(anyhow::Error::msg)?; if operation == "close_page" && data["request"]["cancel_preparation"] == true { + stage = "cancellation_authority"; let root = data_dir.to_owned(); let request = data["request"].clone(); tokio::time::timeout( @@ -190,6 +193,7 @@ pub(super) async fn invoke( .await .context("Engine cancellation authority deadline")???; } + stage = "grant_authority"; let grant = if matches!(operation, "close_page" | "status") { None } else { @@ -205,6 +209,7 @@ pub(super) async fn invoke( .await?, ) }; + stage = "runtime_operation"; let response = crate::api::gateway::invoke_remote_browser_engine( data_dir, registry.clone(), @@ -218,6 +223,7 @@ pub(super) async fn invoke( ) .await?; if let Some(grant) = grant { + stage = "post_operation_authority"; anyhow::ensure!( read_authority( data_dir, @@ -232,7 +238,7 @@ pub(super) async fn invoke( } Ok::<_, anyhow::Error>(response) }; - return match tokio::time::timeout( + let outcome = tokio::time::timeout( Duration::from_secs(if matches!(operation, "prepare_launch" | "launch") { 55 } else { @@ -240,12 +246,11 @@ pub(super) async fn invoke( }), outcome, ) - .await - { + .await; + return match outcome { Ok(Ok(result)) => json!({"ok":true,"result":result}), - _ => { - json!({"ok":false,"code":"browser_engine_unavailable","error":"Runtime Engine operation is unavailable or pending settlement"}) - } + Ok(Err(_)) => execution_failure(operation, stage, "rejected"), + Err(_) => execution_failure(operation, stage, "deadline"), }; } let outcome = async { @@ -325,6 +330,34 @@ pub(super) async fn invoke( } } +fn execution_failure(operation: &str, stage: &str, outcome: &str) -> Value { + // Fixed contract labels identify the boundary without logging provider + // errors, request fields, tickets or native host paths. + let operation = super::browser_engine_binding::EXECUTION_OPERATIONS + .iter() + .copied() + .find(|supported| *supported == operation) + .unwrap_or("unsupported"); + let stage = match stage { + "execution_envelope" + | "invocation_contract" + | "cancellation_authority" + | "grant_authority" + | "runtime_operation" + | "post_operation_authority" => stage, + _ => "unknown", + }; + let outcome = if outcome == "deadline" { + "deadline" + } else { + "rejected" + }; + tracing::warn!(operation, stage, outcome, "Remote Engine execution failed"); + json!({"ok":false,"code":"browser_engine_unavailable", + "error":"Runtime Engine operation is unavailable or pending settlement", + "operation":operation,"stage":stage,"outcome":outcome}) +} + pub(crate) async fn probe( registry: &ProviderRegistry, grant: &Value, @@ -406,6 +439,21 @@ pub(crate) async fn call( mod tests { use super::*; + #[test] + fn engine_execution_failure_keeps_only_fixed_diagnostic_labels() { + let value = execution_failure("input", "runtime_operation", "deadline"); + assert_eq!(value["operation"], "input"); + assert_eq!(value["stage"], "runtime_operation"); + assert_eq!(value["outcome"], "deadline"); + assert_eq!(value["ok"], false); + let value = execution_failure("ticket:private", "/private/host", "secret"); + assert_eq!(value["operation"], "unsupported"); + assert_eq!(value["stage"], "unknown"); + assert_eq!(value["outcome"], "rejected"); + assert!(!value.to_string().contains("private")); + assert!(!value.to_string().contains("secret")); + } + struct EngineCallInvoker { fail: bool, calls: std::sync::Mutex>, From b6b5bd23562068b0d285f2cd44e997b1aacdcbb8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 8 Sep 2026 18:48:35 -0500 Subject: [PATCH 214/535] test(browser): observe committed profile storage across sessions --- scripts/lib/browser-journey-fixture.mjs | 127 +++++++++++- scripts/lib/browser-journey-profile.test.mjs | 207 +++++++++++++++++++ 2 files changed, 327 insertions(+), 7 deletions(-) create mode 100644 scripts/lib/browser-journey-profile.test.mjs diff --git a/scripts/lib/browser-journey-fixture.mjs b/scripts/lib/browser-journey-fixture.mjs index 979d76908..ab3b3cf10 100644 --- a/scripts/lib/browser-journey-fixture.mjs +++ b/scripts/lib/browser-journey-fixture.mjs @@ -9,7 +9,80 @@ const MAX_EVENTS = 128; const TTL_MS = 10 * 60_000; const RUN_ID = /^[a-zA-Z0-9_-]{8,64}$/; -function fixturePage(page, run, media = false, qualification = false) { +// Serialized into the opt-in page. Only this fixture's fixed storage keys are touched. +export async function browserProfileStorageProbe({ mode, marker }) { + if (!["write", "read"].includes(mode) || !/^[a-zA-Z0-9_-]{8,64}$/.test(marker)) throw new Error("invalid profile probe"); + const evidence = { schema: "elastos.browser.profile-storage/v1", mode, marker, measurement_ok: false, ok: false, + cookie: { present: false, matches: false }, local_storage: { present: false, matches: false }, + indexed_db: { write_request_succeeded: false, write_committed: false, + read_request_succeeded: false, read_completed: false, present: false, matches: false }, error: null }; + const key = "elastos_browser_profile_probe", database = "elastos-browser-profile-probe-v1"; + const observed = value => ({ present: value !== null && value !== undefined, matches: value === marker }); + let db; + try { + if (mode === "write") { + document.cookie = key + "=" + marker + "; Path=/; SameSite=Strict; Max-Age=86400"; + localStorage.setItem(key, marker); + } + const cookie = document.cookie.split(";").map(s => s.trim()).find(s => s.startsWith(key + "=")); + evidence.cookie = observed(cookie ? cookie.slice(key.length + 1) : null); + evidence.local_storage = observed(localStorage.getItem(key)); + db = await new Promise((resolve, reject) => { + const request = indexedDB.open(database, 1); + let expired = false; + const timer = setTimeout(() => { expired = true; reject(new Error("storage_timeout")); }, 5000); + request.onupgradeneeded = () => { + // Opening an absent database in read mode must not create probe state. + if (expired || mode === "read") { request.transaction.abort(); return; } + request.result.createObjectStore("markers"); + }; + request.onsuccess = () => { + clearTimeout(timer); + if (expired) { request.result.close(); return; } + resolve(request.result); + }; + request.onerror = () => { clearTimeout(timer); reject(new Error("database_open_failed")); }; + }); + const transaction = write => new Promise((resolve, reject) => { + const tx = db.transaction("markers", write ? "readwrite" : "readonly"); + const store = tx.objectStore("markers"); + const request = write ? store.put(marker, "value") : store.get("value"); + let expired = false; + const timer = setTimeout(() => { + expired = true; + try { tx.abort(); } catch {} + reject(new Error("storage_timeout")); + }, 5000); + request.onsuccess = () => { + if (expired) return; + if (write) evidence.indexed_db.write_request_succeeded = true; + else { + evidence.indexed_db.read_request_succeeded = true; + Object.assign(evidence.indexed_db, observed(request.result)); + } + }; + tx.oncomplete = () => { + clearTimeout(timer); if (expired) return; + evidence.indexed_db[write ? "write_committed" : "read_completed"] = true; + resolve(); + }; + tx.onabort = () => { clearTimeout(timer); reject(new Error("transaction_aborted")); }; + tx.onerror = () => { /* The transaction abort is the terminal failure witness. */ }; + }); + if (mode === "write") await transaction(true); + await transaction(false); + evidence.measurement_ok = true; + evidence.ok = evidence.cookie.matches && evidence.local_storage.matches && evidence.indexed_db.matches && + evidence.indexed_db.read_request_succeeded && evidence.indexed_db.read_completed && + (mode === "read" || evidence.indexed_db.write_request_succeeded && evidence.indexed_db.write_committed); + } catch (error) { + evidence.error = ["storage_timeout", "database_open_failed", "transaction_aborted"].includes(error.message) ? + error.message : "storage_unavailable"; + } finally { db?.close(); } + return evidence; +} + +function fixturePage(page, run, media = false, qualification = false, profile = null) { return ` Browser journey ${page} + + `); + return; + } + if (url.pathname.startsWith("/api/")) { + assert.equal(req.headers["x-elastos-home-token"], "fixture"); + const chunks = []; for await (const chunk of req) chunks.push(chunk); + const body = Buffer.concat(chunks).toString("utf8"); + const input = body ? JSON.parse(body) : null; + calls.push({ method: req.method, path: url.pathname, revision: input?.if_revision }); + if (url.pathname === workspacePath) { + if (req.method === "GET") { json(stored || initial); return; } + assert.equal(req.method, "PUT"); + assert.equal(input.schema, schema); + if (input.if_revision !== (stored?.revision ?? 0)) { + conflicts++; json({ error: "workspace revision conflict" }, 409); return; + } + if (!stored) assert.equal(input.migration_revision, initial.migration_revision); + stored = { schema, revision: (stored?.revision ?? 0) + 1, document: clone(input.document) }; + json(stored); return; + } + if (url.pathname === "/api/provider/model/offers_list") { + json({ offers: [{ id: offerId, title: "Qwen installed fixture", operation: "text.generate", + input_modalities: ["text/plain"], output_modalities: ["text/plain"] }] }); return; + } + if (url.pathname === "/api/capsules/catalog") { + json({ schema: "elastos.capsules.catalog/v1", model_catalog_state: "verified", capsules: [{ + source: "signed-model-catalog", role: "content", title: "Qwen installed fixture", cid, + installed: false, launchable: false, signature_state: "catalog-signature-verified", + model_runtime: { admitted: true, dispatch_ready: true, offer_id: offerId }, + }] }); return; + } + throw new Error(`Unexpected API operation: ${req.method} ${url.pathname}`); + } + if (url.pathname === "/favicon.ico") { res.writeHead(204); res.end(); return; } + const match = url.pathname.match(/^\/apps\/(assistant|home)\/(.*)$/); + assert.ok(match, `Unexpected asset: ${url.pathname}`); + const base = resolve(root, "capsules", match[1], "browser"); + const path = resolve(base, match[2] || "index.html"); + assert.ok(path.startsWith(`${base}/`)); + const bytes = await readFile(path); + res.writeHead(200, { ...headers, "content-type": ({ ".html": "text/html", ".js": "text/javascript", + ".mjs": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".woff2": "font/woff2" })[extname(path)] || "application/octet-stream" }); + res.end(bytes); + } catch (error) { errors.push(String(error)); json({ error: "fixture failure" }, 500); } +}); + +async function until(predicate, label) { + const deadline = Date.now() + 15000; + while (Date.now() < deadline) { + if (predicate()) return; + await new Promise(resolve => setTimeout(resolve, 25)); + } + throw new Error(`Timed out: ${label}`); +} +function assertComplete(document) { + assert.equal(new Set(document.sessions.map(item => item.id)).size, document.sessions.length); + const imported = document.sessions.find(item => item.title === "Assistant complete history"); + assert.equal(imported.messages.length, 64); + imported.messages.forEach((message, index) => { + assert.ok(message.text === messages[index].content, `Exact text changed at message ${index}`); + assert.ok(message.content === messages[index].content, `Exact content changed at message ${index}`); + assert.equal(message.request_id, messages[index].request_id); + assert.deepEqual(message.futureMetadata, messages[index].futureMetadata); + }); + for (const [source, snapshot] of Object.entries(legacy)) { + assert.deepEqual(document.legacyImports[source].snapshot, snapshot); + } + assert.equal(document.liveOfferId, offerId); + assert.equal(document.selectedModelCid, cid); +} + +let browser, page, stage = "launch"; +const ownedProof = !process.env.ASSISTANT_CONVERGENCE_PROOF_DIR; +const proofDir = ownedProof + ? await mkdtemp(resolve(tmpdir(), "assistant-convergence-")) + : resolve(process.env.ASSISTANT_CONVERGENCE_PROOF_DIR); +await mkdir(proofDir, { recursive: true }); +try { + server.listen(0, "127.0.0.1"); await once(server, "listening"); + browser = await chromium.launch({ executablePath: brave, headless: true }); + const openPage = async () => { + const next = await browser.newPage({ viewport: { width: 1280, height: 900 } }); + next.on("pageerror", error => pageErrors.push(String(error))); + await next.goto(`http://127.0.0.1:${server.address().port}/`); + await next.frameLocator("iframe").locator("#agent-composer-input").waitFor({ state: "visible" }); + return next; + }; + page = await openPage(); + let frame = page.frameLocator("iframe"); + const choose = async (target, title) => { + await target.locator(".agent-harness-session-btn").filter({ hasText: title }).click(); + }; + stage = "migration and one sidebar"; + await until(() => stored?.document?.legacyImports, "canonical migration save"); + assertComplete(stored.document); + assert.equal(await frame.locator("#agent-harness-sidebar").count(), 1); + assert.equal(await frame.locator("#assistant-sidebar").count(), 0); + for (const title of ["Assistant complete history", "Sash imported conversation", "Home session imported conversation", + "Assistant saved draft", "Home Agent saved draft", "Home saved draft"]) { + assert.equal(await frame.locator(".agent-harness-session-btn").filter({ hasText: title }).count(), 1, title); + } + stage = "complete rendered history"; + await choose(frame, "Assistant complete history"); + await frame.locator("#agent-harness-stream-column").getByText("END-63", {exact:false}).first().waitFor({state:"attached"}); + const transcript = await frame.locator("#agent-harness-stream-column").textContent(); + for (let index = 0; index < 64; index++) assert.ok(transcript.includes(`END-${index}`), `Missing rendered message ${index}`); + await choose(frame, "Sash imported conversation"); + for (const role of ["grant", "tool", "system"]) { + await frame.locator(`#agent-harness-stream-column .agent-msg-${role}`).getByText(`Historical ${role}`, {exact:false}).waitFor({state:"attached"}); + } + stage = "editable imported drafts"; + for (const [title, text] of [["Assistant saved draft", legacy.assistant.draft], + ["Home saved draft", legacy.homeSessionAgent.composerDraft.text], + ["Home Agent saved draft", legacy.homeAgent.document.composerDraft.text]]) { + await choose(frame, title); + assert.equal(await frame.locator("#agent-composer-input").inputValue(), text, title); + } + const edited = "Edited imported draft: café 世界 🧭"; + await frame.locator("#agent-composer-input").fill(edited); + await until(() => stored.document.composerDraft?.text === edited, "edited draft persisted"); + stage = "reload and exact selected model"; + await page.reload(); frame = page.frameLocator("iframe"); + await frame.locator("#agent-composer-input").waitFor({ state: "visible" }); + assert.equal(await frame.locator("#agent-composer-input").inputValue(), edited); + await frame.locator("#agent-model-picker").click(); + await frame.locator(`#agent-model-menu [data-model-cid="${cid}"][aria-selected="true"]`).waitFor({ state: "visible" }); + assert.equal(await frame.locator("#agent-model-menu").getByRole("option", { name: "Qwen installed fixture", exact: true }).count(), 1); + await frame.locator("#agent-model-picker").click(); + stage = "Chat Build Studio controls"; + for (const mode of ["build", "studio", "chat"]) { + const button = frame.locator(`button[data-assistant-mode="${mode}"]`); + await button.click(); + assert.equal(await button.getAttribute("aria-pressed"), "true"); + assert.equal(await frame.locator("#assistant-studio").isVisible(), mode === "studio"); + } + // Let preceding real UI writes settle, then open a second stale writer. + await until(() => stored.document.sessionMode === "chat", "mode save"); + const second = await openPage(); + const secondFrame = second.frameLocator("iframe"); + page = second; // Capture the active conflict UI if this phase fails. + stage = "two-page CAS draft conflict"; + const firstDraft = "Window one keeps this draft 🪟"; + const secondDraft = "Window two keeps this draft 🌍"; + await frame.locator("#agent-composer-input").fill(firstDraft); + await until(() => stored.document.composerDraft?.text === firstDraft, "first window save"); + await secondFrame.locator("#agent-composer-input").fill(secondDraft); + await until(() => conflicts > 0 && stored.document.composerDraft?.text === secondDraft + && stored.document.sessions.some(item => item.composerDraft?.text === firstDraft), "both conflicting drafts saved"); + const recovered = stored.document.sessions.find(item => item.composerDraft?.text === firstDraft && item.recoveryIdentity); + assert.ok(recovered, "Conflict must have a visible recovery entry"); + await choose(secondFrame, recovered.title); + assert.equal(await secondFrame.locator("#agent-composer-input").inputValue(), firstDraft); + assertComplete(stored.document); + assert.equal(calls.filter(call => call.path.includes("/runs_")).length, 0, "Opening, migration, modes and recovery must not invoke runs"); + assert.deepEqual(errors, []); + assert.deepEqual(pageErrors, []); + console.log("PASS canonical Assistant rendered migration, complete history, editable drafts, model selection, modes and two-page CAS recovery"); + if (ownedProof) await rm(proofDir, { recursive: true, force: true }); +} catch (error) { + const report = { stage, error: String(error.stack || error), calls, errors, pageErrors, conflicts, revision: stored?.revision }; + await writeFile(resolve(proofDir, "first-failure.json"), JSON.stringify(report, null, 2)); + if (page) await page.screenshot({ path: resolve(proofDir, "first-failure.png"), fullPage: true }).catch(() => {}); + console.error(`Assistant convergence failed at ${stage}. Proof: ${proofDir}`); + throw error; +} finally { + await browser?.close(); + server.closeAllConnections(); + if (server.listening) await new Promise(resolve => server.close(resolve)); +} diff --git a/scripts/assistant-studio-preservation.test.mjs b/scripts/assistant-studio-preservation.test.mjs new file mode 100644 index 000000000..dbc8234fd --- /dev/null +++ b/scripts/assistant-studio-preservation.test.mjs @@ -0,0 +1,201 @@ +import assert from "node:assert/strict"; +import { test as nodeTest } from "node:test"; +import { readFileSync } from "node:fs"; + +const test = (name, callback) => nodeTest(name, { timeout: 5000 }, callback); + +// Import the production controller with its browser-only absolute import mapped +// to the same local module used by the existing Assistant controller fixture. +const asset = path => new URL(`../capsules/${path}`, import.meta.url).href; +const source = readFileSync(new URL("../capsules/assistant/browser/assistant.js", import.meta.url), "utf8") + .replace('"/apps/home/home-clipboard-client.js?v=home-20260726a"', JSON.stringify(asset("home/browser/home-clipboard-client.js"))) + .replace('"./vendor/katex/katex.mjs"', JSON.stringify(asset("assistant/browser/vendor/katex/katex.mjs"))) + .replace('"./model-selection.js"', JSON.stringify(asset("_shared/model-selection.js"))); +const { createAssistantApp } = await import(`data:text/javascript,${encodeURIComponent(source)}`); +const clone = value => structuredClone(value); +const runId = `run:sha256:${"a".repeat(64)}`; +const output = { schema: "elastos.model.output.content/v1", resource_id: "elastos://content/studio-fixture" }; +const offer = { id: "studio-fixture", title: "Fixture video", operation: "video.generate", + input_modalities: ["application/json"], output_modalities: ["application/json"] }; +const response = data => ({ ok: true, status: 200, json: async () => clone(data) }); +const completed = () => response({ status: "ok", data: { + schema: "elastos.model.run/v1", run_id: runId, status: "completed", sequence_cursor: 1, + terminal: { output }, +} }); +const events = (cursor = 8) => response({ status: "ok", data: { + schema: "elastos.model.run-events/v1", run_id: runId, next_cursor: cursor, has_more: false, + events: [{ sequence: cursor, kind: "output", terminal: true, data: output }], +} }); +const defer = () => { + let resolve, reject; + const promise = new Promise((yes, no) => { resolve = yes; reject = no; }); + return { promise, resolve, reject }; +}; +const flush = () => new Promise(setImmediate); + +async function fixture(t, { studioState = {}, beforeCreate = async () => true, + create = async () => completed(), readEvents = async () => events() } = {}) { + const calls = [], savedBeforeCreate = [], snapshots = [], timers = new Map(); + let ordinal = 0, timerId = 0; + const app = createAssistantApp({ + studioOnly: true, studioSessionId: "studio-session", studioState, + homeToken: "fixture-token", homeOrigin: "https://home.example", + cryptoRef: { randomUUID: () => `fixture-request-${++ordinal}` }, + nowFn: () => 1000, + setTimeoutFn: callback => { const id = ++timerId; timers.set(id, callback); return id; }, + clearTimeoutFn: id => timers.delete(id), + homeClipboardClientFactory: () => ({ start() {}, stop() {} }), + onStateChange: view => snapshots.push(clone(view)), + beforeRunCreate: async pending => { + // This is the host's persistence seam. The complete pending identity must + // be observable here before the HTTP side effect is allowed to begin. + savedBeforeCreate.push({ pending: clone(pending), view: clone(app.snapshot()) }); + return beforeCreate(pending); + }, + fetchFn: async (url, init = {}) => { + assert.equal(init.headers["x-elastos-home-token"], "fixture-token"); + const body = init.body ? JSON.parse(init.body) : null; + calls.push({ url, body }); + if (url === "/api/provider/model/offers_list") return response({ offers: [offer] }); + if (url === "/api/capsules/catalog") return response({ capsules: [] }); + if (url === "/api/provider/model/runs_create") { + assert.ok(savedBeforeCreate.length, "create requires a prior persistence callback"); + assert.equal(body.request_id, savedBeforeCreate.at(-1).pending.createRequestId); + assert.equal(savedBeforeCreate.at(-1).view.activeRun.createRequestId, body.request_id); + return create(body); + } + if (url === "/api/provider/model/runs_events") return readEvents(body); + throw new Error(`Unexpected request: ${url}`); + }, + }); + t.after(() => app.dispose()); + await app.initialize(); + return { app, calls, savedBeforeCreate, snapshots, timers, + operations: () => calls.filter(call => call.url.includes("/runs_")) }; +} + +for (const failure of ["refused", "throws"]) { + test(`Studio persistence ${failure} prevents create and retains draft/request identity`, async t => { + const prompt = "Keep this draft 世界 🧭"; + const f = await fixture(t, { studioState: { studioDraft: prompt }, beforeCreate: async () => { + if (failure === "throws") throw new Error("workspace save failed"); + return false; + } }); + assert.equal(await f.app.sendDraft(), false); + assert.deepEqual(f.operations(), []); + assert.equal(f.app.snapshot().studioDraft, prompt); + const pending = f.savedBeforeCreate[0].pending; + assert.equal(pending.prompt, prompt); + assert.equal(pending.draft, prompt); + assert.equal(pending.sessionId, "studio-session"); + assert.equal(pending.actorCapsule, "assistant"); + assert.equal(f.app.snapshot().activeRun.createRequestId, pending.createRequestId); + assert.equal(f.app.snapshot().activeRun.status, "not_sent"); + assert.equal(f.app.snapshot().studioHistory[0].createRequestId, pending.createRequestId); + }); +} + +test("Studio double-submit during a delayed create makes one request", async t => { + const pending = defer(), started = defer(); + const f = await fixture(t, { studioState: { studioDraft: "One deliberate video" }, + create: body => { started.resolve(body); return pending.promise; } }); + const first = f.app.sendDraft(); + await started.promise; + assert.equal(await f.app.sendDraft(), false); + assert.equal(f.savedBeforeCreate.length, 1); + assert.equal(f.operations().length, 1); + assert.equal(f.app.snapshot().activeRun.status, "submitting"); + pending.resolve(completed()); + assert.equal(await first, true); + assert.deepEqual(f.operations().map(call => call.url), ["/api/provider/model/runs_create"]); +}); + +test("lost Studio create response preserves its request across reload without redispatch", async t => { + const prompt = "Uncertain submission remains editable 🌍"; + const f = await fixture(t, { studioState: { studioDraft: prompt }, create: async () => { + throw new TypeError("response lost after dispatch"); + } }); + assert.equal(await f.app.sendDraft(), false); + const saved = clone(f.app.snapshot()); + assert.equal(saved.activeRun.status, "settlement_unknown"); + assert.equal(saved.activeRun.terminal, false); + assert.equal(saved.activeRun.runId, ""); + assert.equal(saved.activeRun.createRequestId, f.operations()[0].body.request_id); + assert.equal(saved.studioDraft, prompt); + assert.equal(saved.studioHistory[0].createRequestId, saved.activeRun.createRequestId); + assert.equal(await f.app.sendDraft(), false); + await f.app.resumeRun(); + assert.equal(f.operations().length, 1); + const restored = await fixture(t, { studioState: saved }); + await flush(); + await restored.app.resumeRun(); + assert.equal(await restored.app.sendDraft(), false); + assert.equal(await restored.app.stopRun(), false); + assert.deepEqual(restored.operations(), []); + assert.equal(restored.app.snapshot().activeRun.createRequestId, saved.activeRun.createRequestId); + assert.equal(restored.app.snapshot().studioDraft, prompt); +}); + +const restoredRun = { + runId, createRequestId: "original-request", sessionId: "studio-session", actorCapsule: "assistant", + mode: "studio", offerId: offer.id, operation: offer.operation, afterSequence: 7, + terminal: false, status: "running", prompt: "Original prompt", draft: "Original prompt", + outputText: "", pollErrorCount: 0, +}; + +test("restored same-actor Studio run reads its exact cursor without another create", async t => { + const f = await fixture(t, { studioState: { activeRun: restoredRun } }); + await flush(); + assert.deepEqual(f.operations().map(call => call.url), ["/api/provider/model/runs_events"]); + assert.equal(f.operations()[0].body.run_id, runId); + assert.equal(f.operations()[0].body.after_sequence, 7); + assert.equal(f.app.snapshot().activeRun.afterSequence, 8); + assert.equal(f.app.snapshot().activeRun.createRequestId, "original-request"); + assert.equal(f.app.snapshot().activeRun.status, "completed"); + assert.deepEqual(f.app.snapshot().studioHistory[0].output, output); +}); + +for (const [label, binding] of [["foreign actor", { actorCapsule: "home-agent" }], + ["history copy", { attachmentAllowed: false }]]) { + test(`Studio ${label} retains history without events or cancellation`, async t => { + const original = { ...restoredRun, ...binding }; + const f = await fixture(t, { studioState: { activeRun: original } }); + await flush(); + await f.app.resumeRun(); + assert.equal(await f.app.stopRun(), false); + assert.equal(await f.app.sendDraft(), false); + assert.deepEqual(f.operations(), []); + const saved = f.app.snapshot().activeRun; + assert.equal(saved.runId, original.runId); + assert.equal(saved.createRequestId, original.createRequestId); + assert.equal(saved.actorCapsule, original.actorCapsule); + assert.equal(saved.afterSequence, 7); + assert.equal(saved.status, "settlement_unknown"); + }); +} + +test("Studio preserves the complete Unicode prompt and previous output history", async t => { + const prompt = `Full prompt ${"世界🙂 café ".repeat(7000)} END`; + assert.ok(Buffer.byteLength(prompt) > 40000); + const history = [{ runId: "previous-run", createRequestId: "previous-request", terminal: true, + prompt, output: { ...output, resource_id: "elastos://content/previous-output" }, futureMetadata: { retained: true } }]; + const f = await fixture(t, { studioState: { studioHistory: history } }); + f.app.setDraft(prompt); + assert.equal(f.app.snapshot().studioDraft, prompt); + assert.equal(await f.app.sendDraft(), true); + assert.equal(f.operations()[0].body.input.prompt, prompt); + assert.equal(f.operations()[0].body.input.schema, "elastos.model.input.video/v1"); + assert.equal(f.savedBeforeCreate[0].pending.prompt, prompt); + assert.equal(f.savedBeforeCreate[0].view.studioDraft, prompt); + const saved = f.app.snapshot(); + assert.deepEqual(saved.studioHistory[0], history[0]); + assert.equal(saved.studioHistory.length, 2); + assert.equal(saved.studioHistory[1].prompt, prompt); + assert.equal(saved.studioHistory[1].draft, prompt); + assert.deepEqual(saved.studioHistory[1].output, output); + assert.equal(saved.studioResult.resourceId, output.resource_id); + const restored = await fixture(t, { studioState: clone(saved) }); + await flush(); + assert.deepEqual(restored.app.snapshot().studioHistory, saved.studioHistory); + assert.deepEqual(restored.operations(), []); +}); diff --git a/scripts/assistant-workspace-transition.test.mjs b/scripts/assistant-workspace-transition.test.mjs new file mode 100644 index 000000000..e2fd152a8 --- /dev/null +++ b/scripts/assistant-workspace-transition.test.mjs @@ -0,0 +1,170 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { migrateLegacyWorkspaces, mergeWorkspaceDocuments } from "../capsules/assistant/browser/workspace-transition.js"; + +const cid = `bafybei${"a".repeat(52)}`; +function legacyFixture() { + const text = "界".repeat(2730) + "ab"; // Exactly 8192 UTF-8 bytes. + const assistant = { schema: "elastos.assistant.workspace/v1", revision: 4, + selected_offer_id: "chosen", selected_model_cid: cid, draft: "Assistant draft", + future: { secretToKeep: "unknown field" }, sessions: [{ id: "same", title: "A".repeat(160), mode: "build", pinned: true, + messages: Array.from({ length: 64 }, (_, i) => ({ id: `m${i}`, role: ["user", "assistant", "system", "tool"][i % 4], + content: text, run_id: `run-${i}`, futureMessage: i })) }] }; + const homeAgent = { schema: "elastos.home-agent.workspace/v1", revision: 8, document: { + v: 1, activeSessionId: "same", liveOfferId: "chosen", selectedModelCid: cid, + projects: [{ id: "project", title: "Project", extra: true }], + composerDraft: { text: "Home Agent draft", parts: [{ id: "attachment", text, uri: "localhost://Users/user/notes", authority: "untrusted_content", extra: "retain" }] }, + sessions: [{ id: "same", projectId: "project", title: "Sash chat", archived: true, branchId: "branch", messages: [ + { id: "m", branchId: "branch", role: "user", text, parts: [{ uri: "localhost://notes", text, extra: true }] }, + { id: "m2", parentId: "m", branchId: "branch", role: "grant", summary: "Pending", args: { exact: true }, requestId: "original-grant-request" }, + ], lastTurn: { providerRunId: "original-run", createRequestId: "original-request", turnId: "turn", state: "settlement_unknown" } }], + } }; + const homeSessionAgent = { v: 1, activeSessionId: "same", workbenchTab: "plan", plan: "Legacy plan", toolMode: "ask", + projects: [{ id: "project", title: "Legacy project" }], composerDraft: { text: "Old Home draft", parts: [] }, + sessions: [{ id: "same", title: "Legacy chat", projectId: "project", forkedFrom: "parent", messages: [{ id: "m", role: "agent", text: "Earlier" }] }] }; + return { assistant, homeAgent, homeSessionAgent }; +} + +test("imports all stores, full history, unknown fields and each draft without mutating originals", () => { + const legacy = legacyFixture(); + const before = structuredClone(legacy); + const doc = migrateLegacyWorkspaces(legacy); + assert.deepEqual(legacy, before); + for (const source of Object.keys(legacy)) assert.deepEqual(doc.legacyImports[source].snapshot, legacy[source]); + const assistant = doc.sessions.find(s => s.legacyOrigin.source === "assistant" && s.legacyOrigin.id === "same"); + assert.equal(assistant.messages.length, 64); + assert.equal(Buffer.byteLength(assistant.messages[0].text), 8192); + assert.equal(assistant.title.length, 160); + assert.equal(assistant.messages[1].role, "agent"); + assert.equal(assistant.messages[1].legacyOrigin.role, "assistant"); + assert.equal(assistant.messages[2].role, "system"); + assert.equal(assistant.messages[3].role, "tool"); + assert.equal(assistant.messages[63].run_id, "run-63"); + assert.equal(assistant.messages[63].futureMessage, 63); + const sash = doc.sessions.find(s => s.legacyOrigin.source === "homeAgent" && s.legacyOrigin.id === "same"); + assert.equal(sash.archived, true); + assert.equal(sash.messages[1].role, "grant"); + assert.equal(sash.messages[1].requestId, "original-grant-request"); + assert.deepEqual(sash.lastTurn, { ...legacy.homeAgent.document.sessions[0].lastTurn, actorCapsule: "home-agent" }); + assert.equal(sash.messages[1].parentId, sash.messages[0].id); + assert.equal(sash.messages[1].branchId, sash.branchId); + assert.equal(doc.projects.find(p => p.id === sash.projectId).extra, true); + assert.equal(new Set(doc.sessions.map(s => s.id)).size, doc.sessions.length); + assert.equal(new Set(doc.projects.map(p => p.id)).size, doc.projects.length); + const drafts = doc.sessions.filter(s => s.composerDraft); + assert.deepEqual(drafts.map(s => s.composerDraft.text).sort(), ["Assistant draft", "Home Agent draft", "Old Home draft"].sort()); + assert.ok(drafts.every(s => !s.archived && s.pinned && !s.messages.length)); + assert.deepEqual(drafts.find(s => s.legacyOrigin.source === "homeAgent").composerDraft.parts, legacy.homeAgent.document.composerDraft.parts); + assert.equal(doc.liveOfferId, "chosen"); + assert.equal(doc.selectedModelCid, cid); +}); + +test("repeat import is deterministic and does not restore deleted or edited imported work", () => { + const legacy = legacyFixture(); + const doc = migrateLegacyWorkspaces(legacy); + assert.deepEqual(migrateLegacyWorkspaces(legacy), doc); + assert.deepEqual(migrateLegacyWorkspaces(legacy, doc), doc); + doc.sessions.shift(); + doc.sessions[0].title = "Edited"; + const changedLegacy = structuredClone(legacy); + changedLegacy.assistant.draft = "Changed elsewhere"; + assert.deepEqual(migrateLegacyWorkspaces(changedLegacy, doc), doc); +}); + +test("imported turns retain their original actor and exact provider identities", () => { + const legacy = legacyFixture(); + const turn = { providerRunId: "old-run", createRequestId: "old-request", turnId: "old-turn", state: "streaming", custom: { kept: true } }; + for (const [source, doc] of [["assistant", legacy.assistant], ["homeAgent", legacy.homeAgent.document], ["homeSessionAgent", legacy.homeSessionAgent]]) { + doc.sessions[0].lastTurn = structuredClone(turn); + doc.sessions[0].messages[0].turn = structuredClone(turn); + } + const canonical = migrateLegacyWorkspaces(legacy); + for (const source of Object.keys(legacy)) { + const session = canonical.sessions.find(item => item.legacyOrigin.source === source && item.legacyOrigin.id === "same"); + const expected = { ...turn, state: "settlement_unknown", actorCapsule: source === "assistant" ? "assistant" : "home-agent" }; + assert.deepEqual(session.lastTurn, expected); + assert.deepEqual(session.messages[0].turn, expected); + assert.deepEqual(canonical.legacyImports[source].snapshot, legacy[source]); + } +}); + +test("duplicate legacy IDs receive distinct deterministic IDs; providers keep exact identities", () => { + const source = { v: 1, sessions: [ + { id: "duplicate", messages: [{ id: "m", role: "agent", text: "one", run_id: "run" }, { id: "m", role: "agent", text: "two" }] }, + { id: "duplicate", messages: [] }, + ] }; + const doc = migrateLegacyWorkspaces({ homeSessionAgent: source }); + assert.notEqual(doc.sessions[0].id, doc.sessions[1].id); + assert.notEqual(doc.sessions[0].messages[0].id, doc.sessions[0].messages[1].id); + assert.equal(doc.sessions[0].messages[0].run_id, "run"); +}); + +test("source IDs preserve long and unusual identity without breaking quoted UI selectors", () => { + const id = `quotes\"\\${"x".repeat(120)}\ud800`; + const source = { v: 1, projects: [{ id, title: "Project" }], sessions: [{ id, projectId: id, messages: [] }] }; + const doc = migrateLegacyWorkspaces({ homeSessionAgent: source }); + assert.equal(doc.sessions[0].legacyOrigin.id, id); + assert.equal(doc.sessions[0].projectId, doc.projects[0].id); + assert.doesNotMatch(doc.projects[0].id, /["\\]/); + assert.ok(doc.sessions[0].id.length > 120); +}); + +test("three-way merge retains disjoint session, nested setting, and new-record changes", () => { + const base = { v: 1, settings: { a: 1, b: 2 }, sessions: [{ id: "s", title: "Old", pinned: false, messages: [] }], projects: [] }; + const local = structuredClone(base), remote = structuredClone(base); + local.settings.a = 3; local.sessions[0].title = "Local title"; + remote.settings.b = 4; remote.sessions[0].pinned = true; remote.sessions.push({ id: "remote", messages: [] }); + const result = mergeWorkspaceDocuments(base, local, remote); + assert.equal(result.requiresReview, false); + assert.deepEqual(result.document.settings, { a: 3, b: 4 }); + assert.equal(result.document.sessions[0].title, "Local title"); + assert.equal(result.document.sessions[0].pinned, true); + assert.equal(result.document.sessions.length, 2); + assert.equal(base.sessions[0].title, "Old"); +}); + +test("same-session conflict keeps both complete histories in visible deterministic copies", () => { + const base = { activeSessionId: "s", sessions: [{ id: "s", title: "Chat", messages: [{ id: "m", role: "agent", text: "base" }] }], projects: [] }; + const local = structuredClone(base), remote = structuredClone(base); + local.sessions[0].messages.push({ id: "l", role: "user", text: "local" }); + local.sessions[0].lastTurn = { providerRunId: "keep-run", createRequestId: "keep-request" }; + remote.sessions[0].messages.push({ id: "r", role: "user", text: "remote" }); + const result = mergeWorkspaceDocuments(base, local, remote); + assert.equal(result.requiresReview, true); + assert.deepEqual(result.document.sessions.find(s => s.id === "s"), remote.sessions[0]); + const recovered = result.document.sessions.find(s => s.id !== "s"); + assert.deepEqual(recovered.messages, local.sessions[0].messages); + assert.deepEqual(recovered.lastTurn, { ...local.sessions[0].lastTurn, attachmentAllowed: false }); + assert.equal(result.document.activeSessionId, recovered.id); + assert.equal(recovered.archived, false); + assert.match(recovered.title, /recovered changes/); + assert.deepEqual(mergeWorkspaceDocuments(base, local, remote), result); + assert.deepEqual(mergeWorkspaceDocuments(base, result.document, remote).document, result.document); +}); + +test("delete/edit conflicts preserve edited history; uncontested deletion stays deleted", () => { + const base = { sessions: [{ id: "s", title: "Chat", messages: [] }], projects: [] }; + const edited = structuredClone(base); edited.sessions[0].messages.push({ text: "Saved work" }); + const deleted = { sessions: [], projects: [] }; + assert.equal(mergeWorkspaceDocuments(base, deleted, edited).document.sessions[0].messages[0].text, "Saved work"); + assert.equal(mergeWorkspaceDocuments(base, edited, deleted).document.sessions[0].messages[0].text, "Saved work"); + assert.deepEqual(mergeWorkspaceDocuments(base, deleted, base).document.sessions, []); +}); + +test("conflicting workspace drafts retain both values and a visible recovery entry", () => { + const base = { composerDraft: { text: "base", parts: [] }, sessions: [], projects: [] }; + const local = structuredClone(base), remote = structuredClone(base); + local.composerDraft.text = "my unsaved draft"; + remote.composerDraft.text = "other saved draft"; + const result = mergeWorkspaceDocuments(base, local, remote); + assert.equal(result.requiresReview, true); + assert.equal(result.document.composerDraft.text, "my unsaved draft"); + assert.equal(result.conflicts[0].remote, "other saved draft"); + assert.match(result.document.sessions[0].messages[0].text, /other saved draft/); + assert.deepEqual(result.document.sessions[0].composerDraft, remote.composerDraft); + assert.equal(result.document.sessions[0].archived, false); +}); + +test("duplicate canonical record IDs fail explicitly instead of silently dropping a record", () => { + assert.throws(() => mergeWorkspaceDocuments({}, { sessions: [{ id: "same" }, { id: "same" }] }, {}), /Duplicate sessions IDs/); +}); diff --git a/scripts/home-agent-cancellation.test.mjs b/scripts/home-agent-cancellation.test.mjs index 38b7413a4..12c350857 100644 --- a/scripts/home-agent-cancellation.test.mjs +++ b/scripts/home-agent-cancellation.test.mjs @@ -2,13 +2,26 @@ import assert from "node:assert/strict"; import test from "node:test"; import { execFileSync } from "node:child_process"; -globalThis.window = { location: { href: "http://home.invalid/apps/home-agent/", hash: "#home_token=fixture" } }; -const live = await import("../capsules/home-agent/browser/agent-live.js"); -const { turnStoreGet } = await import("../capsules/home-agent/browser/agent-context.js"); -const { recoverStalePersistedTurn } = await import("../capsules/home-agent/browser/agent-stream-qos.js"); -const controller = await import("../capsules/home-agent/browser/agent-stream.js"); -const workspace = await import("../capsules/home-agent/browser/agent-workspace.js"); -const sessions = await import("../capsules/home-agent/browser/agent-sessions.js"); +function fixtureWindow(location) { + // Use real event delivery for workspace/session and beforeunload listeners. + // Older Node versions need the browser's CustomEvent detail field supplied. + globalThis.CustomEvent ??= class CustomEvent extends Event { + constructor(type, options = {}) { + super(type, options); + this.detail = options.detail ?? null; + } + }; + return Object.assign(new EventTarget(), { + location, setTimeout, clearTimeout, setInterval, clearInterval, + }); +} +globalThis.window = fixtureWindow({ href: "http://home.invalid/apps/assistant/", hash: "#home_token=fixture" }); +const live = await import("../capsules/assistant/browser/agent-live.js"); +const { turnStoreGet } = await import("../capsules/assistant/browser/agent-context.js"); +const { recoverStalePersistedTurn } = await import("../capsules/assistant/browser/agent-stream-qos.js"); +const controller = await import("../capsules/assistant/browser/agent-stream.js"); +const workspace = await import("../capsules/assistant/browser/agent-workspace.js"); +const sessions = await import("../capsules/assistant/browser/agent-sessions.js"); const defer = () => { let resolve, reject; const promise = new Promise((r, e) => { resolve = r; reject = e; }); return { promise, resolve, reject }; }; const terminal = (kind) => ({ events: [{ sequence: 1, kind, terminal: true, data: kind === "output" ? { schema: "elastos.model.output.text/v1", text: "Done" } : {} }], next_cursor: 1, has_more: false }); @@ -97,12 +110,12 @@ test("Agent selection pair roundtrips with drafts; invalid pair blocks hydration } assert.equal(workspace.applyAgentWorkspaceSnapshot(saved), true); live.selectLiveOffer("other"); - assert.equal(workspace.getAgentWorkspaceSnapshot().selectedModelCid, undefined); + assert.equal(workspace.getAgentWorkspaceSnapshot().selectedModelCid, null); live.selectLiveOffer(""); }); test("nested Agent Models intent registers top Home with exact origin and leaves draft alone", async () => { - const { openModelsFromAgent } = await import("../capsules/home-agent/browser/harness-host.js"); + const { openModelsFromAgent } = await import("../capsules/assistant/browser/harness-host.js"); const prior = { href: window.location.href, top: window.top }; const messages = []; window.top = { postMessage: (message, origin) => messages.push({ message, origin }) }; @@ -306,10 +319,11 @@ for (const mode of ["lost_response", "invalid_json", "missing_run_id", "invalid_ test("missing local launch token is proved refused before fetch", () => { execFileSync(process.execPath, ["--input-type=module", "-e", ` import assert from "node:assert/strict"; - globalThis.window = { location: { hash: "", href: "http://home.invalid/apps/home-agent/" } }; + const fixtureWindow = ${fixtureWindow.toString()}; + globalThis.window = fixtureWindow({ hash: "", href: "http://home.invalid/apps/assistant/" }); let calls = 0; globalThis.fetch = async () => { calls += 1; throw new Error("unexpected fetch"); }; - const { modelRunCall } = await import(${JSON.stringify(new URL("../capsules/home-agent/browser/agent-live.js", import.meta.url).href)}); + const { modelRunCall } = await import(${JSON.stringify(new URL("../capsules/assistant/browser/agent-live.js", import.meta.url).href)}); await assert.rejects(modelRunCall("runs_create", {}), (error) => error.code === "missing-home-launch-token" && error.preDispatchRefusal === true); assert.equal(calls, 0); @@ -322,7 +336,7 @@ function controllerFixture() { createElement: () => ({ addEventListener(type, callback) { this[type] = callback; } }), getElementById: () => null, addEventListener() {}, removeEventListener() {} }; Object.assign(window, { setTimeout, clearTimeout, setInterval, clearInterval, - requestAnimationFrame: () => 1, cancelAnimationFrame() {}, addEventListener() {}, removeEventListener() {} }); + requestAnimationFrame: () => 1, cancelAnimationFrame() {} }); const ctx = { activeSessionId: "one", sessions: [{ id: "one", messages: [] }, { id: "two", messages: [] }], streamGeneration: 0, turnBusy: true, streamTimer: 0, followUpQueue: [] }; controller.bindAgentStream(ctx, { streamEl: () => null, titleEl: () => null }); @@ -367,7 +381,10 @@ test("hydrated workspace uses the controller resume path and retains a real deni ctx.turnBusy = false; workspace.bindAgentWorkspaceStore({ getSessions: () => ctx.sessions, setSessions: (value) => { ctx.sessions = value; }, - setWorkspaceHydrated() {}, setSessionMode() {}, setActiveSessionId: (value) => { ctx.activeSessionId = value; }, + getActiveSessionId: () => ctx.activeSessionId, + getSessionMode: () => ctx.sessionMode || "chat", + setSessionMode: (value) => { ctx.sessionMode = value; }, + setWorkspaceHydrated() {}, setActiveSessionId: (value) => { ctx.activeSessionId = value; }, }); workspace.applyAgentWorkspaceSnapshot({ v: 1, activeSessionId: "one", sessions: [{ id: "one", messages: [], lastTurn: { turnId: "saved-controller", providerRunId: "saved-runtime-run", state: "streaming" } }] }); @@ -435,9 +452,9 @@ test("actual composer preserves draft for missing runs and unknown acceptance un Object.assign(window, { innerHeight: 800, matchMedia: () => ({ matches: true }) }); globalThis.requestAnimationFrame = window.requestAnimationFrame; globalThis.cancelAnimationFrame = window.cancelAnimationFrame; - const harness = await import("../capsules/home-agent/browser/agent-harness.js"); - const shelf = await import("../capsules/home-agent/browser/agent-shelf.js"); - const bridge = await import("../capsules/home-agent/browser/agent-send.js"); + const harness = await import("../capsules/assistant/browser/agent-harness.js"); + const shelf = await import("../capsules/assistant/browser/agent-shelf.js"); + const bridge = await import("../capsules/assistant/browser/agent-send.js"); bridge.registerAgentHarnessApi({ sendToAgentHarness: harness.sendToAgentHarness }); const calls = []; globalThis.fetch = async (url, init) => { diff --git a/scripts/home-agent-shell-smoke.mjs b/scripts/home-agent-shell-smoke.mjs index 5686b1af2..a2addc4e8 100644 --- a/scripts/home-agent-shell-smoke.mjs +++ b/scripts/home-agent-shell-smoke.mjs @@ -1,12 +1,12 @@ #!/usr/bin/env node -/* Home Agent capsule gate. +/* Canonical Assistant Sash capsule gate. Encodes what the capsule owes the Runtime and Home GUI, in Anders' terms: - inference runs only through the typed model contract (offers_list, runs_create with a typed input, runs_events by after_sequence, runs_cancel); no offer is named in source, no ping, no mock provider, no mock reply, no surface the Runtime does not back (tool grants, plan, workbench, - model catalog, Studio); + speculative model downloads or mock Studio); - the workspace is a Runtime object behind the capsule's launch token, revisioned, never browser storage; - Home GUI owns the morph and the place; the capsule speaks to Home only by @@ -18,21 +18,22 @@ import { readFileSync, readdirSync } from "node:fs"; const root = new URL("../", import.meta.url); const read = (path) => readFileSync(new URL(path, root), "utf8"); -const capsuleDir = new URL("capsules/home-agent/browser/", root); +const capsuleDir = new URL("capsules/assistant/browser/", root); const capsuleScripts = readdirSync(capsuleDir) .filter((name) => name.endsWith(".js")) .map((name) => [name, readFileSync(new URL(name, capsuleDir), "utf8")]); -const agentLive = read("capsules/home-agent/browser/agent-live.js"); -const agentStream = read("capsules/home-agent/browser/agent-stream.js"); -const agentHarness = read("capsules/home-agent/browser/agent-harness.js"); -const harnessHost = read("capsules/home-agent/browser/harness-host.js"); -const entry = read("capsules/home-agent/browser/home-agent.js"); -const indexHtml = read("capsules/home-agent/browser/index.html"); -const manifest = JSON.parse(read("capsules/home-agent/capsule.json")); +const agentLive = read("capsules/assistant/browser/agent-live.js"); +const agentStream = read("capsules/assistant/browser/agent-stream.js"); +const agentHarness = read("capsules/assistant/browser/agent-harness.js"); +const harnessHost = read("capsules/assistant/browser/harness-host.js"); +const entry = read("capsules/assistant/browser/home-agent.js"); +const indexHtml = read("capsules/assistant/browser/index.html"); +const manifest = JSON.parse(read("capsules/assistant/capsule.json")); const components = JSON.parse(read("components.json")); const gateway = read("elastos/crates/elastos-server/src/api/gateway.rs"); -const gatewayHomeAgent = read("elastos/crates/elastos-server/src/api/gateway_home_agent.rs"); +const gatewayWorkspace = read("elastos/crates/elastos-server/src/api/gateway_assistant_workspace_v2.rs"); +const gatewayAssistant = read("elastos/crates/elastos-server/src/api/gateway_assistant.rs"); const homeFace = read("capsules/home-gui/browser/shell-assistant-face.js"); const homeWindows = read("capsules/home-gui/browser/shell-windows.js"); const homeSurface = read("capsules/home-gui/browser/shell-surface.js"); @@ -79,8 +80,8 @@ assert.ok(agentLive.includes("selectedModelOffer(models.map"), "the model menu r /* ---- workspace: a Runtime object ----------------------------------------- */ -assert.ok(harnessHost.includes('const WORKSPACE_URL = "/api/apps/home-agent/workspace"')); -assert.ok(harnessHost.includes('const WORKSPACE_SCHEMA = "elastos.home-agent.workspace/v1"')); +assert.ok(harnessHost.includes('const WORKSPACE_URL = "/api/apps/assistant/workspace-v2"')); +assert.ok(harnessHost.includes('const WORKSPACE_SCHEMA = "elastos.assistant.workspace/v2"')); assert.ok(harnessHost.includes("if_revision: workspaceRevision"), "writes carry the revision they saw"); assert.ok(harnessHost.includes("error?.status === 409"), "a revision conflict reloads, never overwrites"); assert.ok(harnessHost.includes("if (workspaceRevision === null) {"), "no write before the read"); @@ -90,37 +91,41 @@ assert.ok( "ready is posted after the workspace is read", ); assert.equal( - (gateway.match(/\/api\/apps\/home-agent\/workspace/g) || []).length, + (gateway.match(/\/api\/apps\/assistant\/workspace-v2/g) || []).length, 1, "one workspace route", ); -assert.ok(gateway.includes("gateway_home_agent::principal_root_protected_object_inventory(localhost_root)")); -assert.ok(gatewayHomeAgent.includes('const HOME_AGENT_CAPSULE_ID: &str = "home-agent"')); -assert.ok(gatewayHomeAgent.includes("require_home_launch_token_context(&state.data_dir, &headers, HOME_AGENT_CAPSULE_ID)")); -assert.ok(gatewayHomeAgent.includes("write_protected_principal_root_object(")); -assert.ok(gatewayHomeAgent.includes("#[serde(deny_unknown_fields)]")); +assert.ok(gateway.includes("gateway_assistant::principal_root_protected_object_inventory(localhost_root)")); +assert.ok(gatewayAssistant.includes("principal_root_protected_object_inventory(")); +assert.ok(gatewayWorkspace.includes('const RELATIVE_PATH: &str = ".AppData/ElastOS/Assistant/workspace-v2.json"')); +assert.ok(gatewayWorkspace.includes('require_home_launch_token_context(&state.data_dir, &headers, "assistant")')); +assert.ok(gatewayWorkspace.includes("write_protected_principal_root_object(")); +assert.ok(gatewayWorkspace.includes("#[serde(deny_unknown_fields)]")); +assert.ok(harnessHost.includes("migrateLegacyWorkspaces(saved.legacy)"), "legacy sources enter the canonical workspace"); +assert.ok(harnessHost.includes("mergeWorkspaceDocuments(lastSnapshot || {}, local, remote || {})"), "conflicts retain both writers"); /* ---- manifest and install ------------------------------------------------ */ -assert.equal(manifest.name, "home-agent"); +assert.equal(manifest.name, "assistant"); const methods = manifest.interfaces.flatMap((i) => i.methods.map((m) => m.operation)).sort(); assert.deepEqual(methods, ["offers_list", "retention", "runs_cancel", "runs_create", "runs_events", "runs_get"]); -assert.ok(components.external?.["home-agent"], "components.json installs home-agent"); +assert.ok(components.external?.assistant, "components.json installs Assistant"); +assert.ok(!components.external?.["home-agent"], "the legacy app is retired from installation"); assert.ok( - localCarrierSetup.includes('HOME_AGENT_CAPSULE_DIR="${REPO_ROOT}/capsules/home-agent"') && - localCarrierSetup.includes('"home-agent": pathlib.Path(os.environ["HOME_AGENT_CAPSULE_DIR"])') && - localCarrierSetup.includes('"${DATA_DIR}/capsules/home-agent/browser/index.html"'), - "the local Carrier setup fixture stages and verifies home-agent", + localCarrierSetup.includes('ASSISTANT_CAPSULE_DIR="${REPO_ROOT}/capsules/assistant"') && + localCarrierSetup.includes('"assistant": pathlib.Path(os.environ["ASSISTANT_CAPSULE_DIR"])') && + localCarrierSetup.includes('"${DATA_DIR}/capsules/assistant/browser/index.html"'), + "the local Carrier setup fixture stages and verifies canonical Assistant", ); /* ---- ownership split with Home GUI ---------------------------------------- */ -assert.ok(homeFace.includes('const TARGET_ID = "home-agent"')); +assert.ok(homeFace.includes('const TARGET_ID = "assistant"')); assert.ok( - homeWindows.includes('const HOME_AGENT_TARGET_ID = "home-agent"') && + homeWindows.includes('const HOME_AGENT_TARGET_ID = "assistant"') && (homeWindows.match(/targetId === HOME_AGENT_TARGET_ID/g) || []).length === 3 && - (homeWindows.match(/showAssistantFace\(\);/g) || []).length === 2, - "Home Agent target activation and session restore must stay on the one Assistant face", + (homeWindows.match(/showAssistantFace\((?:normalizedLaunchQuery\(options\.query\))?\);/g) || []).length === 2, + "canonical and legacy target activation and session restore stay on one Assistant face", ); assert.ok( /export function openSelectedLauncherTarget\(\)[\s\S]{0,180}openTarget\(shellState\.selectedLauncherTargetId\)/.test(homeSurface) && @@ -156,7 +161,13 @@ assert.ok( assert.ok(!agentStream.includes("asHtml"), "appendMessage has no unused HTML option"); assert.ok(!agentStream.includes("body.innerHTML = text"), "appendMessage has no raw HTML branch"); assert.ok(!indexHtml.includes('data-sidebar-nav="usage"'), "Usage nav is gone"); -assert.ok(!indexHtml.includes('data-sidebar-nav="studio"'), "Studio nav is gone"); +assert.ok(!indexHtml.includes('data-sidebar-nav="studio"'), "Studio uses the shared mode controls, not a second sidebar"); +for (const mode of ["chat", "build", "studio"]) { + assert.ok(indexHtml.includes(`data-assistant-mode="${mode}"`), `${mode} is a canonical Assistant mode`); +} +assert.equal((indexHtml.match(/id="agent-harness-sidebar"/g) || []).length, 1, "one Sash sidebar"); +const modes = read("capsules/assistant/browser/assistant-modes.js"); +assert.ok(modes.includes('from "./assistant.js"') && modes.includes("studioOnly: true"), "Studio uses the typed controller"); for (const theatre of [ "data-workbench", "agent-approve-menu", @@ -180,7 +191,7 @@ assert.ok(!/preview|mock/i.test(indexHtml), "index.html carries no preview or mo for (const shelf of ["taskbar-sortable", "launcher", "space-pager", "agent-shelf-toggle", "shelf-face-apps"]) { assert.ok(!indexHtml.includes(shelf), `index.html carries Home's ${shelf}`); } -const agentShelf = read("capsules/home-agent/browser/agent-shelf.js"); +const agentShelf = read("capsules/assistant/browser/agent-shelf.js"); assert.ok( !/data-agent-morph|agentMorph|flipTaskbarGeometry|launcher/i.test(agentShelf), "the capsule runs no Shelf morph of its own; it asks Home to leave and Home runs the one morph", @@ -195,12 +206,12 @@ assert.ok( ); assert.ok(/dismiss: \(\) => leaveAgentRoom\(\)/.test(agentShelf), "Esc asks Home to leave"); assert.ok( - /closest\?\.\("#agent-harness-home"\)[\s\S]{0,400}leaveAgentRoom\(\);/.test(read("capsules/home-agent/browser/agent-harness.js")), + /closest\?\.\("#agent-harness-home"\)[\s\S]{0,400}leaveAgentRoom\(\);/.test(read("capsules/assistant/browser/agent-harness.js")), "the sidebar Home row asks Home to leave", ); -const configureSections = read("capsules/home-agent/browser/agent-configure.js"); +const configureSections = read("capsules/assistant/browser/agent-configure.js"); assert.ok(configureSections.includes('export const CONFIGURE_SECTIONS = new Set(["models", "prompt"]);'), "Settings shows only what the Runtime backs"); -assert.ok(!/Planning weekend|calm weekend/.test(read("capsules/home-agent/browser/agent-harness.js")), "no seeded conversation"); +assert.ok(!/Planning weekend|calm weekend/.test(read("capsules/assistant/browser/agent-harness.js")), "no seeded conversation"); /* ---- the pure contract module -------------------------------------------- */ @@ -402,4 +413,4 @@ assert.throws( assert.throws(() => contract.applyRunEventsPage({ events: "nope" }, 0), /malformed/); assert.throws(() => contract.applyRunEventsPage(null, 0), /malformed/); -console.log("home-agent shell smoke: ok"); +console.log("canonical Assistant Sash shell smoke: ok"); diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index 99ee87083..9cc5f6725 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -868,25 +868,34 @@ for (const [token, value] of new Map([ } const assistantIndex = read("capsules/assistant/browser/index.html"); -const assistantScript = read("capsules/assistant/browser/assistant.js"); -const assistantStyle = read("capsules/assistant/browser/style.css"); +const assistantController = read("capsules/assistant/browser/assistant.js"); +const assistantEntry = read("capsules/assistant/browser/home-agent.js"); +const assistantWorkspace = read("capsules/assistant/browser/harness-host.js"); +const assistantModes = read("capsules/assistant/browser/assistant-modes.js"); +const assistantScript = [assistantController, assistantEntry, assistantWorkspace, assistantModes, + read("capsules/assistant/browser/agent-live.js")].join("\n"); +const assistantStyle = ["style.css", "home-agent.css", "agent-harness.css", "assistant-modes.css"] + .map(name => read(`capsules/assistant/browser/${name}`)).join("\n"); +const assistantWorkspaceGateway = read("elastos/crates/elastos-server/src/api/gateway_assistant_workspace_v2.rs"); const assistantGateway = read( "elastos/crates/elastos-server/src/api/gateway_assistant.rs", ); const gatewaySource = read("elastos/crates/elastos-server/src/api/gateway.rs"); assert( - assistantIndex.includes(''), - "Assistant must boot through its capsule-owned browser shell", + assistantIndex.includes('') && + assistantEntry.includes('from "./assistant-modes.js"') && + assistantModes.includes('from "./assistant.js"') && assistantModes.includes("studioOnly: true"), + "Assistant boots the Sash shell and uses the typed controller for Studio", ); assert( !/\blocalStorage\b|\bsessionStorage\b|\bindexedDB\b/.test(assistantScript), "Assistant must not add browser-owned persistence", ); assert( - assistantScript.includes("/api/apps/assistant/workspace") && - !assistantScript.includes("workspace.json") && - !assistantScript.includes("session.agent"), + assistantWorkspace.includes('const WORKSPACE_URL = "/api/apps/assistant/workspace-v2"') && + !assistantWorkspace.includes("workspace.json") && + !assistantWorkspace.includes("session.agent"), "Assistant shell must use the dedicated Runtime workspace route without reaching for Home session.agent or raw path literals", ); assert( @@ -923,17 +932,27 @@ assert( ); assert( !/https?:\/\/|ws:\/\/|wss:\/\/|127\.0\.0\.1|carrier_route|connect_ticket|backend_url|api_key|bearer/i.test( - `${assistantIndex}\n${assistantScript}\n${assistantStyle}`, + // SVG namespace identifiers inside local data images are not endpoints. + `${assistantIndex}\n${assistantScript}\n${assistantStyle}`.replaceAll("http://www.w3.org/2000/svg", "svg-namespace"), ), "Assistant capsule source must not expose backend endpoints, topology, or credentials", ); assert( - (gatewaySource.match(/\/api\/apps\/assistant\/workspace/g) || []).length === 1, - "Assistant workspace must expose exactly one dedicated Runtime route", + (gatewaySource.match(/"\/api\/apps\/assistant\/workspace-v2"/g) || []).length === 1 && + (gatewaySource.match(/"\/api\/apps\/assistant\/workspace"/g) || []).length === 1, + "Assistant has one canonical v2 route and retains the legacy workspace route", +); +assert( + assistantWorkspaceGateway.includes('const SCHEMA: &str = "elastos.assistant.workspace/v2"') && + assistantWorkspaceGateway.includes("MAX_BYTES: usize = 8 * 1024 * 1024") && + assistantWorkspaceGateway.includes('const RELATIVE_PATH: &str = ".AppData/ElastOS/Assistant/workspace-v2.json"') && + assistantWorkspaceGateway.includes("read_principal_root_object(") && + assistantWorkspaceGateway.includes("write_protected_principal_root_object("), + "Canonical Assistant uses bounded v2 storage under the protected principal root", ); assert( (assistantGateway.match(/elastos\.assistant\.workspace\/v1/g) || []).length >= 1, - "Assistant workspace must use one bounded v1 schema", + "Legacy Assistant reads retain their v1 schema", ); assert( (assistantGateway.match( @@ -944,7 +963,7 @@ assert( ) || []).length === 1 && (assistantGateway.match(/fn assistant_workspace_uri\(/g) || []).length === 1 && (assistantGateway.match(/\.AppData\/ElastOS\/Assistant\/workspace\.json/g) || []).length === 0, - "Assistant workspace must use exactly one protected principal-root file", + "Legacy Assistant retains its exact protected workspace file for adoption", ); assert( assistantGateway.includes("read_principal_root_object(") && @@ -953,7 +972,7 @@ assert( ); assert( !/https?:\/\/|ws:\/\/|wss:\/\/|127\.0\.0\.1|carrier_route|connect_ticket|backend_url|api_key|bearer/i.test( - assistantGateway, + `${assistantGateway}\n${assistantWorkspaceGateway}`, ), "Assistant workspace route must not encode backend endpoints, topology, or credentials", ); @@ -11426,7 +11445,7 @@ assert( gatewayApi.includes('WALLET_CAPSULE_ID => "Wallet"') && gatewayApi.includes("fn home_launch_target") && gatewayApi.includes("fn is_home_visible_target") && - gatewayApi.includes( + gatewayApi.replace(/\s+/g, " ").includes( "WALLET_UNISAT_CAPSULE_ID | WALLET_WALLETCONNECT_CAPSULE_ID", ) && gatewayTests.includes( diff --git a/scripts/home-recovery-save-smoke.mjs b/scripts/home-recovery-save-smoke.mjs index 7f31db95a..de5efa6c6 100644 --- a/scripts/home-recovery-save-smoke.mjs +++ b/scripts/home-recovery-save-smoke.mjs @@ -36,7 +36,7 @@ function hostFixture({ invalid = false, fail = false, ordinary = false } = {}) { const hooks = { holdHomeSetupAct: () => false }; const context = vm.createContext({ pendingSystemRecoverySave: null, SYSTEM_APP_ID: "system", shellState: state, - HOME_AGENT_TARGET_ID: "home-agent", windowHooks: hooks, + HOME_AGENT_TARGET_ID: "assistant", windowHooks: hooks, requireWindowHooks: () => hooks, targetById: () => target, pendingWindowLaunches: new Map(), ignoreRepeatedAction: () => false, diff --git a/scripts/home-shell-regression-smoke.mjs b/scripts/home-shell-regression-smoke.mjs index e90f54501..a02e9200e 100644 --- a/scripts/home-shell-regression-smoke.mjs +++ b/scripts/home-shell-regression-smoke.mjs @@ -597,7 +597,7 @@ assert( const upgradedLayoutSummary = { authority: { signed_in: true }, targets: [ - { target: "home-agent", title: "Home Agent", route: "/apps/home-agent/" }, + { target: "assistant", title: "Assistant", route: "/apps/assistant/" }, { target: "wallet", title: "Wallet", route: "/apps/wallet/" }, ], browser_state: { @@ -613,7 +613,7 @@ const upgradedLayoutSummary = { shellCore.initializeShellLayout(upgradedLayoutSummary); assert( - JSON.stringify(shellCore.shellState.shellLayoutState.desktopHidden) === JSON.stringify(["home-agent"]), + JSON.stringify(shellCore.shellState.shellLayoutState.desktopHidden) === JSON.stringify(["assistant"]), "newly installed target became visible on an existing desktop", shellCore.shellState.shellLayoutState, ); @@ -1133,17 +1133,18 @@ const restoredWithHomeAgent = shellWindows.normalizeRestorableSession( ...summary, targets: [ ...summary.targets, - { target: "home-agent", title: "Home Agent", route: "/apps/home-agent/" }, + { target: "home-agent", title: "Legacy Home Agent", route: "/apps/home-agent/" }, + { target: "assistant", title: "Assistant", route: "/apps/assistant/" }, ], }, { root_shell: "home-gui", - windows: [{ target: "home-agent", active: true }], + windows: [{ target: "home-agent", active: true }, { target: "assistant", active: false }], }, ); assert( restoredWithHomeAgent.length === 0, - "Home Agent was restored as a normal browser window", + "canonical Assistant or its legacy alias was restored as a normal browser window", restoredWithHomeAgent, ); @@ -1227,11 +1228,13 @@ shellWindows.configureWindowHooks({ }, }); +shellWindows.openTarget("assistant"); +shellWindows.handleTaskbarTargetClick("assistant"); shellWindows.openTarget("home-agent"); shellWindows.handleTaskbarTargetClick("home-agent"); assert( restoredBrowserLaunches.length === 0 && shellCore.shellState.windows.size === 0, - "Home Agent activation created a normal browser window or launch", + "Assistant or legacy alias activation created a normal browser window or launch", { launches: restoredBrowserLaunches, windows: [...shellCore.shellState.windows.keys()] }, ); @@ -2160,8 +2163,11 @@ const simpleSingleApps = [ "assistant", "marketplace", "services", "wallet-metamask", "wallet-unisat", "wallet-walletconnect", ]; -const simpleTargets = simpleSingleApps.map((target) => { - const manifest = JSON.parse(readFileSync(new URL(`../capsules/${target}/capsule.json`, import.meta.url), "utf8")); +const simpleTargets = simpleSingleApps.map((source) => { + const manifest = JSON.parse(readFileSync(new URL(`../capsules/${source}/capsule.json`, import.meta.url), "utf8")); + // Keep the complete metadata-driven window-policy test. Assistant's owned + // face and its legacy alias are checked separately above. + const target = source === "assistant" ? "single-policy-fixture" : source; assert(manifest.window_policy === "single", `${target} must declare its single-window policy`); return { target, title: target, route: `/apps/${target}/`, window_policy: manifest.window_policy }; }); @@ -2230,8 +2236,8 @@ for (const target of simpleTargets) { `${target.target}: launcher still imposes single status without Runtime projection`); target.window_policy = "single"; } -const ownedAgent = JSON.parse(readFileSync(new URL("../capsules/home-agent/capsule.json", import.meta.url), "utf8")); -assert(!ownedAgent.window_policy, "standalone Assistant policy changed the owned Home Agent role"); +const ownedAssistant = JSON.parse(readFileSync(new URL("../capsules/assistant/capsule.json", import.meta.url), "utf8")); +assert(ownedAssistant.window_policy === "single", "canonical Assistant declares its single-instance policy"); console.log("[home-shell-regression] single-window consumers: 6 PASS"); const hybridContentQueries = { "chat-room": { conversation_id: "conversation:fixture-one" }, @@ -2527,7 +2533,7 @@ const inventory = readdirSync(new URL("../capsules/", import.meta.url)).flatMap( const path = new URL(`../capsules/${name}/capsule.json`, import.meta.url); return existsSync(path) ? [JSON.parse(readFileSync(path, "utf8"))] : []; }); -assert(inventory.length === 41, "Update the exact capsule role/policy inventory when membership changes"); +assert(inventory.length === 40, "Update the exact capsule role/policy inventory when membership changes"); for (const [policy, count] of [["single", 10], ["hybrid", 6], ["multiple", 1]]) { assert(inventory.filter((manifest) => manifest.window_policy === policy).length === count, `${policy} inventory drift`); } @@ -2535,9 +2541,9 @@ for (const [role, count] of [["provider", 18], ["content", 2], ["shell", 2]]) { const owned = inventory.filter((manifest) => manifest.role === role); assert(owned.length === count && owned.every((manifest) => !manifest.window_policy), `${role} must retain its owned non-window role`); } -assert(inventory.filter((manifest) => manifest.role === "app" && !manifest.window_policy).map((manifest) => manifest.name).sort().join(",") === "home,home-agent", - "Home host and owned Agent must stay separate from ordinary app window policy"); -console.log("[home-shell-regression] all41 roles, own menus and selected Player: PASS"); +assert(inventory.filter((manifest) => manifest.role === "app" && !manifest.window_policy).map((manifest) => manifest.name).sort().join(",") === "home", + "Home alone owns the app host role without a window policy"); +console.log("[home-shell-regression] all40 roles, own menus and selected Player: PASS"); // A verified in-app selection must replace only that window's // persisted presentation selector, then restore through a fresh Runtime launch. diff --git a/scripts/local-carrier-setup-smoke.sh b/scripts/local-carrier-setup-smoke.sh index a190fc98b..57a44b339 100755 --- a/scripts/local-carrier-setup-smoke.sh +++ b/scripts/local-carrier-setup-smoke.sh @@ -141,7 +141,6 @@ for capsule in \ home-gui \ archive-manager \ assistant \ - home-agent \ browser \ system \ services \ @@ -204,7 +203,6 @@ LIBRARY_CAPSULE_DIR="${REPO_ROOT}/capsules/library" \ MARKETPLACE_CAPSULE_DIR="${REPO_ROOT}/capsules/marketplace" \ ARCHIVE_MANAGER_CAPSULE_DIR="${REPO_ROOT}/capsules/archive-manager" \ ASSISTANT_CAPSULE_DIR="${REPO_ROOT}/capsules/assistant" \ -HOME_AGENT_CAPSULE_DIR="${REPO_ROOT}/capsules/home-agent" \ ELACITY_PLAYER_CAPSULE_DIR="${REPO_ROOT}/capsules/elacity-player" \ INBOX_CAPSULE_DIR="${REPO_ROOT}/capsules/inbox" \ WALLET_CAPSULE_DIR="${REPO_ROOT}/capsules/wallet" \ @@ -324,7 +322,6 @@ browser_capsules = { "marketplace": pathlib.Path(os.environ["MARKETPLACE_CAPSULE_DIR"]), "archive-manager": pathlib.Path(os.environ["ARCHIVE_MANAGER_CAPSULE_DIR"]), "assistant": pathlib.Path(os.environ["ASSISTANT_CAPSULE_DIR"]), - "home-agent": pathlib.Path(os.environ["HOME_AGENT_CAPSULE_DIR"]), "elacity-player": pathlib.Path(os.environ["ELACITY_PLAYER_CAPSULE_DIR"]), "wallet": pathlib.Path(os.environ["WALLET_CAPSULE_DIR"]), "wallet-metamask": pathlib.Path(os.environ["WALLET_METAMASK_CAPSULE_DIR"]), @@ -527,7 +524,7 @@ for installed in \ "${DATA_DIR}/capsules/marketplace/browser/marketplace.css" \ "${DATA_DIR}/capsules/marketplace/browser/marketplace.js" \ "${DATA_DIR}/capsules/archive-manager/browser/index.html" \ - "${DATA_DIR}/capsules/home-agent/browser/index.html" \ + "${DATA_DIR}/capsules/assistant/browser/index.html" \ "${DATA_DIR}/capsules/wallet/browser/index.html" \ "${DATA_DIR}/capsules/wallet-metamask/browser/index.html" \ "${DATA_DIR}/capsules/wallet-unisat/browser/index.html" \ diff --git a/scripts/model-selection-browser-smoke.mjs b/scripts/model-selection-browser-smoke.mjs index c9d3f7d38..3cc6ff0e7 100644 --- a/scripts/model-selection-browser-smoke.mjs +++ b/scripts/model-selection-browser-smoke.mjs @@ -9,7 +9,7 @@ import { chromium, brave } from "./system-uiux-fixture.mjs"; const root = resolve(new URL("../", import.meta.url).pathname); const cid = `bafybei${"a".repeat(52)}`; const offer = (id, title) => ({ id, title, operation: "text.generate", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }); -let offers, rows, failed = false, responseDelay = 0; +let offers, rows, workspace, failed = false, responseDelay = 0; const calls = [], errors = []; const server = createServer(async (req, res) => { const url = new URL(req.url, "http://fixture"); @@ -32,11 +32,13 @@ const server = createServer(async (req, res) => { json({ offers }, failed ? 503 : 200); return; } if (url.pathname === "/api/capsules/catalog") { json({ schema: "elastos.capsules.catalog/v1", model_catalog_state: "verified", capsules: rows }); return; } - if (url.pathname.endsWith("workspace")) { - if (req.method === "PUT") { json({ ...input, revision: input.if_revision + 1 }); return; } - json(url.pathname.includes("home-agent") - ? { schema: "elastos.home-agent.workspace/v1", revision: 0, document: { v: 1, liveOfferId: "chosen", selectedModelCid: cid, composerDraft: { text: "Keep my draft", parts: [] } } } - : { schema: "elastos.assistant.workspace/v1", revision: 0, sessions: [], draft: "Keep my draft", selected_offer_id: "chosen", selected_model_cid: cid }); return; + if (url.pathname === "/api/apps/assistant/workspace-v2") { + if (req.method === "PUT") { + assert.equal(input.schema, "elastos.assistant.workspace/v2"); + if (input.if_revision !== workspace.revision) { json({ error: "revision conflict" }, 409); return; } + workspace = { schema: input.schema, revision: workspace.revision + 1, document: input.document }; + } + json(workspace); return; } throw new Error(`Unexpected operation ${url.pathname}`); } @@ -51,7 +53,14 @@ const server = createServer(async (req, res) => { server.listen(0, "127.0.0.1"); await once(server, "listening"); const browser = await chromium.launch({ executablePath: brave, headless: true }); try { - for (const app of ["assistant", "home-agent"]) { + // The retired standalone renderer's disabled-select/CID/controller behavior + // remains covered in assistant-shell-smoke.mjs. Render the canonical UI here. + { + const app = "assistant"; + workspace = { schema: "elastos.assistant.workspace/v2", revision: 0, document: { + v: 1, liveOfferId: "chosen", selectedModelCid: cid, sessions: [], + composerDraft: { text: "Keep my draft", parts: [] }, + } }; offers = [offer("chosen", "Prepared model"), offer("hosted", "Hosted service")]; failed = false; rows = [{ source: "signed-model-catalog", role: "content", installed: false, launchable: false, cid, title: "Prepared model", signature_state: "catalog-signature-verified", model_runtime: { admitted: true, dispatch_ready: true, offer_id: "chosen" } }]; @@ -59,16 +68,20 @@ try { const pageErrors = []; page.on("pageerror", error => pageErrors.push(String(error))); await page.goto(`http://127.0.0.1:${server.address().port}/?app=${app}`); const frame = page.frameLocator("iframe"); - const agent = app === "home-agent"; - const draft = frame.locator(agent ? "#agent-composer-input" : "#assistant-composer-input"); - const send = frame.locator(agent ? "#agent-composer-send" : "#assistant-send"); + const draft = frame.locator("#agent-composer-input"); + const send = frame.locator("#agent-composer-send"); await draft.filter({ visible: true }).waitFor({ timeout: 5000 }); - await frame.locator(`${agent ? "#agent-composer-send" : "#assistant-send"}:enabled`).waitFor(); + await frame.locator("#agent-composer-send:enabled").waitFor(); assert.equal(await draft.inputValue(), "Keep my draft"); - if (!agent) { + { for (const width of [680, 420]) { await page.setViewportSize({ width, height: 800 }); - const controls = await frame.locator('[data-action="refresh-models"], [data-action="open-models"]').evaluateAll(buttons => buttons.map(button => { + await frame.locator("#agent-model-picker").click(); + await frame.locator("#agent-model-menu").waitFor({ state: "visible" }); + await frame.locator("#agent-model-menu").evaluate(async element => { + await Promise.all(element.getAnimations().map(animation => animation.finished)); + }); + const controls = await frame.locator('#agent-model-menu [data-model-action="refresh-models"], #agent-model-menu [data-model-action="open-models"]').evaluateAll(buttons => buttons.map(button => { const rect = button.getBoundingClientRect(); return { left: rect.left, right: rect.right, top: rect.top, bottom: rect.bottom, width: rect.width, height: rect.height, viewportWidth: innerWidth, viewportHeight: innerHeight, @@ -83,14 +96,15 @@ try { const [a, b] = controls; assert.ok(a.right <= b.left || b.right <= a.left || a.bottom <= b.top || b.bottom <= a.top, `Assistant model controls overlap at ${width}px`); + await frame.locator("#agent-model-picker").click(); } await page.setViewportSize({ width: 1100, height: 800 }); } - if (agent) await frame.locator("#agent-model-picker").click(); - const menu = agent ? frame.locator("#agent-model-menu") : frame.locator("#assistant-offer-select"); + await frame.locator("#agent-model-picker").click(); + const menu = frame.locator("#agent-model-menu"); assert.equal(await menu.getByRole("option", { name: "Prepared model", exact: true }).count(), 1); assert.equal(await menu.getByRole("option", { name: "Hosted service", exact: true }).count(), 1); - if (agent) { + { offers = [offer("chosen", "Prepared model")]; await menu.getByRole("button", { name: "Refresh models", exact: true }).click(); const selected = menu.locator(`[data-model-cid="${cid}"][aria-selected="true"]`); @@ -123,20 +137,20 @@ try { await frame.locator("#agent-model-picker").click(); } offers = [offer("hosted", "Hosted service")]; rows = []; - await (agent ? menu : frame).getByRole("button", { name: "Refresh models", exact: true }).click(); + await menu.getByRole("button", { name: "Refresh models", exact: true }).click(); await menu.getByRole("option", { name: "Prepared model", exact: true }).waitFor({ state: "detached" }); assert.equal(await send.isDisabled(), true); assert.equal(await draft.inputValue(), "Keep my draft"); - if (agent) await menu.getByRole("option", { name: "Hosted service", exact: true }).click(); - else await menu.selectOption("hosted"); + await menu.getByRole("option", { name: "Hosted service", exact: true }).click(); assert.equal(await send.isEnabled(), true); assert.equal(await draft.inputValue(), "Keep my draft"); failed = true; - if (agent) await frame.locator("#agent-model-picker").click(); - await (agent ? menu : frame).getByRole("button", { name: "Refresh models", exact: true }).click(); - await frame.locator(`${agent ? "#agent-composer-send" : "#assistant-send"}:disabled`).waitFor(); - if (!agent) assert.equal(await menu.isDisabled(), true); - await (agent ? menu : frame).getByRole("button", { name: "Open Models", exact: true }).click(); + await frame.locator("#agent-model-picker").click(); + await menu.getByRole("button", { name: "Refresh models", exact: true }).click(); + await frame.locator("#agent-composer-send:disabled").waitFor(); + await menu.locator('[role="option"][aria-selected="true"]').waitFor({ state: "detached" }); + assert.equal(await menu.locator('[role="option"][aria-selected="true"]').count(), 0, "failed refresh leaves no selectable current model"); + await menu.getByRole("button", { name: "Open Models", exact: true }).click(); const message = await page.waitForFunction(() => window.messages.find(m => m.type === "home:open-target")); assert.deepEqual(await message.jsonValue(), { type: "home:open-target", homeToken: "fixture", target: "system", query: { settings: "models" } }); assert.equal(await draft.inputValue(), "Keep my draft"); @@ -145,5 +159,5 @@ try { } assert.equal(calls.filter(call => call.path.endsWith("runs_create")).length, 0); assert.deepEqual(errors, []); - console.log("PASS Assistant/Home Agent model selection browser smoke"); + console.log("PASS canonical Assistant model selection browser smoke"); } finally { await browser.close(); server.closeAllConnections(); await new Promise(resolve => server.close(resolve)); } diff --git a/scripts/publish-release.sh b/scripts/publish-release.sh index ea6f8ba84..7f695ead9 100755 --- a/scripts/publish-release.sh +++ b/scripts/publish-release.sh @@ -76,7 +76,6 @@ DEFAULT_CAPSULES=( archive-manager inbox assistant - home-agent elacity-player ) CAPSULES=("${DEFAULT_CAPSULES[@]}") @@ -110,7 +109,6 @@ REQUIRED_SUPPORTED_CAPSULES=( archive-manager inbox assistant - home-agent elacity-player ) SUPPORT_BINARY_ASSETS=( @@ -1018,7 +1016,6 @@ build_platform_independent_direct_assets() { gba-nonogram \ chat-room \ assistant \ - home-agent \ elacity-player; do if [[ -n "${ARTIFACTS_DIR:-}" && -f "${ARTIFACTS_DIR}/${capsule}.capsule.tar.gz" ]]; then archive="${ARTIFACTS_DIR}/${capsule}.capsule.tar.gz" diff --git a/scripts/setup-source-home.sh b/scripts/setup-source-home.sh index 07ea16f6b..7a0fcce8c 100755 --- a/scripts/setup-source-home.sh +++ b/scripts/setup-source-home.sh @@ -616,7 +616,6 @@ APP_CAPSULES=( archive-manager inbox assistant - home-agent wallet wallet-metamask wallet-unisat @@ -634,6 +633,7 @@ APP_CAPSULES_JSON="$(printf '%s\n' "${APP_CAPSULES[@]}" | python3 -c 'import jso RETIRED_SOURCE_HOME_CAPSULES=( chat-wasm gba-engine-provider + home-agent ) RETIRED_SOURCE_HOME_PROVIDER_BINARIES=( diff --git a/state.md b/state.md index fd2ef3997..3a55abd53 100644 --- a/state.md +++ b/state.md @@ -13,7 +13,7 @@ changes. The proposed full old-URUX-tip merge is superseded by review of concret remaining behavior differences. The remaining protected-content stack retains its source order and feature-preservation gates in [the integration check](docs/audits/2026-09-11-integration-preservation.md). -Assistant consolidation, original UI parity and full cross-platform J3 acceptance remain open. +The approved Assistant consolidation now passes source and rendered preservation checks; its installed proof is next. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). The isolated Mac J3 installation runs Runtime `fbf1a4b0` with matching build, @@ -872,36 +872,23 @@ complete. Installed proof and the atomic authority cutover remain open. `base_path`, `allowed_paths`, `read_only`, `encryption_key`, and `extra` without weakening strict unknown-field handling. The zero-offer stdio Init test passes with the Runtime envelope in source tests. -- Assistant is a standalone first-party capsule with its own protected - workspace. Its tested Chat, Build, and Studio behavior uses typed model - offers and runs. Transcript copy goes only through the trusted Home - Clipboard path. -- Home Agent is the Home-owned Agent face. Home GUI owns the Shelf transition - to its composer, Agent Space, and `launchHomeTarget`. `home-agent` owns - sessions, transcript, composer, and settings. Runtime owns the protected, - revisioned workspace and model-proxy binding. -- Home Agent uses the typed `offers_list`, `runs_create`, `runs_events`, and - `runs_cancel` operations. Its Home message contract is pinned. The source has - one Shelf and one Home-owned Agent face, and the Agent room stays in the - Space ring. Fixes `450db538` and `900d7e5c` keep activation, message routing, - and saved layout upgrades on that canonical path without a duplicate generic - window. -- Runtime accepts one opaque, bounded workspace envelope for this local - contract. Typed document schemas remain future work when a cross-authority - operation needs them. -- Assistant model messages render a self-contained safe markdown subset with - escaped HTML, inert links, headings/lists/blockquotes/tables, fenced and - inline code, and inline/display math through vendored KaTeX 0.18.3. Focused - source proof lives in `scripts/assistant-shell-smoke.mjs`. The Home audit - records observed UI behavior separately; configured model-run and advanced - workflow acceptance remain open. -- Home Agent is the canonical Home-integrated Agent surface. Standalone - Assistant remains installed while the product inventories and tests its - distinct working behavior. It can remain as an explicitly scoped optional - app, or its useful behavior can move before removal. Future Home Agent tools, - Library reads, web search, Studio, Usage, and sampling controls require their - typed Runtime operations first. The zero-offer Home Agent state is installed - proof of honest absence, not configured inference. +- Assistant is the canonical first-party capsule for Sash’s Agent Space, + conversation UI and composer, with Chat, Build and Studio controls. Home GUI + owns shelf motion and the frame; Runtime owns protected workspace v2 and model + authority. Legacy Home Agent launches resolve to Assistant before token creation. +- Source installers retire the old Home Agent capsule tree while retaining its + protected workspace. Migration preserves all three old stores, full records, + editable drafts and exact run identities. Concurrent edits retain both versions. +- Typed model controls retain exact CID/offer selection. Studio saves request + identity before create and keeps per-session drafts, run cursors and output + history. Historical foreign runs and copied sessions retain their original + identities without acquiring authority. Copy uses trusted Home Clipboard. +- The source and rendered preservation gates pass. Installed canonical Assistant + and actual Qwen proof are pending on the frozen Mac candidate. Earlier installed + Home Agent and standalone Assistant observations above remain historical target + receipts. Advanced tools, Library reads, search and broader Studio capabilities + retain their typed-contract gates. + ## System Truth From 67a2acc4a79dceed28b3f763bb2dde52a9cf63f1 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 08:06:13 -0500 Subject: [PATCH 294/535] docs: record installed Assistant and Qwen checkpoint proof --- TASKS.md | 2 +- docs/audits/2026-09-10-execution-handover.md | 107 ++++++++---------- .../2026-09-11-assistant-convergence.md | 53 ++++++++- .../2026-09-11-integration-preservation.md | 3 +- state.md | 15 ++- 5 files changed, 108 insertions(+), 72 deletions(-) diff --git a/TASKS.md b/TASKS.md index 169fabf91..67fb85e73 100644 --- a/TASKS.md +++ b/TASKS.md @@ -44,7 +44,7 @@ The C1-C7 sequence in the approved plan controls integration and publication. | C1-delivery | Signed three-platform candidate installation | Functional preparation complete; final assembly after C5 freeze / delivery owner | Native inputs, bootstrap and publisher import have bounded source/artifact proof. Keep working builds for functional tests and rebuild only affected components. Complete atomic promotion, final installer stamping, platform regeneration and signed fresh installs at the final release boundary. | | J1 / C2 | Install, passkey, Recovery Kit, Profile and usable Home | Human passkey proof pending / Home and installer owners | Registration binding and atomic/conflict-safe identity persistence are integrated with 66 focused tests passing. Durable owner intent and Profile-inclusive recovery now pass source and UI checks. Real signed Mac installation and seed export pass. Source repairs for kit-first recovery, gateway shutdown and media-tool reuse are accepted. Both targets pass installed browser recovery/reload/retry, complete owned shutdown/restart, full Home setup/media reuse and Carrier cleanup/reuse. User-review account label, Recovery/Advanced spacing and cross-port session isolation are repaired on `1e320578`; shared-browser Mac/Linux proof passes. Next: AUTH-01 final user acceptance on the updated preserved Homes; concurrent-save and window/selection UI; W2-W5; rerun the smoke with System entry through the ElastOS menu, then complete the app matrix; full Linux/Mac staging. Publish J1 for review first under D1. | | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | -| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Next install the frozen candidate in the test-owned Mac Home, prove one entry plus an actual Qwen reply/reload, and provide a separate human Home. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | +| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The separate human Mac Home is ready for normal passkey setup with a verified locally cached Qwen package. Next obtain human acceptance there; Linux remains waiting on disk headroom and a reviewed startup profile. Cold Carrier delivery, Marketplace handoff and retention/removal keep their Required gates. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | | J4 / C4 | Browser works in the two agreed placements | Planned / Browser owner | Retain canonical B01-B16 on the Browser branch; Mac A/A/A and Linux Home/Exit with Mac Engine A/B/A, including reload. Full claimed-role proof remains C6. | | J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | Review PR60 then PR59 against final callers. The focused PR62 video preparation repair entered C1 after actual media dependency testing exposed its blocker. Full PR60/PR59 integration and J5 acceptance remain Required; extensions enter by readiness before freeze. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | diff --git a/docs/audits/2026-09-10-execution-handover.md b/docs/audits/2026-09-10-execution-handover.md index 9b5a31041..1ec0355d5 100644 --- a/docs/audits/2026-09-10-execution-handover.md +++ b/docs/audits/2026-09-10-execution-handover.md @@ -23,66 +23,53 @@ local operator ledger and current checkpoint. Resolve their directory with `git rev-parse --path-format=absolute --git-common-dir`; read `development-loop-current.md` there. A linked worktree's `.git` is a file. -Use `feat/0.7.1-website-execution`, based on the user-selected -`origin/upstream/0.7.1-dev` at `6c61c990`. Preserve the integration and Browser -donors. Before edits, recheck HEAD/tree, dirt, fetched divergence and target -receipts. Human preview Homes remain `1e320578`; the separate Mac J3 Home has -accepted Runtime `fbf1a4b0` Qwen proof. The approved one-Assistant source now passes -preservation and rendered checks. Install its frozen candidate next, as recorded -in [the convergence check](2026-09-11-assistant-convergence.md). The affected -Runtime build has passed; reuse providers, engine and model data. - -The installed checkpoint code is `1e320578b6b8a77753c4b5ef8fab51c12ea3d9e2`, tree -`44b08f00907c02dd3fe3bb2d3ed8e79e1bb76788`, 54 ahead / zero behind dev `6c61c990`. -Closeout documentation follows that candidate. The default checkout contains -unrelated dirty donor work; use the website checkout explicitly. - -One owner built the Runtime once per platform for the revised candidate and -reused unchanged verified artifacts. The old pinned assembly helper was reviewed -and left unused. Official installation receipts were refreshed after manifest -changes. Automated test gateways are stopped; only isolated human-test Homes -remain for AUTH-01. Existing user previews and public stage retain `d790a48e`. - -The full model donor `6972e165` is merged by `b318bfda`. Sash's shelf history -is reconciled by `37c82d4f` after full behavior comparison. Irzhy -foundation `617796a9` (PR60, which replaced PR57) is merged by `dd21d8bd`. Further -intake is frozen for one installed Qwen journey in a fresh test-owned Mac Home. Reuse the existing signed model package and engine. Prove -catalog visibility and normal cancel before a large transfer, then preparation, -selection, reply, stop and restart/reuse. Carrier cold delivery retains its -separate Required gate. Existing user Homes stay preserved. - -The [preservation check](2026-09-11-integration-preservation.md) accounts for all -13 Irzhy and Sash PRs created or updated in September. It records original -history, adopted behavior, explicit differences and remaining proof. Adapted -URUX/UIUX already entered through Irzhy reconstruction; compare remaining UI -differences against that baseline. Irzhy authority cutover and Required follow-up -work retain their dependency order. Keep both Assistant workspace stores intact; -one visible Assistant requires full history and draft preservation. Integrated -UI and full model acceptance remain pending. Installed Mac Runtime `fbf1a4b0` -now passes exact Qwen admission, cold startup, selection, an actual typed reply -and saved conversation. Reply, draft and exact CID/offer survive reload with zero -implicit dispatches. Stop returns an honest unknown outcome, as permitted by J3 -Step 5; confirmed backend cancellation remains separate. The native adapter -intentionally records unknown when stream closure cannot prove backend stop. -Owned shutdown closes all 29 processes including the engine, a held HTTP connection -and the listener. The same installed Home restarts, produces a second real reply -and restores the original draft. All six package files, protected files and prior -message hashes match; measured reuse receives zero Bitswap payload and makes zero -Content mutations. Independent review accepts this bounded Mac milestone. Current manifests and receipts bind all changed artifacts; -providers, engine and the admitted package are reused. The model menu repair `8f28b6e3` passes installed refresh, placement, normal close -and workspace-preservation checks. A stale copied archive identity was corrected -in source-install metadata; the bounded stamper repair prevents recurrence. -The original URUX full-merge proposal is superseded: the UIUX candidate -`8b547590` entered through reconstruction `7dd1780b` and Irzhy merge `985fdffc`, -followed by newer Sash dock/lock changes. Next, verify concrete remaining UI -differences and preserve all Assistant workspaces before consolidation. Preserve the -legacy `homeBrowserState.session.agent` object as well as both current workspace -stores. Repair concurrent saves and durable history limits on the combined clients. -The 79-conflict preview remains historical; no actual URUX merge was started. -Linux needs additional disk headroom and a reviewed startup profile. The private -checkpoint owns current processes and receipts; user Homes remain preserved. -Public staging needs exact-candidate review and approval. Final installer stamping -and signing follow C5 source freeze. +Use the explicit website checkout on `feat/0.7.1-website-execution`, based on +the user-selected `origin/upstream/0.7.1-dev` at `6c61c990`. The default checkout +contains unrelated dirty donor work. Preserve all 70 branches and 12 worktrees. +Before edits, recheck HEAD/tree, dirt, fetched divergence and process ownership. +Published review remains `00003b6f`; subsequent integration and Assistant work is +local. Closeout documentation follows the code candidate. + +The current installed Assistant candidate is +`fa297cb59a07971a472c6020586ea876fdc9a220`, tree +`4c1eaa05448ab2de64c685df4872f7a7d23b245d`. It contains Sash's UI in the canonical +Assistant capsule and protected v2 workspace adoption. The bounded Mac checkpoint +passes one identity, canonical and legacy launch, real Qwen reply/save/reload, +original draft/exact model preservation and full Runtime restart. Shutdown closed +29 owned processes and the held HTTP connection. Runtime, capsule and served +hashes match their current installation receipts. Providers, engine and the six +admitted package files are reused. See +[the convergence check](2026-09-11-assistant-convergence.md) for the acceptance +mapping and the distinction between installed data and three-store fixture proof. + +A separate fresh Mac human Home serves the same candidate and has a verified +locally cached Qwen package. Its normal passkey setup is ready. The private +checkpoint records its URL and receipt. Human enrollment and normal model +admission there require the user's presence. The earlier recovery human Homes +remain `1e320578`; existing user previews and public stage remain `d790a48e`. +The original recovery, retry, reload, media reuse and Carrier cleanup receipts +remain accepted. Full AUTH-01 human acceptance is still pending. + +The full model donor `6972e165` is merged by `b318bfda`. Sash shelf history is +reconciled by `37c82d4f`; Irzhy foundation `617796a9` (PR60 replacing PR57) is +merged by `dd21d8bd`. Adapted URUX/UIUX `8b547590` already entered through +reconstruction `7dd1780b` and merge `985fdffc`, followed by newer Sash dock/lock +work. The full old-tip merge proposal remains superseded. The +[preservation check](2026-09-11-integration-preservation.md) accounts for all +13 September contributor PRs and retains each remaining feature gate. + +This implementation checkpoint is closed at its bounded Mac result and waits +for human acceptance. Linux needs the recorded additional disk headroom and a +reviewed startup profile. Cold Carrier delivery, Marketplace handoff, +retention/removal, remaining UI differences, Irzhy authority cutover and follow-up, +dedicated provider-host/J5 proof and full J1–J5 retain their dependency order and +Required gates. Confirmed backend cancellation remains distinct from the accepted +honest unknown result. Recurring monitoring remains paused; Jetson is deferred. + +Further public deployment requires review and approval of the exact candidate. +The seed's existing GitHub credentials remain available for a future authorized +push. Required installation manifests and receipts are current; final installer +stamping, packaging and signing follow C5 source freeze and the release gate. ## Earlier preview evidence diff --git a/docs/audits/2026-09-11-assistant-convergence.md b/docs/audits/2026-09-11-assistant-convergence.md index d24c56753..eebbd59bd 100644 --- a/docs/audits/2026-09-11-assistant-convergence.md +++ b/docs/audits/2026-09-11-assistant-convergence.md @@ -10,13 +10,13 @@ change J1–J5 or D1–D6, publish another branch, or deploy the public Home. | Behavior | Source repair | Target and check | Current result | | --- | --- | --- | --- | -| One Assistant from Apps, shelf and legacy launch | Canonical manifest, launch normalization, catalog and installer retirement | Mac source/shell regression; installed Home | Source passes; installed pending | -| Complete old work and editable drafts | Protected workspace v2, three-store adoption, full records and source-scoped IDs | Seven Runtime tests, ten migration tests, rendered Mac fixture | Pass; installed pending | +| One Assistant from Apps, shelf and legacy launch | Canonical manifest, launch normalization, catalog and installer retirement | Mac source/shell regression; installed Home | Pass on Mac source and installed canonical/legacy launches | +| Complete old work and editable drafts | Protected workspace v2, three-store adoption, full records and source-scoped IDs | Seven Runtime tests, ten migration tests, rendered Mac fixture | All three stores pass Runtime/rendered tests; installed Home Agent adoption and preservation pass | | Preserve concurrent edits and failed saves | Revision checks, visible conflict copies, deferred merge during active runs | Rendered two-page conflict and Runtime race/failure tests | Pass | -| Exact model choice and reload | Existing CID/offer contract and per-session selection | Rendered Qwen choice and draft/reply restoration | Fixture passes; actual installed Qwen pending | +| Exact model choice and reload | Existing CID/offer contract and per-session selection | Rendered Qwen choice and draft/reply restoration | Pass: real Qwen reply, visible/saved text, reload and Runtime restart | | Real Chat, Build and Studio controls | Sash UI plus existing typed Studio controller; per-session state | Rendered modes and eight Studio preservation tests | Pass within offered capabilities | | Preserve run authority and unknown outcomes | Original actor/run/request identity, detached history copies, save before create | 26 cancellation/recovery tests; eight Studio tests | Pass | -| Safe installation and restart | Owned shutdown, current receipts and affected artifact parity | Test-owned Mac Home only | Pending | +| Safe installation and restart | Owned shutdown, current receipts and affected artifact parity | Test-owned Mac Home only | Pass: owned shutdown/restart and matching installed/served hashes | Migration retains immutable raw copies of the three original stores and leaves their protected objects in place. Unknown fields, complete text, roles, projects, @@ -30,6 +30,44 @@ run cursor and complete output history. Reload reads the same run; opening or switching modes creates no run. Chat and Studio keep separate drafts. An absent image/video offer produces the existing unavailable state. +## Installed Mac result + +Candidate `fa297cb59a07971a472c6020586ea876fdc9a220`, tree +`4c1eaa05448ab2de64c685df4872f7a7d23b245d`, passes the bounded installed +Assistant checkpoint. Runtime built and installed SHA-256 is +`fcc0b7d91667f2f291ac06728f9bb8936954775c8bcad1dcad16b2bf89ebc0fa`. +The receipt binds source, components, capsule files and current process identity. +All 118 affected artifact records match; browser files also match served bytes. + +Home summary and catalog show one Assistant. The old launch target resolves to +Assistant with the exact model query. Old bookmarks return to Home with their +launch fragment cleared. One explicit Qwen request produced `4`; provider output, +visible text and acknowledged saved text agree. The reply, original draft and +exact CID/offer survive browser reload and a full Runtime restart with zero +implicit creates. The native provider, engine and six admitted package files are +reused; package hashes, sizes, inodes and modification times remain unchanged. + +The first installed adoption captured the existing Home Agent workspace. The +final reply proof resumed canonical revision 6. All-three-store adoption and long +Unicode history are established by Runtime and rendered tests; they are separate +from that installed data set. Installed Studio proves mode selection and honest +unavailability of image/video offers. Its generation and foreign-run protections +retain source/test evidence. + +Shutdown closed 29 owned processes, the held HTTP connection and the listener. +Restart succeeded, proving release of the data-root lock. Identity, Profile and +old protected workspace bytes remain exact. The test intentionally saved new +canonical messages and Home window state; readback proves their contents. +A separate fresh Mac Home serves the same candidate for human passkey setup. Its +Qwen package was imported from the verified local closure with zero network +download; normal user enrollment and model admission remain to be exercised there. +Existing human previews remain unchanged. + +Independent review accepts this bounded Mac result. Linux convergence remains +pending on the recorded disk headroom and startup-profile requirements. Human +acceptance, cold Carrier delivery, Marketplace handoff, retention/removal and +full cross-platform J3 retain their Required gates. + ## Review and failure disposition Independent review covered the initial storage/import contract, source @@ -44,6 +82,13 @@ rendered Unicode failure came from the fixture decoding each HTTP chunk separately; whole-body UTF-8 decoding preserves exact text. A product sidebar path bypassed draft restoration and was routed through the shared selector. Explicit New Chat after an unknown run retains the unsent draft in both histories. +The first installed phrase test completed with a model refusal. Its prompt hash +and provider journal match the unchanged contract. A neutral arithmetic request +then passed the provider-to-UI-to-storage check. Early readiness and selector +failures, plus historical-run and ciphertext assertions in reused proof helpers, +were diagnosed before their corrected checks. These failures remain in private +receipts; they are not hidden by the passing final run. + Diagnostics and exact source/build/process receipts stay in the private operator checkpoint. The Mac Runtime build passes; providers, engine and model data are reused. Linux remains pending on its recorded capacity and startup-profile gates. diff --git a/docs/audits/2026-09-11-integration-preservation.md b/docs/audits/2026-09-11-integration-preservation.md index a5744f94f..ed611c885 100644 --- a/docs/audits/2026-09-11-integration-preservation.md +++ b/docs/audits/2026-09-11-integration-preservation.md @@ -389,5 +389,6 @@ and carries the existing typed Studio controller with it. The old Home Agent source remains in Git history. Protected legacy stores remain intact, and v2 adoption preserves all three stores before the installed app retires. [The convergence check](2026-09-11-assistant-convergence.md) records behavior, -review findings and the remaining installed gate. This is a named preservation +review findings and the passing bounded installed Mac gate. Human acceptance and +the remaining full J3 target gates stay pending. This is a named preservation repair on the integrated source, not another extraction of URUX history. diff --git a/state.md b/state.md index 3a55abd53..8a6ff999b 100644 --- a/state.md +++ b/state.md @@ -3,7 +3,7 @@ The September 11 integration now uses the full donor history. The combined source preserves current recovery, cookie and shutdown code while adding model admission, provider lifecycle, storage and window behavior. Focused source and rendered -fixtures pass; installed human Homes remain `1e320578`. Full model history is +fixtures pass; the earlier recovery human Homes remain `1e320578`. Full model history is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` retains current lifecycle fixes after a reproduced explicit Carrier bind defect was repaired. Its focused source checks pass; dedicated @@ -13,10 +13,10 @@ changes. The proposed full old-URUX-tip merge is superseded by review of concret remaining behavior differences. The remaining protected-content stack retains its source order and feature-preservation gates in [the integration check](docs/audits/2026-09-11-integration-preservation.md). -The approved Assistant consolidation now passes source and rendered preservation checks; its installed proof is next. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). +The approved Assistant consolidation `fa297cb5` now passes source, rendered and installed Mac checks: one identity, real Qwen reply/save/reload, original draft/model preservation and full Runtime restart. A separate fresh human Home serves the same candidate for passkey setup. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). -The isolated Mac J3 installation runs Runtime `fbf1a4b0` with matching build, +The earlier isolated Mac J3 receipt records Runtime `fbf1a4b0` with matching build, installed and served hashes and current manifests. Signed catalog visibility, Use/cancel/retry, exact local package admission, cold startup, exact model selection, an actual Qwen reply and its saved conversation pass. Reply and unsent @@ -884,9 +884,12 @@ complete. Installed proof and the atomic authority cutover remain open. history. Historical foreign runs and copied sessions retain their original identities without acquiring authority. Copy uses trusted Home Clipboard. - The source and rendered preservation gates pass. Installed canonical Assistant - and actual Qwen proof are pending on the frozen Mac candidate. Earlier installed - Home Agent and standalone Assistant observations above remain historical target - receipts. Advanced tools, Library reads, search and broader Studio capabilities + `fa297cb5` passes one catalog identity, legacy launch, a real Qwen reply, exact + saved/reloaded text and model selection, and owned shutdown/restart. Current + build/installed/served receipts match. The new human Home has fresh passkey + setup and a locally cached Qwen package; human enrollment/admission remains + pending. Earlier Home Agent and standalone Assistant observations above are + historical target receipts. Advanced tools, Library reads, search and broader Studio capabilities retain their typed-contract gates. From 822c4e3d0364540ad9dbe6870acc6aed7b087fb0 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 08:26:58 -0500 Subject: [PATCH 295/535] fix(setup): reject stale archive metadata for source capsules --- scripts/install-source-home-runtime-smoke.py | 111 +++++++++++++++++++ scripts/install-source-home-runtime.py | 92 ++++++++++++++- 2 files changed, 201 insertions(+), 2 deletions(-) diff --git a/scripts/install-source-home-runtime-smoke.py b/scripts/install-source-home-runtime-smoke.py index 58d9d0551..a460c19a1 100755 --- a/scripts/install-source-home-runtime-smoke.py +++ b/scripts/install-source-home-runtime-smoke.py @@ -282,6 +282,116 @@ def corrupt_stage(runtime, parent): raise AssertionError("mismatch failure left a staged file") +def test_source_capsule_archive_metadata(temp_root): + installer = load_installer() + source = temp_root / "metadata-source" + data = temp_root / "metadata-installed" + built = temp_root / "metadata-build/elastos" + initialize_source(source) + initialize_data(data) + write(built, b"runtime-before-metadata-check\n", 0o700) + invoke(source, data, built) + write(built, b"runtime-after-metadata-check\n", 0o700) + components = data / "components.json" + archive = { + "cid": "bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi", + "checksum": "sha256:" + "a" * 64, + "size": 123, + "release_path": "capsules/system.tar.gz", + "strategy": "download", + } + source_record = {"strategy": "source-build", "install_path": "capsules/system"} + + def manifest(mappings, *, genuine_archive=False): + return { + "capsules": {"system": { + "cid": archive["cid"] if genuine_archive else "", + "sha256": "a" * 64 if genuine_archive else "", + "entrypoint_sha256": "b" * 64, + "install_path": "capsules/system", + }}, + "external": { + "system": {"install_path": "capsules/system", "platforms": mappings}, + # Unrelated native release metadata remains outside this check. + "native-provider": {"platforms": {"*": archive}}, + }, + } + + def check_case(value, platform="linux-amd64", failure=None): + payload = value if isinstance(value, bytes) else json.dumps(value).encode() + write(components, payload) + before = snapshot(data) + args = SimpleNamespace(source_root=source, data_dir=data, built_runtime=built, platform=platform) + staged = [] + + def track_stage(runtime, parent): + staged.append(True) + return installer.stage_runtime(runtime, parent) + + try: + if installer.PLATFORM_RE.fullmatch(platform): + receipt = installer.install(args, runtime_stager=track_stage) + else: + # The installer CLI accepts canonical platform spellings. Exercise + # the reverse underscore alias at its metadata-selection boundary. + receipt = {"components_sha256": installer.source_components_hash(components, platform)} + except installer.InstallError as error: + if str(error) != failure: + raise AssertionError(f"unexpected metadata failure: {error}") from error + if staged or snapshot(data) != before: + raise AssertionError("metadata rejection staged or changed an installed artifact") + else: + if failure: + raise AssertionError(f"installer accepted invalid metadata: {failure}") + if receipt["components_sha256"] != sha256(components): + raise AssertionError("validated components bytes differ from receipt hash") + + # Reproduces the source System/Marketplace launch failure before replacing + # either the previous Runtime or its canonical success receipt. + check_case(manifest({"*": archive}), failure="source_capsule_archive_identity") + check_case(manifest({"linux-amd64": source_record, "*": archive, "darwin-arm64": archive})) + check_case(manifest({"*": archive}, genuine_archive=True)) + for platform, alias in installer.PLATFORM_ALIASES.items(): + check_case(manifest({alias: archive, "*": source_record}), platform, "source_capsule_archive_identity") + check_case(manifest({alias: source_record, "*": archive}), platform) + check_case(manifest({platform: {}, alias: archive, "*": archive}), platform) + check_case(manifest({"linux-amd64": {"install_path": None}, "*": archive})) + for selected_path in ("", "capsules/other"): + check_case(manifest({"linux-amd64": {"install_path": selected_path}}), failure="source_capsule_install_path") + for field, value in (("cid", archive["cid"]), ("checksum", archive["checksum"]), ("size", 1), ("url", "https://example.invalid/archive"), ("release_path", "archive.tar")): + check_case(manifest({"*": {**source_record, field: value}}), failure="source_capsule_archive_identity") + check_case(manifest({"linux-amd64": []}), failure="components_shape") + check_case(manifest({"*": {"size": True}}), failure="components_shape") + for malformed in (b"{", b"[]", b'{"capsules":null}', b'{"capsules":{},"capsules":{}}', b'{"number":NaN}', b'\xff'): + check_case(malformed, failure="components_shape" if malformed in (b"[]", b'{"capsules":null}') else "components_json") + check_case(b" " * (installer.MAX_COMPONENTS_BYTES + 1), failure="unsafe_artifact") + + # The exact bytes validated above also own the receipt hash. A metadata + # change during Runtime staging must stop before receipt invalidation. + write(components, json.dumps(manifest({"*": source_record})).encode()) + runtime_before = snapshot(data)["bin/elastos"] + receipt_before = snapshot(data)[RECEIPT.as_posix()] + + def change_metadata(runtime, parent): + staged = installer.stage_runtime(runtime, parent) + write(components, json.dumps(manifest({"*": archive})).encode()) + return staged + + args = SimpleNamespace(source_root=source, data_dir=data, built_runtime=built, platform="linux-amd64") + try: + installer.install(args, runtime_stager=change_metadata) + except installer.InstallError as error: + if str(error) != "components_changed": + raise AssertionError(f"unexpected changed-metadata failure: {error}") from error + else: + raise AssertionError("installer accepted metadata changed after validation") + after = snapshot(data) + if after["bin/elastos"] != runtime_before or after[RECEIPT.as_posix()] != receipt_before: + raise AssertionError("changed metadata replaced the Runtime or receipt") + if any(path.name.startswith(".source-home-install.") for path in data.rglob("*")): + raise AssertionError("changed metadata left a staged artifact") + + def test_post_runtime_receipt_failure(temp_root): installer = load_installer() source = temp_root / "publication-source" @@ -605,6 +715,7 @@ def main(): fixture.mkdir(mode=0o700) test_good_dirty_and_rerun(fixture) test_installed_mismatch(fixture) + test_source_capsule_archive_metadata(fixture) test_post_runtime_receipt_failure(fixture) test_runtime_publication_restores_pair(fixture) test_unsafe_destinations(fixture) diff --git a/scripts/install-source-home-runtime.py b/scripts/install-source-home-runtime.py index a51966552..eb2b6ab36 100755 --- a/scripts/install-source-home-runtime.py +++ b/scripts/install-source-home-runtime.py @@ -21,6 +21,15 @@ MAX_RECEIPT_BYTES = 16 * 1024 GIT_ID_RE = re.compile(r"^[0-9a-f]{40,64}$") PLATFORM_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,31}$") +# Keep selection aligned with Runtime setup.rs and the source capsule stamper. +PLATFORM_ALIASES = { + "x86_64-linux": "linux-amd64", + "linux-amd64": "x86_64-linux", + "aarch64-linux": "linux-arm64", + "linux-arm64": "aarch64-linux", + "aarch64-darwin": "darwin-arm64", + "darwin-arm64": "aarch64-darwin", +} class InstallError(Exception): @@ -118,7 +127,7 @@ def open_regular(path, limit, *, executable=False, trusted_runtime=False): fail("unsafe_artifact") -def hash_descriptor(descriptor, metadata, limit): +def hash_descriptor(descriptor, metadata, limit, *, payload=None): digest = hashlib.sha256() total = 0 while True: @@ -129,6 +138,8 @@ def hash_descriptor(descriptor, metadata, limit): if total > limit: fail("artifact_oversize") digest.update(chunk) + if payload is not None: + payload.extend(chunk) final = os.fstat(descriptor) if ( total != metadata.st_size @@ -149,6 +160,81 @@ def hash_file(path, limit, *, executable=False, trusted_runtime=False): os.close(descriptor) +def unique_json_object(pairs): + result = {} + for key, value in pairs: + if key in result: + fail("components_json") + result[key] = value + return result + + +def validate_source_components(manifest, platform): + if not isinstance(manifest, dict): + fail("components_shape") + capsules = manifest.get("capsules", {}) + external = manifest.get("external", {}) + if not isinstance(capsules, dict) or not isinstance(external, dict): + fail("components_shape") + for name, entry in capsules.items(): + if not isinstance(entry, dict): + fail("components_shape") + # A source stamp binds the installed entrypoint and clears archive identity. + # Genuine archive entries may also carry entrypoint hashes. + for field in ("cid", "sha256", "entrypoint_sha256"): + if field in entry and not isinstance(entry[field], str): + fail("components_shape") + if not entry.get("entrypoint_sha256") or entry.get("cid", "").strip() or entry.get("sha256", "").strip(): + continue + component = external.get(name) + if component is None: + continue + if not isinstance(component, dict): + fail("components_shape") + mappings = component.get("platforms", {}) + if not isinstance(mappings, dict) or any(not isinstance(value, dict) for value in mappings.values()): + fail("components_shape") + # Presence, including an empty exact record, wins over alias/wildcard. + selected = next((mappings[key] for key in ( + platform, PLATFORM_ALIASES.get(platform), "*", + ) if key in mappings), {}) + for field in ("cid", "checksum", "url", "release_path"): + value = selected.get(field) + if value is not None and not isinstance(value, str): + fail("components_shape") + if value and value.strip(): + fail("source_capsule_archive_identity") + size = selected.get("size") + if size is not None and (type(size) is not int or size < 0): + fail("components_shape") + if size: + fail("source_capsule_archive_identity") + install_path = selected.get("install_path") + if install_path is None: + install_path = component.get("install_path") + if install_path != entry.get("install_path") or install_path != f"capsules/{name}": + fail("source_capsule_install_path") + + +def source_components_hash(path, platform): + descriptor, metadata = open_regular(path, MAX_COMPONENTS_BYTES) + payload = bytearray() + try: + digest = hash_descriptor(descriptor, metadata, MAX_COMPONENTS_BYTES, payload=payload) + finally: + os.close(descriptor) + try: + manifest = json.loads( + payload, + object_pairs_hook=unique_json_object, + parse_constant=lambda _value: fail("components_json"), + ) + except (ValueError, UnicodeError, RecursionError): + fail("components_json") + validate_source_components(manifest, platform) + return digest + + def git_value(source_root, revision): env = os.environ.copy() env["GIT_OPTIONAL_LOCKS"] = "0" @@ -316,7 +402,7 @@ def install( require_safe_destination(installed_runtime, 0o700) require_safe_destination(receipt_path, 0o600) - components_hash = hash_file(data_dir / "components.json", MAX_COMPONENTS_BYTES) + components_hash = source_components_hash(data_dir / "components.json", args.platform) capsules_hash = hash_file( receipts_dir / CAPSULE_RECEIPT_NAME, MAX_CAPSULE_RECEIPT_BYTES ) @@ -332,6 +418,8 @@ def install( runtime_stage, staged_hash = runtime_stager(built_runtime, bin_dir) if staged_hash != built_hash: fail("built_runtime_mismatch") + if hash_file(data_dir / "components.json", MAX_COMPONENTS_BYTES) != components_hash: + fail("components_changed") receipt = { "schema": SCHEMA, "source": source, From 84825f9e11a0616f0273b483506032c2769d0fe8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 08:33:52 -0500 Subject: [PATCH 296/535] docs: record System and Marketplace human launch repair --- TASKS.md | 2 +- docs/audits/2026-09-10-execution-handover.md | 8 ++++--- .../2026-09-11-assistant-convergence.md | 23 +++++++++++++++++++ state.md | 8 +++---- 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/TASKS.md b/TASKS.md index 67fb85e73..cd45e2886 100644 --- a/TASKS.md +++ b/TASKS.md @@ -44,7 +44,7 @@ The C1-C7 sequence in the approved plan controls integration and publication. | C1-delivery | Signed three-platform candidate installation | Functional preparation complete; final assembly after C5 freeze / delivery owner | Native inputs, bootstrap and publisher import have bounded source/artifact proof. Keep working builds for functional tests and rebuild only affected components. Complete atomic promotion, final installer stamping, platform regeneration and signed fresh installs at the final release boundary. | | J1 / C2 | Install, passkey, Recovery Kit, Profile and usable Home | Human passkey proof pending / Home and installer owners | Registration binding and atomic/conflict-safe identity persistence are integrated with 66 focused tests passing. Durable owner intent and Profile-inclusive recovery now pass source and UI checks. Real signed Mac installation and seed export pass. Source repairs for kit-first recovery, gateway shutdown and media-tool reuse are accepted. Both targets pass installed browser recovery/reload/retry, complete owned shutdown/restart, full Home setup/media reuse and Carrier cleanup/reuse. User-review account label, Recovery/Advanced spacing and cross-port session isolation are repaired on `1e320578`; shared-browser Mac/Linux proof passes. Next: AUTH-01 final user acceptance on the updated preserved Homes; concurrent-save and window/selection UI; W2-W5; rerun the smoke with System entry through the ElastOS menu, then complete the app matrix; full Linux/Mac staging. Publish J1 for review first under D1. | | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | -| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The separate human Mac Home is ready for normal passkey setup with a verified locally cached Qwen package. Next obtain human acceptance there; Linux remains waiting on disk headroom and a reviewed startup profile. Cold Carrier delivery, Marketplace handoff and retention/removal keep their Required gates. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | +| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Next obtain human model admission and reply acceptance there; Linux remains waiting on disk headroom and a reviewed startup profile. Cold Carrier delivery, Marketplace handoff and retention/removal keep their Required gates. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | | J4 / C4 | Browser works in the two agreed placements | Planned / Browser owner | Retain canonical B01-B16 on the Browser branch; Mac A/A/A and Linux Home/Exit with Mac Engine A/B/A, including reload. Full claimed-role proof remains C6. | | J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | Review PR60 then PR59 against final callers. The focused PR62 video preparation repair entered C1 after actual media dependency testing exposed its blocker. Full PR60/PR59 integration and J5 acceptance remain Required; extensions enter by readiness before freeze. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | diff --git a/docs/audits/2026-09-10-execution-handover.md b/docs/audits/2026-09-10-execution-handover.md index 1ec0355d5..50353a79c 100644 --- a/docs/audits/2026-09-10-execution-handover.md +++ b/docs/audits/2026-09-10-execution-handover.md @@ -43,9 +43,11 @@ admitted package files are reused. See mapping and the distinction between installed data and three-store fixture proof. A separate fresh Mac human Home serves the same candidate and has a verified -locally cached Qwen package. Its normal passkey setup is ready. The private -checkpoint records its URL and receipt. Human enrollment and normal model -admission there require the user's presence. The earlier recovery human Homes +locally cached Qwen package. The user has enrolled a passkey account. The private +checkpoint records its URL and receipt. Human review found stale release metadata +for System and Marketplace; installer guard `822c4e3d` and corrected metadata now +pass normal launches in the actual browser. Account and Profile data remain +intact. Human Qwen admission and reply acceptance are next. The earlier recovery human Homes remain `1e320578`; existing user previews and public stage remain `d790a48e`. The original recovery, retry, reload, media reuse and Carrier cleanup receipts remain accepted. Full AUTH-01 human acceptance is still pending. diff --git a/docs/audits/2026-09-11-assistant-convergence.md b/docs/audits/2026-09-11-assistant-convergence.md index eebbd59bd..c20abb9fb 100644 --- a/docs/audits/2026-09-11-assistant-convergence.md +++ b/docs/audits/2026-09-11-assistant-convergence.md @@ -96,3 +96,26 @@ reused. Linux remains pending on its recorded capacity and startup-profile gates Final packaging/signing follows source freeze and the release gate. Human recovery acceptance, cold Carrier delivery, remaining contributor integration, Browser and protected-content gates retain their existing status. + +## Human follow-up: System and Marketplace launch + +Human review found a real installation gap after passkey enrollment. System +launch returned `502`: its source entrypoint record retained an external release +archive identity without the matching registry archive hash. Marketplace had the +same mismatch. Their source and installed app files already matched. + +Installer guard `822c4e3d` now checks source capsule metadata before staging the +Runtime or replacing the success receipt. It applies Runtime's platform selection +order, permits valid source and archive entries, and rejects conflicts before +mutation. Regression cases cover stale identity, platform aliases, install paths, +malformed input and metadata changes during staging; the installer smoke passes. + +Both Mac J3 Homes now have corrected metadata and fresh canonical receipts. The +Runtime and capsule bytes remain those of the accepted candidate; all 147 affected +artifact records match source, built, installed and served bytes where applicable. +Owned restart preserves the enrolled human account, Profile and app data. System +and Marketplace open through normal dock controls in the test Home and Anders's +actual browser. System shows the current passkey account; Marketplace Models +shows the verified Qwen package with its Use action. Full human Qwen admission +and reply acceptance remain pending. Existing recovery Homes and Linux are +unchanged. The original failure receipt remains available in the private checkpoint. diff --git a/state.md b/state.md index 8a6ff999b..fec3d094a 100644 --- a/state.md +++ b/state.md @@ -13,7 +13,7 @@ changes. The proposed full old-URUX-tip merge is superseded by review of concret remaining behavior differences. The remaining protected-content stack retains its source order and feature-preservation gates in [the integration check](docs/audits/2026-09-11-integration-preservation.md). -The approved Assistant consolidation `fa297cb5` now passes source, rendered and installed Mac checks: one identity, real Qwen reply/save/reload, original draft/model preservation and full Runtime restart. A separate fresh human Home serves the same candidate for passkey setup. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). +The approved Assistant consolidation `fa297cb5` now passes source, rendered and installed Mac checks: one identity, real Qwen reply/save/reload, original draft/model preservation and full Runtime restart. The separate human Home now has an enrolled passkey account. Human review exposed stale System/Marketplace archive metadata; guard `822c4e3d` and corrected installation receipts close that launch gap. Both apps open in the actual user browser, and Marketplace shows Qwen with Use available. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). The earlier isolated Mac J3 receipt records Runtime `fbf1a4b0` with matching build, @@ -886,9 +886,9 @@ complete. Installed proof and the atomic authority cutover remain open. - The source and rendered preservation gates pass. Installed canonical Assistant `fa297cb5` passes one catalog identity, legacy launch, a real Qwen reply, exact saved/reloaded text and model selection, and owned shutdown/restart. Current - build/installed/served receipts match. The new human Home has fresh passkey - setup and a locally cached Qwen package; human enrollment/admission remains - pending. Earlier Home Agent and standalone Assistant observations above are + build/installed/served receipts match. The new human Home has an enrolled passkey account and a locally cached + Qwen package. System/Marketplace metadata repair `822c4e3d` passes actual + launch; human model admission and reply acceptance remain pending. Earlier Home Agent and standalone Assistant observations above are historical target receipts. Advanced tools, Library reads, search and broader Studio capabilities retain their typed-contract gates. From d3d757ce53e06a516ec9abe499fa7b0364ab5f17 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 10:38:35 -0500 Subject: [PATCH 297/535] fix(assistant): show distinct activity and reasoning in disclosures --- capsules/assistant/browser/agent-harness.css | 11 ++ capsules/assistant/browser/agent-stream.js | 74 +++++----- scripts/assistant-progress-browser-smoke.mjs | 134 +++++++++++++++++++ 3 files changed, 188 insertions(+), 31 deletions(-) create mode 100644 scripts/assistant-progress-browser-smoke.mjs diff --git a/capsules/assistant/browser/agent-harness.css b/capsules/assistant/browser/agent-harness.css index 7a5f8786d..20e6be40a 100644 --- a/capsules/assistant/browser/agent-harness.css +++ b/capsules/assistant/browser/agent-harness.css @@ -1597,6 +1597,17 @@ body.agent-harness-sidebar-collapsed #agent-harness-search-open-main { -webkit-text-fill-color: currentColor; } +/* A lone current activity is a status line, with no empty disclosure. */ +.agent-progress[data-disclosure="false"] .agent-progress-summary { + cursor: default; +} + +.agent-progress[data-disclosure="false"] .agent-thinking-chevron, +.agent-progress[data-disclosure="false"] .agent-progress-trace, +.agent-progress-milestones[hidden] { + display: none; +} + .agent-progress-trace { min-width: 0; } diff --git a/capsules/assistant/browser/agent-stream.js b/capsules/assistant/browser/agent-stream.js index a414bf695..b74290718 100644 --- a/capsules/assistant/browser/agent-stream.js +++ b/capsules/assistant/browser/agent-stream.js @@ -955,7 +955,9 @@ export function appendProgressBlock() { details.hidden = true; const summary = document.createElement("summary"); summary.className = "agent-thinking-summary agent-progress-summary"; - summary.title = "Show activity"; + summary.addEventListener("click", (event) => { + if (details.dataset.disclosure !== "true") event.preventDefault(); + }); const chevron = document.createElement("span"); chevron.className = "agent-thinking-chevron"; chevron.setAttribute("aria-hidden", "true"); @@ -1013,7 +1015,7 @@ function appendProgressItem(list, { className, glyph, text }) { list.append(li); } -function paintProgressView(el, progress, { reasoning = "", reasoningVisible = false } = {}) { +export function paintProgressView(el, progress, { reasoning = "", reasoningVisible = false } = {}) { if (!el || !progress) { return; } @@ -1032,35 +1034,50 @@ function paintProgressView(el, progress, { reasoning = "", reasoningVisible = fa if (label && label.textContent !== nextLabel) { label.textContent = nextLabel; } + // The summary owns the current activity; the disclosure owns distinct history. + const milestones = (progress.milestones || []).filter( + (milestone) => milestone.text?.trim() && milestone.text.trim() !== nextLabel.trim(), + ); + const showRaw = Boolean(reasoningVisible && reasoning.trim()); + const hasDisclosure = milestones.length > 0 || showRaw; + el.dataset.disclosure = String(hasDisclosure); + if (!hasDisclosure) el.open = false; + const summary = el.querySelector("summary"); + if (summary) { + summary.tabIndex = hasDisclosure ? 0 : -1; + if (hasDisclosure) { + summary.removeAttribute("aria-disabled"); + summary.title = showRaw ? "Show activity and reasoning" : "Show completed activity"; + } else { + summary.setAttribute("aria-disabled", "true"); + summary.removeAttribute("title"); + } + } const list = el.querySelector(".agent-progress-milestones"); if (list) { - const currentBit = live && progress.current ? `|${progress.current.key}:${progress.current.text}` : ""; - const keys = `${(progress.milestones || []).map((m) => m.key).join("|")}${currentBit}`; + const keys = JSON.stringify(milestones); if (list.dataset.keys !== keys) { list.dataset.keys = keys; list.replaceChildren(); - for (const m of progress.milestones || []) { + for (const m of milestones) { appendProgressItem(list, { className: m.kind === "finding" ? "is-finding" : "is-done", glyph: progressGlyph(m.phase, m.kind), text: m.text, }); } - if (live && progress.current && progress.current.kind !== "finding") { - appendProgressItem(list, { - className: "is-current", - glyph: progressGlyph(progress.current.phase, progress.current.kind), - text: progress.current.text, - }); - } } + list.hidden = milestones.length === 0; } const raw = el.querySelector(".agent-progress-raw"); if (raw) { - const showRaw = Boolean(reasoningVisible && reasoning); raw.hidden = !showRaw; if (showRaw) { - raw.dataset.mdSource = reasoning; + // Actual model reasoning stays plain text, including during streaming. + if (raw.dataset.mdSource !== reasoning) { + raw.dataset.mdSource = reasoning; + raw.textContent = reasoning; + } } } } @@ -1755,7 +1772,6 @@ async function startLiveTurnForPrompt(userText, { resumeTurn = null } = {}) { let sawFirstToken = false; let stickToBottom = true; let answerNotifiedAt = 0; - let reasoningRendered = false; let hiddenBuffered = false; let progressEl = null; let revealTimer = 0; @@ -1808,24 +1824,11 @@ async function startLiveTurnForPrompt(userText, { resumeTurn = null } = {}) { } if (!progressEl && next.revealed) { progressEl = appendProgressBlock(); - progressEl?.addEventListener("toggle", () => { - pauseFollow(); - if (!progressEl.open || reasoningRendered || !state.complete) { - return; - } - reasoningRendered = true; - const raw = progressEl.querySelector(".agent-progress-raw"); - if (raw && ctx.reasoningVisible) { - const thinking = getReasoning(); - raw.textContent = thinking; - raw.dataset.mdSource = thinking; - raw.hidden = false; - } - }); + progressEl?.addEventListener("toggle", () => pauseFollow()); } paintProgressView(progressEl, next, { - reasoning: "", - reasoningVisible: false, + reasoning: ctx.reasoningVisible ? getReasoning() : "", + reasoningVisible: ctx.reasoningVisible, }); }, }); @@ -2380,6 +2383,11 @@ async function startLiveTurnForPrompt(userText, { resumeTurn = null } = {}) { progress.dispatch({ type: "REVEAL" }); }, }); + // A reasoning delta need not change the current progress phase. + paintProgressView(progressEl, progress.getState(), { + reasoning: ctx.reasoningVisible ? getReasoning() : "", + reasoningVisible: ctx.reasoningVisible, + }); }, 0); } } @@ -2478,6 +2486,10 @@ async function startLiveTurnForPrompt(userText, { resumeTurn = null } = {}) { progress.dispatch({ type: stoppedEarly ? "GENERATION_STOPPED" : "GENERATION_DONE", }); + paintProgressView(progressEl, progress.getState(), { + reasoning: thinking, + reasoningVisible: ctx.reasoningVisible, + }); const progressSnap = snapshotProgress(progress.getState()); if (progressEl && !progressSnap?.milestones?.length && !thinking) { progressEl.remove(); diff --git a/scripts/assistant-progress-browser-smoke.mjs b/scripts/assistant-progress-browser-smoke.mjs new file mode 100644 index 000000000..d4d5bd16a --- /dev/null +++ b/scripts/assistant-progress-browser-smoke.mjs @@ -0,0 +1,134 @@ +#!/usr/bin/env node +import assert from "node:assert/strict"; +import { createServer } from "node:http"; +import { once } from "node:events"; +import { readFile, mkdtemp, mkdir, writeFile } from "node:fs/promises"; +import { resolve, extname, join } from "node:path"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; +import { chromium, brave } from "./system-uiux-fixture.mjs"; + +const root = fileURLToPath(new URL("../", import.meta.url)); +const errors = []; +const server = createServer(async (req, res) => { + try { + const pathname = new URL(req.url, "http://fixture").pathname; + if (pathname === "/favicon.ico") { res.writeHead(204); res.end(); return; } + if (pathname === "/") { + res.setHeader("content-type", "text/html; charset=utf-8"); + res.end(` + +
`); + return; + } + const match = pathname.match(/^\/apps\/([^/]+)\/(.+)$/); + assert.ok(match, `Unexpected fixture request: ${pathname}`); + const capsuleRoot = resolve(root, "capsules", match[1], "browser"); + const path = resolve(capsuleRoot, match[2]); + assert.ok(path.startsWith(`${capsuleRoot}/`)); + const bytes = await readFile(path); + res.setHeader("content-type", ({ ".js": "text/javascript", ".mjs": "text/javascript", ".css": "text/css" })[extname(path)] || "application/octet-stream"); + res.end(bytes); + } catch (error) { + errors.push(String(error)); + res.writeHead(500); res.end("Fixture asset error"); + } +}); +server.listen(0, "127.0.0.1"); +await once(server, "listening"); +let browser, page; +try { + browser = await chromium.launch({ executablePath: brave, headless: true }); + page = await browser.newPage({ viewport: { width: 900, height: 700 } }); + page.on("pageerror", error => errors.push(String(error))); + await page.goto(`http://127.0.0.1:${server.address().port}/`); + await page.waitForFunction(() => window.fixtureReady === true); + const block = page.locator('.agent-progress'); + const summary = block.locator('summary'); + const rows = block.locator('.agent-progress-milestones li'); + const raw = block.locator('.agent-progress-raw'); + const label = "Analyzing the findings…"; + const current = { key: "current", text: label, phase: "analyzing" }; + let progress = { revealed: true, phase: "analyzing", currentText: label, current, milestones: [] }; + const paint = (options = {}) => page.evaluate(({ progress, options }) => window.paint(progress, options), { progress, options }); + const mouseClickSummary = async () => { + const box = await summary.boundingBox(); + assert.ok(box, "Current status is rendered"); + await page.mouse.click(box.x + box.width / 2, box.y + box.height / 2); + }; + + await paint(); + assert.equal(await summary.innerText(), label); + assert.equal(await rows.count(), 0, "Current activity is not repeated in the trace"); + assert.equal(await summary.getAttribute('aria-disabled'), 'true'); + assert.equal(await block.locator('.agent-thinking-chevron').isVisible(), false); + await mouseClickSummary(); + assert.equal(await block.evaluate(el => el.open), false, "A status alone has no disclosure"); + assert.equal(await raw.isVisible(), false, "Generic activity is not presented as reasoning"); + + // An identical historical label also gives the disclosure nothing new to show. + progress.milestones = [{ key: "same", text: label, phase: "analyzing" }]; + await paint(); + assert.equal(await rows.count(), 0); + assert.equal(await block.getAttribute('data-disclosure'), 'false'); + + progress.milestones.push({ key: "read", text: "Read the source file", phase: "reading" }); + await paint(); + await summary.click(); + assert.equal(await block.evaluate(el => el.open), true); + assert.deepEqual(await rows.allTextContents(), ["Read the source file"]); + assert.equal((await block.innerText()).split(label).length - 1, 1, "Expanded activity shows the current label once"); + // A corrected milestone with the same key must repaint, without losing open state. + progress.milestones[1].text = "Read both source files"; + await paint(); + assert.deepEqual(await rows.allTextContents(), ["Read both source files"]); + assert.equal(await block.evaluate(el => el.open), true); + + progress.milestones = []; + const reasoning = 'Compare α and β.\n 🧪'; + await paint({ reasoning, reasoningVisible: true }); + assert.equal(await block.getAttribute('data-disclosure'), 'true'); + assert.equal(await raw.isVisible(), true); + assert.equal(await raw.textContent(), reasoning); + assert.equal(await raw.locator('script').count(), 0, "Model reasoning stays plain text"); + await paint({ reasoning: `${reasoning}\nThen check the result.`, reasoningVisible: true }); + assert.equal(await raw.textContent(), `${reasoning}\nThen check the result.`, "Further reasoning deltas repaint at the same phase"); + await paint({ reasoning, reasoningVisible: false }); + assert.equal(await block.evaluate(el => el.open), false); + assert.equal(await raw.isVisible(), false); + assert.equal(await block.getAttribute('data-disclosure'), 'false', "Hidden reasoning cannot enable an empty disclosure"); + await paint({ reasoning: ' \n ', reasoningVisible: true }); + assert.equal(await block.getAttribute('data-disclosure'), 'false'); + + for (const phase of ['done', 'stopped', 'error']) { + progress = { revealed: true, phase, secondary: true, current: null, currentText: phase === 'done' ? '✓ 2 steps' : phase, + milestones: [{ key: 'read', text: 'Read both source files', phase: 'reading' }, + { key: 'finding', text: 'The result matches', kind: 'finding', phase: 'verifying' }] }; + await paint(); + if (!(await block.evaluate(el => el.open))) await summary.click(); + assert.deepEqual(await rows.allTextContents(), ['Read both source files', 'The result matches']); + assert.equal(await block.locator('.is-finding').count(), 1); + assert.equal(await block.evaluate(el => el.classList.contains('is-streaming')), false); + assert.equal(await block.evaluate(el => el.classList.contains('is-complete')), true); + assert.equal(await summary.innerText(), progress.currentText); + } + assert.deepEqual(errors, []); + console.log('PASS Assistant rendered progress: single status, distinct history, raw reasoning, terminal milestones'); +} catch (error) { + const proofDir = process.env.ASSISTANT_PROGRESS_PROOF_DIR || await mkdtemp(join(tmpdir(), 'assistant-progress-failure-')); + await mkdir(proofDir, { recursive: true }); + await writeFile(join(proofDir, 'report.json'), JSON.stringify({ error: String(error), errors }, null, 2)); + if (page) await page.screenshot({ path: join(proofDir, 'first-failure.png'), fullPage: true }).catch(() => {}); + console.error(`Assistant progress failure evidence: ${proofDir}`); + throw error; +} finally { + await browser?.close(); + server.close(); + await once(server, 'close'); +} From 3c2f9a80012ff510449998f0b423a21c47ad2a98 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 10:38:35 -0500 Subject: [PATCH 298/535] fix(models): retain pending content and use capsule marketplace presentation --- capsules/marketplace/browser/index.html | 5 - capsules/marketplace/browser/marketplace.css | 5 + capsules/marketplace/browser/marketplace.js | 92 ++++-- .../marketplace/browser/model-management.css | 19 +- .../marketplace/browser/model-management.js | 59 ++-- capsules/system/browser/model-management.css | 19 +- capsules/system/browser/model-management.js | 59 ++-- .../src/api/capsule_inventory/preparation.rs | 275 ++++++++++++++++-- scripts/home-entropy-check.mjs | 4 +- scripts/model-management-browser-smoke.mjs | 120 +++++++- 10 files changed, 541 insertions(+), 116 deletions(-) diff --git a/capsules/marketplace/browser/index.html b/capsules/marketplace/browser/index.html index 94380d47c..414019912 100644 --- a/capsules/marketplace/browser/index.html +++ b/capsules/marketplace/browser/index.html @@ -64,10 +64,6 @@ Media - @@ -79,7 +75,6 @@

Discover

- diff --git a/capsules/marketplace/browser/marketplace.css b/capsules/marketplace/browser/marketplace.css index 762212f98..a0a4ba72d 100644 --- a/capsules/marketplace/browser/marketplace.css +++ b/capsules/marketplace/browser/marketplace.css @@ -266,6 +266,11 @@ button { cursor: default; } +.model-content-identity, +.modal-developer { + overflow-wrap: anywhere; +} + .store-row:focus-visible { outline: 2px solid var(--el-focus); outline-offset: 2px; diff --git a/capsules/marketplace/browser/marketplace.js b/capsules/marketplace/browser/marketplace.js index ffbedc8fc..3d8eb942f 100644 --- a/capsules/marketplace/browser/marketplace.js +++ b/capsules/marketplace/browser/marketplace.js @@ -23,10 +23,12 @@ appLoadError: null, mediaLoadError: null, mediaTruncated: false, + modelCatalogState: "unconfigured", }; const pendingMediaBuys = new Set(); let detailPreviousFocus = null; + let detailModels = null; let homeChromeReady = false; let lastHomeMenuManifestSignature = ""; @@ -46,16 +48,12 @@ const categories = [ { id: "apps", label: "Apps", icon: "package" }, { id: "viewers", label: "Viewers", icon: "play" }, + { id: "models", label: "Models", icon: "package" }, { id: "content", label: "Content", icon: "document" }, { id: "providers", label: "Services", icon: "server" }, { id: "shells", label: "Home views", icon: "system" }, ]; - const models = window.ElastosModelManagement.create({ - root: document.querySelector("[data-model-management]"), capsule: "marketplace", token: homeToken, - buttonClass: "store-pill", - }); - const icons = { package: '', play: '', @@ -69,7 +67,6 @@ }; const CAPSULE_ICON_ROUTE = /^\/apps\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_./-]+\.png$/; - document.querySelector("[data-model-icon]").innerHTML = icons.package; boot(); @@ -243,6 +240,17 @@ throw new Error(catalogErrorMessage(interfacesResponse, interfaces)); } const capsules = Array.isArray(catalog.capsules) ? catalog.capsules : []; + state.modelCatalogState = catalog.model_catalog_state || "unconfigured"; + const modelCids = new Set(); + for (const capsule of capsules.filter(item => item.source === "signed-model-catalog")) { + if (catalog.model_catalog_state !== "verified" || capsule.role !== "content" || capsule.installed !== false || capsule.launchable !== false || + capsule.signature_state !== "catalog-signature-verified" || !/^bafybei[a-z2-7]{52}$/.test(capsule.cid) || + typeof capsule.publisher_did !== "string" || !capsule.publisher_did.startsWith("did:") || + !Number.isSafeInteger(capsule.content_size_bytes) || capsule.content_size_bytes <= 0 || modelCids.has(capsule.cid)) { + throw new Error("Model catalog verification is unavailable."); + } + modelCids.add(capsule.cid); + } const capsulesByName = new Map(capsules.map((capsule) => [String(capsule.name || ""), capsule])); const interfacesByCapsule = new Map(); for (const entry of Array.isArray(interfaces.interfaces) ? interfaces.interfaces : []) { @@ -296,9 +304,10 @@ } function capsuleToApp(capsule, capsulesByName, interfaceEntries) { + const model = capsule.source === "signed-model-catalog"; const role = String(capsule.role || "").toLowerCase(); const installed = capsule.installed === true; - const launchable = Boolean(capsule.launchable && capsule.launch_target); + const launchable = !model && Boolean(capsule.launchable && capsule.launch_target); const dependencies = (Array.isArray(capsule.requires) ? capsule.requires : []) .map((entry) => { const dependency = capsulesByName.get(String(entry && entry.name || "")); @@ -306,15 +315,17 @@ }) .filter(Boolean); return { - id: String(capsule.name || ""), + id: model ? `model:${capsule.cid}` : String(capsule.name || ""), + modelCid: model ? capsule.cid : null, + contentBytes: model ? capsule.content_size_bytes : null, name: publicTitle(capsule), - developer: String(capsule.author || "Unknown publisher"), + developer: String(model ? capsule.publisher_did : capsule.author || "Unknown publisher"), category: appCategory(capsule, role), - description: publicDescription(capsule), + description: model ? `${publicTitle(capsule)} model for Assistant.` : publicDescription(capsule), version: String(capsule.version || ""), installed, launchable, - launchTarget: String(capsule.launch_target || ""), + launchTarget: model ? "" : String(capsule.launch_target || ""), role, capsuleType: String(capsule.type || capsule.capsule_type || ""), trustState: String(capsule.trust_state || ""), @@ -324,17 +335,18 @@ iconRoute: catalogIconRoute(capsule), icon: appIcon(role), gradient: appGradient(role), - badges: appBadges(role, capsule, installed), + badges: model ? ["model"] : appBadges(role, capsule, installed), acceptedContent: acceptedContentLabels(capsule, capsulesByName, interfaceEntries), dependencies, - availableActions: executableActions(interfaceEntries), - viewerTitle: String(capsule.viewer_title || ""), - size: capsule.cid ? "Verified app" : "Local app", + availableActions: model ? [] : executableActions(interfaceEntries), + viewerTitle: model ? "Assistant" : String(capsule.viewer_title || ""), + size: model ? `${capsule.content_size_bytes.toLocaleString()} bytes` : capsule.cid ? "Verified app" : "Local app", sourceSummary: capsule.cid ? "SmartWeb" : "Local", }; } function appCategory(capsule, role) { + if (capsule.source === "signed-model-catalog") return "models"; const category = String(capsule.category || "").toLowerCase(); const canonical = { apps: "apps", @@ -457,7 +469,6 @@ } function normalizeDestination(id) { - if (id === "models") return "models"; if (id === "media") return "media"; if (id === "installed") return "installed"; if (categories.some((category) => category.id === id)) return id; @@ -465,7 +476,6 @@ } function destinationTitle(id) { - if (id === "models") return "Models"; if (id === "media") return "Media"; if (id === "installed") return "Installed"; const category = categories.find((entry) => entry.id === id); @@ -475,7 +485,6 @@ function selectDestination(id, options = {}) { state.destination = normalizeDestination(id); - models.setVisible(state.destination === "models"); document.querySelectorAll("[data-destination]").forEach((node) => { const selected = node.dataset.destination === state.destination; node.classList.toggle("selected", selected); @@ -508,12 +517,6 @@ } function renderSurfaceState() { - if (state.destination === "models") { - els.loadingState.classList.add("hidden"); - els.loadError.classList.add("hidden"); - els.storeSections.classList.add("hidden"); - return true; - } const surface = surfaceState(); els.loadingState.classList.toggle("hidden", !surface.loading); if (surface.loading) { @@ -614,6 +617,14 @@ function renderCategorySections() { const apps = filteredByDestination(); if (!apps.length) { + if (state.destination === "models" && !state.search) { + els.storeSections.innerHTML = emptyState( + state.modelCatalogState === "unavailable" ? "Models are unavailable" : "No models available", + state.modelCatalogState === "unavailable" ? "The model catalog could not be verified. Refresh to check again." : "Verified models on this Home will appear here.", + icons.package, + ); + return; + } els.storeSections.innerHTML = emptyState( state.search ? "No results" : "No apps in this category", state.search ? "Try a different search." : "Choose another category from the sidebar.", @@ -621,7 +632,7 @@ ); return; } - els.storeSections.innerHTML = renderSection(destinationTitle(state.destination), apps); + els.storeSections.innerHTML = renderSection(destinationTitle(state.destination), apps, { showHeading: false }); bindAppActions(els.storeSections); } @@ -635,7 +646,7 @@ ); return; } - els.storeSections.innerHTML = renderSection("Installed", installed); + els.storeSections.innerHTML = renderSection("Installed", installed, { showHeading: false }); bindAppActions(els.storeSections); } @@ -684,16 +695,16 @@ }); } - function renderSection(title, apps, { seeAllDestination } = {}) { + function renderSection(title, apps, { seeAllDestination, showHeading = true } = {}) { const seeAll = seeAllDestination && apps.length ? `` : ""; return `
-
+ ${showHeading ? `

${escapeHtml(title)}

${seeAll} -
+
` : ""}
${apps.map(renderAppRow).join("")}
@@ -713,6 +724,7 @@ } function rowSubtitle(app) { + if (app.modelCid) return app.description; if (!isFirstPartyPublisher(app.developer)) { return app.developer; } @@ -810,6 +822,7 @@ } function actionButton(app) { + if (app.modelCid) return ``; if (!app.launchable) { return ""; } @@ -846,6 +859,8 @@ if (!app) { return; } + detailModels?.destroy(); + detailModels = null; detailPreviousFocus = document.activeElement; const openButton = app.launchable ? `` @@ -873,12 +888,12 @@
${relationshipSection(app)} - `} ${technicalDetails(app)}
@@ -890,6 +905,13 @@
`; bindAppActions(els.detailContent); + if (app.modelCid) { + detailModels = window.ElastosModelManagement.create({ + root: els.detailContent.querySelector("[data-model-management]"), capsule: "marketplace", token: homeToken, + buttonClass: "modal-btn secondary", cid: app.modelCid, compact: true, + }); + detailModels.setVisible(true); + } els.detailModal.classList.add("active"); const focusTarget = els.detailContent.querySelector(".modal-btn.primary") || els.detailContent.querySelector("[data-action='close-detail']"); @@ -897,6 +919,8 @@ } function closeDetail() { + detailModels?.destroy(); + detailModels = null; els.detailModal.classList.remove("active"); const restore = detailPreviousFocus; detailPreviousFocus = null; @@ -924,6 +948,7 @@ } function statusItems(app) { + if (app.modelCid) return ["Publisher signature verified", "Assistant uses this model after preparation."]; const items = [ `Trust: ${trustLabel(app.trustState)}`, `Status: ${app.installed ? "Installed on this Home" : "Not installed on this Home"}`, @@ -974,6 +999,7 @@
  • ${icons.check}${escapeHtml(signatureLabel(app.signatureState))}
  • + ${app.modelCid ? `
  • Content ID: ${escapeHtml(app.modelCid)}
  • ${app.contentBytes.toLocaleString()} bytes
  • ` : ""}
  • ${icons.check}${escapeHtml(packageLabel(app))}
@@ -981,6 +1007,7 @@ } function packageLabel(app) { + if (app.modelCid) return "Verified publisher"; if (app.trustState === "cid-with-manifest-signature") return "Verified"; if (app.trustState === "local-manifest-signature") return "Signed local"; if (app.installed) return "On this device"; @@ -1001,6 +1028,7 @@ const labels = { "manifest-signature-declared": "Manifest signature declared", "no-manifest-signature": "Manifest signature not declared", + "catalog-signature-verified": "Catalog publisher signature verified", }; return labels[stateValue] || "Manifest signature status unavailable"; } @@ -1072,7 +1100,7 @@ if (action !== "detail") { event.stopPropagation(); } - if (action === "detail") { + if (action === "detail" || action === "model-detail") { if (target instanceof HTMLElement) { target.focus(); } diff --git a/capsules/marketplace/browser/model-management.css b/capsules/marketplace/browser/model-management.css index 62a16ef7e..d87a1de9f 100644 --- a/capsules/marketplace/browser/model-management.css +++ b/capsules/marketplace/browser/model-management.css @@ -1,9 +1,14 @@ +/* Shared tokens follow the active System or Marketplace theme. */ .model-management[hidden] { display: none; } -.model-management { min-width: 0; color: var(--ink); } -.model-row { padding: 16px 0; border-top: 1px solid var(--line); min-width: 0; } -.model-row h2 { font-size: 18px; margin: 0 0 12px; } +.model-management { min-width: 0; color: var(--el-text); font-size: 13px; } +.model-management > p:empty { display: none; } +.model-row { padding: 16px; margin-top: 12px; border: 1px solid var(--el-hairline); border-radius: 8px; background: var(--el-surface-raised); min-width: 0; } +.model-row h2 { font-size: 16px; margin: 0 0 8px; } .model-row p { overflow-wrap: anywhere; line-height: 1.5; margin: 8px 0; } -.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 16px; } -.model-keep { display: inline-flex; align-items: center; gap: 8px; } -.model-keep input { accent-color: var(--accent); } -.model-row progress { width: 100%; max-width: 440px; accent-color: var(--accent); } +.model-row .model-size, .model-row .model-hint, .model-details { color: var(--el-muted); font-size: 12px; } +.model-details summary { cursor: pointer; } +.model-details p { font-size: 11px; } +.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; margin: 12px 0; } +.model-keep { display: inline-flex; align-items: center; gap: 8px; cursor: pointer; } +.model-keep input { accent-color: var(--el-accent); width: 16px; height: 16px; margin: 0; } +.model-row progress { width: 100%; accent-color: var(--el-accent); } diff --git a/capsules/marketplace/browser/model-management.js b/capsules/marketplace/browser/model-management.js index ce6378009..2ba57fcb3 100644 --- a/capsules/marketplace/browser/model-management.js +++ b/capsules/marketplace/browser/model-management.js @@ -28,6 +28,11 @@ } else check(!r.admitted && !r.kept && !r.dispatch_ready); return r; } + function formatBytes(bytes) { + const units = ["B", "KB", "MB", "GB", "TB"]; + const rank = bytes ? Math.min(4, Math.floor(Math.log(bytes) / Math.log(1000))) : 0; + return `${(bytes / 1000 ** rank).toLocaleString(undefined, { maximumFractionDigits: rank ? 2 : 0 })} ${units[rank]}`; + } function element(tag, value, className) { const node = document.createElement(tag); if (value) node.textContent = value; @@ -43,7 +48,7 @@ dispatch_ready: output.dispatch_ready, offer_id: output.offer_id, preparation }, cid); } window.ElastosModelManagement = { - create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary" }) { + create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: selectedCid = null, compact = false }) { let visible = false, closed = false, generation = 0, busy = false, timer, reader; let model = null, methods = new Map(), message = "", loading = false, polls = 0; let unresolvedUse = null, reconcileRequired = false; @@ -84,6 +89,7 @@ check(entries.length <= 1 && (!entries.length || catalog.model_catalog_state === "verified")); if (!entries.length) return null; const entry = entries[0]; + if (selectedCid && entry.cid !== selectedCid) return null; check(entry.role === "content" && entry.installed === false && entry.launchable === false && CID.test(entry.cid) && text(entry.title) && text(entry.publisher_did, 256) && entry.publisher_did.startsWith("did:") && entry.signature_state === "catalog-signature-verified" @@ -166,11 +172,14 @@ if (epoch !== generation || !show() || model?.cid !== cid) return; check(result.cid === cid); if (operation === "retention") { - check(result.kept === keep && result.admitted === true); - // Keep is confirmed only by the current caller's Runtime response. - const candidate = operationRuntime(result, cid, model.model_runtime.preparation); - check(candidate.kept === keep); - model.model_runtime = candidate; + check(result.kept === keep && typeof result.admitted === "boolean"); + // Admission can finish while Keep is in flight. Read a fresh catalog + // instead of combining new readiness with an older preparation. + const catalog = await request("/api/capsules/catalog", null, io.signal); + if (epoch !== generation || !show() || model?.cid !== cid) return; + const candidate = parseCatalog(catalog); + check(candidate?.cid === cid); + model = candidate; } else { const r = operationRuntime(result, cid); check(r.preparation?.operation_id === result.operation_id && (operation !== "cancel" || result.operation_id === id)); @@ -213,15 +222,20 @@ } const r = model.model_runtime, p = r.preparation; const row = element("article", "", "model-row"); - row.append(element("h2", model.title), element("p", `Verified publisher: ${model.publisher_did}`), - element("p", `Content ID: ${model.cid}`, "model-identity"), element("p", `${model.content_size_bytes.toLocaleString()} bytes`)); - const label = message ? "Current status unavailable" : r.dispatch_ready ? "Ready to use" : r.admitted ? "Prepared. The model offer is unavailable." - : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : "Preparing…") - : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Not prepared"; + if (!compact) { + row.append(element("h2", model.title), element("p", formatBytes(model.content_size_bytes), "model-size")); + const identity = element("details", "", "model-details"); + identity.append(element("summary", "Content details"), element("p", `Verified publisher: ${model.publisher_did}`), + element("p", `Content ID: ${model.cid}`, "model-identity")); + row.append(identity); + } + const label = message ? "Current status unavailable" : r.dispatch_ready ? "Available on this device" : r.admitted ? "Available on this device. Model service unavailable." + : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : p.state === "verifying" ? "Checking model files…" : "Preparing local files…") + : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Ready to prepare"; row.append(element("p", label)); if (active(p)) { const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; - progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${p.completed_bytes.toLocaleString()} of ${p.total_bytes.toLocaleString()} bytes`)); + progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${formatBytes(p.completed_bytes)} of ${formatBytes(p.total_bytes)} prepared`)); } const controls = element("div", "", "model-controls"); const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); @@ -229,10 +243,15 @@ if (!active(p) && !r.dispatch_ready) controls.append(use); if (active(p)) { const cancel = button("Cancel preparation", () => void act("cancel"), p.cancel_requested || reconcileRequired); cancel.dataset.modelControl = "cancel"; controls.append(cancel); } const labelNode = element("label", "", "model-keep"), toggle = element("input"); - toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !r.admitted || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; + toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !(r.admitted || active(p)) || p?.cancel_requested || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; toggle.addEventListener("change", () => void act("retention", toggle.checked)); labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); - row.append(controls); root.append(row); + row.append(controls); + row.append(element("p", r.kept + ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") + : "Prepared files can be removed during cache cleanup.", "model-hint")); + row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + root.append(row); restoreFocus(); } function setVisible(value) { @@ -240,9 +259,15 @@ visible = value; root.hidden = !value; ++generation; stopRead(); if (show()) void load(); } - document.addEventListener("visibilitychange", () => { ++generation; stopRead(); if (show()) void load(); }); - window.addEventListener("pagehide", () => { closed = true; ++generation; stopRead(); }); - return { setVisible, refresh: load }; + const onVisibility = () => { ++generation; stopRead(); if (show()) void load(); }; + function destroy() { + closed = true; ++generation; stopRead(); + document.removeEventListener("visibilitychange", onVisibility); + window.removeEventListener("pagehide", destroy); + } + document.addEventListener("visibilitychange", onVisibility); + window.addEventListener("pagehide", destroy); + return { setVisible, refresh: load, destroy }; }, }; })(); diff --git a/capsules/system/browser/model-management.css b/capsules/system/browser/model-management.css index 62a16ef7e..d87a1de9f 100644 --- a/capsules/system/browser/model-management.css +++ b/capsules/system/browser/model-management.css @@ -1,9 +1,14 @@ +/* Shared tokens follow the active System or Marketplace theme. */ .model-management[hidden] { display: none; } -.model-management { min-width: 0; color: var(--ink); } -.model-row { padding: 16px 0; border-top: 1px solid var(--line); min-width: 0; } -.model-row h2 { font-size: 18px; margin: 0 0 12px; } +.model-management { min-width: 0; color: var(--el-text); font-size: 13px; } +.model-management > p:empty { display: none; } +.model-row { padding: 16px; margin-top: 12px; border: 1px solid var(--el-hairline); border-radius: 8px; background: var(--el-surface-raised); min-width: 0; } +.model-row h2 { font-size: 16px; margin: 0 0 8px; } .model-row p { overflow-wrap: anywhere; line-height: 1.5; margin: 8px 0; } -.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 16px; } -.model-keep { display: inline-flex; align-items: center; gap: 8px; } -.model-keep input { accent-color: var(--accent); } -.model-row progress { width: 100%; max-width: 440px; accent-color: var(--accent); } +.model-row .model-size, .model-row .model-hint, .model-details { color: var(--el-muted); font-size: 12px; } +.model-details summary { cursor: pointer; } +.model-details p { font-size: 11px; } +.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; margin: 12px 0; } +.model-keep { display: inline-flex; align-items: center; gap: 8px; cursor: pointer; } +.model-keep input { accent-color: var(--el-accent); width: 16px; height: 16px; margin: 0; } +.model-row progress { width: 100%; accent-color: var(--el-accent); } diff --git a/capsules/system/browser/model-management.js b/capsules/system/browser/model-management.js index ce6378009..2ba57fcb3 100644 --- a/capsules/system/browser/model-management.js +++ b/capsules/system/browser/model-management.js @@ -28,6 +28,11 @@ } else check(!r.admitted && !r.kept && !r.dispatch_ready); return r; } + function formatBytes(bytes) { + const units = ["B", "KB", "MB", "GB", "TB"]; + const rank = bytes ? Math.min(4, Math.floor(Math.log(bytes) / Math.log(1000))) : 0; + return `${(bytes / 1000 ** rank).toLocaleString(undefined, { maximumFractionDigits: rank ? 2 : 0 })} ${units[rank]}`; + } function element(tag, value, className) { const node = document.createElement(tag); if (value) node.textContent = value; @@ -43,7 +48,7 @@ dispatch_ready: output.dispatch_ready, offer_id: output.offer_id, preparation }, cid); } window.ElastosModelManagement = { - create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary" }) { + create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: selectedCid = null, compact = false }) { let visible = false, closed = false, generation = 0, busy = false, timer, reader; let model = null, methods = new Map(), message = "", loading = false, polls = 0; let unresolvedUse = null, reconcileRequired = false; @@ -84,6 +89,7 @@ check(entries.length <= 1 && (!entries.length || catalog.model_catalog_state === "verified")); if (!entries.length) return null; const entry = entries[0]; + if (selectedCid && entry.cid !== selectedCid) return null; check(entry.role === "content" && entry.installed === false && entry.launchable === false && CID.test(entry.cid) && text(entry.title) && text(entry.publisher_did, 256) && entry.publisher_did.startsWith("did:") && entry.signature_state === "catalog-signature-verified" @@ -166,11 +172,14 @@ if (epoch !== generation || !show() || model?.cid !== cid) return; check(result.cid === cid); if (operation === "retention") { - check(result.kept === keep && result.admitted === true); - // Keep is confirmed only by the current caller's Runtime response. - const candidate = operationRuntime(result, cid, model.model_runtime.preparation); - check(candidate.kept === keep); - model.model_runtime = candidate; + check(result.kept === keep && typeof result.admitted === "boolean"); + // Admission can finish while Keep is in flight. Read a fresh catalog + // instead of combining new readiness with an older preparation. + const catalog = await request("/api/capsules/catalog", null, io.signal); + if (epoch !== generation || !show() || model?.cid !== cid) return; + const candidate = parseCatalog(catalog); + check(candidate?.cid === cid); + model = candidate; } else { const r = operationRuntime(result, cid); check(r.preparation?.operation_id === result.operation_id && (operation !== "cancel" || result.operation_id === id)); @@ -213,15 +222,20 @@ } const r = model.model_runtime, p = r.preparation; const row = element("article", "", "model-row"); - row.append(element("h2", model.title), element("p", `Verified publisher: ${model.publisher_did}`), - element("p", `Content ID: ${model.cid}`, "model-identity"), element("p", `${model.content_size_bytes.toLocaleString()} bytes`)); - const label = message ? "Current status unavailable" : r.dispatch_ready ? "Ready to use" : r.admitted ? "Prepared. The model offer is unavailable." - : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : "Preparing…") - : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Not prepared"; + if (!compact) { + row.append(element("h2", model.title), element("p", formatBytes(model.content_size_bytes), "model-size")); + const identity = element("details", "", "model-details"); + identity.append(element("summary", "Content details"), element("p", `Verified publisher: ${model.publisher_did}`), + element("p", `Content ID: ${model.cid}`, "model-identity")); + row.append(identity); + } + const label = message ? "Current status unavailable" : r.dispatch_ready ? "Available on this device" : r.admitted ? "Available on this device. Model service unavailable." + : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : p.state === "verifying" ? "Checking model files…" : "Preparing local files…") + : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Ready to prepare"; row.append(element("p", label)); if (active(p)) { const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; - progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${p.completed_bytes.toLocaleString()} of ${p.total_bytes.toLocaleString()} bytes`)); + progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${formatBytes(p.completed_bytes)} of ${formatBytes(p.total_bytes)} prepared`)); } const controls = element("div", "", "model-controls"); const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); @@ -229,10 +243,15 @@ if (!active(p) && !r.dispatch_ready) controls.append(use); if (active(p)) { const cancel = button("Cancel preparation", () => void act("cancel"), p.cancel_requested || reconcileRequired); cancel.dataset.modelControl = "cancel"; controls.append(cancel); } const labelNode = element("label", "", "model-keep"), toggle = element("input"); - toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !r.admitted || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; + toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !(r.admitted || active(p)) || p?.cancel_requested || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; toggle.addEventListener("change", () => void act("retention", toggle.checked)); labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); - row.append(controls); root.append(row); + row.append(controls); + row.append(element("p", r.kept + ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") + : "Prepared files can be removed during cache cleanup.", "model-hint")); + row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + root.append(row); restoreFocus(); } function setVisible(value) { @@ -240,9 +259,15 @@ visible = value; root.hidden = !value; ++generation; stopRead(); if (show()) void load(); } - document.addEventListener("visibilitychange", () => { ++generation; stopRead(); if (show()) void load(); }); - window.addEventListener("pagehide", () => { closed = true; ++generation; stopRead(); }); - return { setVisible, refresh: load }; + const onVisibility = () => { ++generation; stopRead(); if (show()) void load(); }; + function destroy() { + closed = true; ++generation; stopRead(); + document.removeEventListener("visibilitychange", onVisibility); + window.removeEventListener("pagehide", destroy); + } + document.addEventListener("visibilitychange", onVisibility); + window.addEventListener("pagehide", destroy); + return { setVisible, refresh: load, destroy }; }, }; })(); diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs index 1d558e808..59f9e91f9 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs @@ -410,12 +410,9 @@ impl PreparationInventory { bounded_id(&claim.principal, 160) && canonical_cid(&claim.cid, 0x70) && claims.insert((&claim.principal, &claim.cid)) - && self - .records - .iter() - .any(|record| record.state == PreparationState::Admitted - && record.request_binding.principal == claim.principal - && record.package_cid == claim.cid), + && self.records.iter().any(|record| record.backs_retention() + && record.request_binding.principal == claim.principal + && record.package_cid == claim.cid), "invalid retention claim" ); } @@ -428,6 +425,19 @@ impl PreparationInventory { .any(|claim| claim.principal == principal && claim.cid == cid) } + fn prune_retention(&mut self) -> bool { + let before = self.retention_claims.len(); + let records = &self.records; + self.retention_claims.retain(|claim| { + records.iter().any(|record| { + record.backs_retention() + && record.request_binding.principal == claim.principal + && record.package_cid == claim.cid + }) + }); + self.retention_claims.len() != before + } + fn expire(&mut self, now: u64) -> bool { // Pre-dispatch records have no provider work. Active work must // settle and drain before expiry can release its byte charge. @@ -445,7 +455,7 @@ impl PreparationInventory { changed = true; } } - changed + self.prune_retention() || changed } } @@ -665,25 +675,36 @@ fn set_retention( let inventory = Inventory::open(data_dir, false)?; let mut state = inventory.load()?; let principal = &caller.context.principal_id; - let admitted = state + let record = state .records .iter() - .find(|record| { - record.state == PreparationState::Admitted + .filter(|record| { + record.backs_retention() && record.request_binding.principal == *principal && record.package_cid == cid }) - .context("retention admission unavailable")?; - inventory.admitted(&admitted.admission_id)?.check()?; + .max_by_key(|record| record.state == PreparationState::Admitted) + .context("retention preparation unavailable")?; + let admitted = record.state == PreparationState::Admitted; + if admitted { + inventory.admitted(&record.admission_id)?.check()?; + } else if keep && !state.kept(principal, cid) { + ensure!( + !record.cancel_requested && now()? < record.expires_at, + "retention preparation is settling" + ); + } ensure!( state .retirement .as_ref() - .is_none_or(|r| r.admission_id != admitted.admission_id), + .is_none_or(|r| r.admission_id != record.admission_id), "model retirement pending" ); - // Desired local retention is independent of model-use permission. Aliases - // share one principal/CID claim; releasing it does not evict bytes or offers. + // Desired local retention is independent of model-use permission. Pending + // intent survives reconciliation and admission, but ends when its last + // operation settles without admission. It does not pin or fetch content. + // Aliases share one principal/CID claim; release does not evict bytes/offers. if state.kept(principal, cid) != keep { if keep { ensure!( @@ -702,7 +723,7 @@ fn set_retention( revalidate()?; inventory.save(&state)?; } - Ok(serde_json::json!({"cid":cid,"kept":keep,"admitted":true})) + Ok(serde_json::json!({"cid":cid,"kept":keep,"admitted":admitted})) } fn status( @@ -761,6 +782,7 @@ fn manage( changed = true; } let result = record.clone(); + changed |= state.prune_retention(); if changed { inventory.save(&state)?; } @@ -768,6 +790,10 @@ fn manage( } impl PreparationRecord { + fn backs_retention(&self) -> bool { + self.state == PreparationState::Admitted || self.pre_dispatch() || self.active() + } + fn pre_dispatch(&self) -> bool { matches!( self.state, @@ -2489,6 +2515,7 @@ fn settle_failure(data_dir: &Path, id: &str, drained: bool) -> anyhow::Result<() }; record.reserved_bytes = 0; } + snapshot.prune_retention(); inventory.save(&snapshot) } @@ -7296,6 +7323,213 @@ server.serve_forever() ); } + #[tokio::test] + async fn model_retention_pending_replay_restart_and_admission_preserve_intent() { + let (root, record, backend, registry) = staged_fixture(now().unwrap(), true).await; + let calls = backend.calls.lock().unwrap().clone(); + let path = root.path().join("model-preparation/state.json"); + let mut foreign = context(); + foreign.principal_id = "person:other-pending-retention".into(); + for keep in [true, false] { + let before = std::fs::read(&path).unwrap(); + assert!(retention_intent(root.path(), &foreign, &record.package_cid, keep).is_err()); + assert_eq!(std::fs::read(&path).unwrap(), before); + } + for keep in [true, true, false, false, true] { + let reply = + retention_intent(root.path(), &context(), &record.package_cid, keep).unwrap(); + assert_eq!( + reply, + serde_json::json!({"cid":record.package_cid,"kept":keep,"admitted":false}) + ); + let before = std::fs::read(&path).unwrap(); + // A fresh owner reads the same durable intent without starting a worker. + assert_eq!( + retention_intent(root.path(), &context(), &record.package_cid, keep).unwrap(), + reply + ); + assert_eq!(std::fs::read(&path).unwrap(), before); + assert_eq!( + load_operation(root.path(), &record.operation_id).unwrap(), + record + ); + assert_eq!(*backend.calls.lock().unwrap(), calls); + } + let projection = + model_runtime_projection(root.path(), None, &context(), &record.package_cid, None) + .await; + assert_eq!(projection["kept"], true); + assert_eq!(projection["admitted"], false); + assert_eq!(projection["dispatch_ready"], false); + assert!(projection["offer_id"].is_null()); + settle_failure(root.path(), &record.operation_id, false).unwrap(); + assert_eq!( + load_operation(root.path(), &record.operation_id) + .unwrap() + .state, + PreparationState::Uncertain + ); + assert_eq!( + retention_intent(root.path(), &context(), &record.package_cid, true).unwrap() + ["admitted"], + false + ); + assert_eq!(*backend.calls.lock().unwrap(), calls); + finish_retention_fixture(root.path(), &record, registry).await; + assert_eq!( + retention_intent(root.path(), &context(), &record.package_cid, true).unwrap(), + serde_json::json!({"cid":record.package_cid,"kept":true,"admitted":true}) + ); + } + + #[test] + fn model_retention_pending_cancel_expire_and_fresh_retry_clear_intent() { + for terminal in ["cancel", "expire", "capacity-cancel", "capacity-expire"] { + let root = tempfile::tempdir().unwrap(); + let (payload, _) = package_fixture(b"GGUF\x03\0\0\0fixture".to_vec()); + write_preparation_catalog(root.path(), &payload); + let cid = payload["entries"][0]["cid"].as_str().unwrap(); + let record = reserve( + root.path(), + &caller(&context(), &method("use")), + "original", + cid, + ) + .unwrap(); + if terminal.starts_with("capacity-") { + update_operation(root.path(), &record.operation_id, |pending| { + pending.state = PreparationState::CapacityPending; + pending.reserved_bytes = 0; + }) + .unwrap(); + } + retention_intent(root.path(), &context(), cid, true).unwrap(); + let cancelling = terminal.ends_with("cancel"); + let result = manage( + root.path(), + &caller( + &context(), + &method(if cancelling { "cancel" } else { "status" }), + ), + &record.operation_id, + cancelling, + if cancelling { + record.created_at + } else { + record.expires_at + }, + ) + .unwrap(); + assert_eq!( + result.state, + if cancelling { + PreparationState::Cancelled + } else { + PreparationState::Expired + } + ); + assert_eq!(result.reserved_bytes, 0); + let inventory = Inventory::open(root.path(), false).unwrap(); + assert!(!inventory.load().unwrap().kept(&context().principal_id, cid)); + drop(inventory); + let next = reserve( + root.path(), + &caller(&context(), &method("use")), + "fresh-retry", + cid, + ) + .unwrap(); + assert_ne!(next.operation_id, record.operation_id); + let inventory = Inventory::open(root.path(), false).unwrap(); + assert!(!inventory.load().unwrap().kept(&context().principal_id, cid)); + } + } + + #[tokio::test] + async fn model_retention_active_cancel_waits_for_drain_and_failure_clears_intent() { + for cancelled in [false, true] { + let (root, record, backend, _) = staged_fixture(now().unwrap(), false).await; + retention_intent(root.path(), &context(), &record.package_cid, true).unwrap(); + let calls = backend.calls.lock().unwrap().clone(); + if cancelled { + let pending = cancel( + root.path(), + &caller(&context(), &method("cancel")), + &record.operation_id, + ) + .unwrap(); + assert!(pending.cancel_requested); + } + settle_failure(root.path(), &record.operation_id, false).unwrap(); + { + let inventory = Inventory::open(root.path(), false).unwrap(); + assert!(inventory + .load() + .unwrap() + .kept(&context().principal_id, &record.package_cid)); + } + settle_failure(root.path(), &record.operation_id, true).unwrap(); + let settled = load_operation(root.path(), &record.operation_id).unwrap(); + assert_eq!( + settled.state, + if cancelled { + PreparationState::Cancelled + } else { + PreparationState::Failed + } + ); + assert_eq!(settled.reserved_bytes, 0); + let inventory = Inventory::open(root.path(), false).unwrap(); + assert!(!inventory + .load() + .unwrap() + .kept(&context().principal_id, &record.package_cid)); + assert_eq!(*backend.calls.lock().unwrap(), calls); + } + } + + #[tokio::test] + async fn model_retention_cancelled_shared_preparation_preserves_admitted_claim() { + let (root, record, backend, registry) = staged_fixture(now().unwrap(), true).await; + finish_retention_fixture(root.path(), &record, registry).await; + retention_intent(root.path(), &context(), &record.package_cid, true).unwrap(); + let calls = backend.calls.lock().unwrap().clone(); + let mut foreign = context(); + foreign.principal_id = "person:shared-pending-retention".into(); + for principal in [context(), foreign] { + let shared = reserve( + root.path(), + &caller(&principal, &method("use")), + "pending-alias", + &record.package_cid, + ) + .unwrap(); + assert_eq!(shared.admission_id, record.operation_id); + let reply = + retention_intent(root.path(), &principal, &record.package_cid, true).unwrap(); + assert_eq!( + reply["admitted"], + principal.principal_id == context().principal_id + ); + cancel( + root.path(), + &caller(&principal, &method("cancel")), + &shared.operation_id, + ) + .unwrap(); + let inventory = Inventory::open(root.path(), false).unwrap(); + let state = inventory.load().unwrap(); + assert!(state.kept(&context().principal_id, &record.package_cid)); + assert_eq!( + state.kept(&principal.principal_id, &record.package_cid), + principal.principal_id == context().principal_id + ); + assert_eq!(state.retention_claims.len(), 1); + assert!(inventory.admitted(&record.operation_id).is_ok()); + } + assert_eq!(*backend.calls.lock().unwrap(), calls); + } + #[tokio::test] async fn model_retention_desired_state_replay_restart_preserves_admission_and_bytes() { use std::os::unix::fs::MetadataExt as _; @@ -7453,9 +7687,10 @@ server.serve_forever() #[tokio::test] async fn model_retention_invalid_input_and_authority_preserve_inventory() { let (root, record, _, registry) = staged_fixture(now().unwrap(), true).await; - assert!( - retention_intent(root.path(), &context(), &record.package_cid, true).is_err(), - "pending admission cannot be kept" + assert_eq!( + retention_intent(root.path(), &context(), &record.package_cid, true).unwrap() + ["admitted"], + false ); finish_retention_fixture(root.path(), &record, registry).await; retention_intent(root.path(), &context(), &record.package_cid, true).unwrap(); @@ -7585,7 +7820,7 @@ server.serve_forever() #[test] fn model_retention_first_party_manifests_declare_local_management() { let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); - for name in ["marketplace", "system", "assistant", "home-agent"] { + for name in ["marketplace", "system", "assistant"] { let value: serde_json::Value = serde_json::from_slice( &std::fs::read(root.join("capsules").join(name).join("capsule.json")).unwrap(), ) diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index 9cc5f6725..c65367af8 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -5035,13 +5035,13 @@ assert( "Marketplace must project canonical roles, relationships, executable bindings, and declared icon routes without name-based guesses", ); assert( - marketplaceUi.includes('size: capsule.cid ? "Verified app" : "Local app"') && + marketplaceUi.includes('capsule.cid ? "Verified app" : "Local app"') && marketplaceUi.includes('sourceSummary: capsule.cid ? "SmartWeb" : "Local"') && marketplaceUi.includes("Trust:") && marketplaceUi.includes("Status:") && marketplaceUi.includes("Available actions") && !marketplaceUi.includes("CID-backed") && - !marketplaceUi.includes("Content ID") && + !marketplaceUi.replace(/function technicalDetails\(app\) \{[\s\S]*?function packageLabel/, "").includes("Content ID") && !marketplaceUi.includes("Signed package") && !marketplaceUi.includes("Package identity:") && !marketplaceUi.includes('price-tag">${app.cid ? "CID"') && diff --git a/scripts/model-management-browser-smoke.mjs b/scripts/model-management-browser-smoke.mjs index 22779e0cf..27286214a 100644 --- a/scripts/model-management-browser-smoke.mjs +++ b/scripts/model-management-browser-smoke.mjs @@ -17,6 +17,7 @@ let dispatchPending = false; let failure = false; let delayStatus = null; let delayUse = null; +let delayRetention = null; let wrongIdentity = false; let executable = true; let selectedCid = cid; @@ -81,7 +82,7 @@ const server = createServer(async (req, res) => { if (method.operation === "use") { assert.deepEqual(input.input, { cid }); phase = "preparing"; } if (method.operation === "use" && loseUseResponse) { req.socket.destroy(); return; } if (["status", "cancel"].includes(method.operation)) assert.deepEqual(input.input, { operation_id: operation }); - if (method.operation === "cancel") phase = "cancelled"; + if (method.operation === "cancel") { phase = "cancelled"; kept = false; } if (method.operation === "retention") { assert.deepEqual(input.input, { cid, keep: !kept }); kept = input.input.keep; } const readiness = runtime(); const facts = { admitted: readiness.admitted, kept: readiness.kept, dispatch_ready: readiness.dispatch_ready, offer_id: readiness.offer_id }; @@ -95,6 +96,7 @@ const server = createServer(async (req, res) => { const reply = () => json(result); if (method.operation === "status" && delayStatus) { delayStatus(reply); return; } if (method.operation === "use" && delayUse) { delayUse(reply); return; } + if (method.operation === "retention" && delayRetention) { delayRetention(reply, output); return; } reply(); return; } if (["/api/apps/home/summary", "/api/apps/system/summary"].includes(url.pathname)) { json(makeSystemSummary(makeAppearanceRecord())); return; } @@ -111,18 +113,37 @@ const server = createServer(async (req, res) => { }); server.listen(0, "127.0.0.1"); await once(server, "listening"); const browser = await chromium.launch({ executablePath: brave, headless: true }); +let activePage; try { for (const app of ["marketplace", "system"]) { phase = "unprepared"; kept = false; failure = false; calls.length = 0; selectedCid = cid; executable = true; trust = "verified"; const page = await browser.newPage({ viewport: { width: 1100, height: 800 } }); + activePage = page; const pageErrors = []; page.on("pageerror", error => pageErrors.push(String(error))); await page.goto(`http://127.0.0.1:${server.address().port}/?app=${app}`); const frame = page.frameLocator("iframe"); + if (app === "marketplace") { + for (const title of ["Installed", "Apps", "Models"]) { + await frame.locator("#store-sections").getByRole("heading", { name: title, exact: true }).waitFor(); + } + for (const title of ["Apps", "Installed"]) { + await frame.locator(`[data-destination="${title.toLowerCase()}"]`).first().click(); + await frame.locator("#store-sections").getByText("People", { exact: true }).waitFor(); + assert.equal(await frame.getByRole("heading", { name: title, exact: true }).count(), 1, `${title} has one page title`); + assert.equal(await frame.locator("#store-sections .store-section-head").count(), 0, `${title} omits the repeated section heading`); + } + } await frame.getByRole("button", { name: "Models", exact: true }).click({ timeout: 3000 }); - await frame.getByRole("heading", { name: "Fixture model", exact: true }).waitFor(); - assert.match(await frame.locator("[data-model-management]").innerText(), /did:key:zFixturePublisher/); + if (app === "marketplace") { + await frame.getByRole("button", { name: "Details", exact: true }).waitFor(); + assert.equal(await frame.getByRole("heading", { name: "Models", exact: true }).count(), 1, "Models has one page title"); + assert.equal(await frame.locator("#store-sections .store-section-head").count(), 0, "Models omits the repeated section heading"); + await frame.getByRole("button", { name: "Details", exact: true }).click(); + } + await (app === "marketplace" ? frame.locator(".modal-title") : frame.getByRole("heading", { name: "Fixture model", exact: true })).waitFor(); + assert.match(await frame.locator("body").textContent(), /did:key:zFixturePublisher/); phase = "capacity_pending"; await frame.getByRole("button", { name: "Refresh models" }).click(); await frame.getByText("Waiting for local capacity…", { exact: true }).waitFor(); @@ -135,7 +156,7 @@ try { assert.equal(await frame.getByRole("checkbox", { name: "Keep on this device" }).isEnabled(), false); phase = "unprepared"; await frame.getByRole("button", { name: "Refresh models" }).click(); - await frame.getByText("Not prepared", { exact: true }).waitFor(); + await frame.getByText("Ready to prepare", { exact: true }).waitFor(); let releaseUse; const useReceived = new Promise(resolveUse => { delayUse = reply => { releaseUse = reply; resolveUse(); }; }); await frame.getByRole("button", { name: "Use", exact: true }).evaluate(button => { button.click(); button.click(); }); @@ -144,16 +165,22 @@ try { delayUse = null; releaseUse(); await frame.getByRole("button", { name: "Cancel preparation" }).waitFor(); assert.equal(calls.filter(c => c.method === "content.use").length, 1); + await frame.getByRole("checkbox", { name: "Keep on this device" }).check(); + await frame.getByText("Keep choice saved for this preparation.", { exact: true }).waitFor(); + await frame.getByRole("button", { name: "Refresh models" }).click(); + await frame.locator('[data-model-management] input:enabled:checked').waitFor(); + assert.equal(calls.filter(c => c.method === "content.use").length, 1, "retention never starts another preparation"); await frame.getByRole("button", { name: "Cancel preparation" }).click(); await frame.getByRole("button", { name: "Retry", exact: true }).waitFor(); + assert.equal(await frame.getByRole("checkbox", { name: "Keep on this device" }).isChecked(), false, "cancelled attempt releases pending choice"); await frame.getByRole("button", { name: "Retry", exact: true }).click(); await frame.getByRole("button", { name: "Cancel preparation" }).waitFor(); phase = "admitted"; activationPending = app === "marketplace"; dispatchPending = true; - await frame.getByText("Prepared. The model offer is unavailable.", { exact: true }).waitFor(); + await frame.getByText("Available on this device. Model service unavailable.", { exact: true }).waitFor(); const activationUses = calls.filter(c => c.method === "content.use").length; assert.equal(await frame.getByRole("button", { name: "Cancel preparation" }).count(), 0, "admitted content is not cancellable"); activationPending = false; dispatchPending = false; - await frame.locator("[data-model-management] p").filter({ hasText: /^Ready to use$/ }).waitFor(); + await frame.locator("[data-model-management] p").filter({ hasText: /^Available on this device$/ }).waitFor(); assert.equal(calls.filter(c => c.method === "content.use").length, activationUses, "activation readiness uses status without another Use"); const readyStatuses = calls.filter(c => c.method === "content.status").length; await page.waitForTimeout(1750); @@ -162,6 +189,29 @@ try { await frame.locator('[data-model-management] input:enabled:checked').waitFor(); await frame.getByRole("checkbox", { name: "Keep on this device" }).uncheck(); await frame.locator('[data-model-management] input:enabled:not(:checked)').waitFor(); + phase = "preparing"; + await frame.getByRole("button", { name: "Refresh models" }).click(); + await frame.getByRole("button", { name: "Cancel preparation" }).waitFor(); + const usesBeforeRetentionRace = calls.filter(c => c.method === "content.use").length; + const statusesBeforeRetentionRace = calls.filter(c => c.method === "content.status").length; + let releaseRetention; + const retentionReceived = new Promise(resolveRetention => { + delayRetention = (reply, output) => { releaseRetention = reply; resolveRetention(output); }; + }); + // Click does not wait for the checked state while the test holds the ack. + await frame.getByRole("checkbox", { name: "Keep on this device" }).click(); + const pendingRetentionAck = await retentionReceived; + assert.equal(pendingRetentionAck.admitted, false, "retention acknowledgement captures the pending preparation"); + assert.equal(pendingRetentionAck.kept, true); + phase = "admitted"; + delayRetention = null; releaseRetention(); + await frame.locator("[data-model-management] p").filter({ hasText: /^Available on this device$/ }).waitFor(); + await frame.locator('[data-model-management] input:enabled:checked').waitFor(); + assert.equal(await frame.getByRole("button", { name: "Cancel preparation" }).count(), 0, "fresh admission replaces the pending preparation after retention settles"); + assert.equal(calls.filter(c => c.method === "content.use").length, usesBeforeRetentionRace, "admission during retention does not dispatch another Use"); + assert.equal(calls.filter(c => c.method === "content.status").length, statusesBeforeRetentionRace, "retention reads fresh admission before another status poll can hide stale state"); + await frame.getByRole("checkbox", { name: "Keep on this device" }).uncheck(); + await frame.locator('[data-model-management] input:enabled:not(:checked)').waitFor(); failure = true; await frame.getByRole("checkbox", { name: "Keep on this device" }).click(); await frame.getByText("Model action could not be confirmed. Refresh to check its status.", { exact: true }).waitFor(); @@ -189,6 +239,10 @@ try { await frame.getByRole("checkbox", { name: "Keep on this device" }).scrollIntoViewIfNeeded(); if (process.env.MODEL_UI_SCREENSHOTS) await page.screenshot({ path: `${process.env.MODEL_UI_SCREENSHOTS}/${app}-models-narrow.png` }); + await frame.locator("html").evaluate(node => node.dataset.elTheme = "light"); + await bounds(); + if (process.env.MODEL_UI_SCREENSHOTS) await page.screenshot({ path: `${process.env.MODEL_UI_SCREENSHOTS}/${app}-models-light.png` }); + await frame.locator("html").evaluate(node => node.dataset.elTheme = "dark"); await page.setViewportSize({ width: 1100, height: 800 }); phase = "unprepared"; kept = false; await frame.getByRole("button", { name: "Refresh models" }).click(); @@ -265,7 +319,10 @@ try { const hiddenCount = calls.filter(c => c.method === "content.status").length; await new Promise(resolveTick => setTimeout(resolveTick, 1750)); assert.equal(calls.filter(c => c.method === "content.status").length, hiddenCount, "hidden document stops polling"); - const leave = () => frame.getByRole("button", { name: app === "marketplace" ? "Discover" : "About", exact: true }).click(); + const leave = async () => { + if (app === "marketplace") await frame.getByRole("button", { name: "Close", exact: true }).first().click(); + await frame.getByRole("button", { name: app === "marketplace" ? "Discover" : "About", exact: true }).click(); + }; await leave(); const statusCount = calls.filter(c => c.method === "content.status").length; await new Promise(resolveTick => setTimeout(resolveTick, 1750)); @@ -277,17 +334,62 @@ try { } selectedCid = `bafybei${"b".repeat(52)}`; phase = "unprepared"; await frame.locator("body").evaluate(() => { delete document.hidden; document.dispatchEvent(new Event("visibilitychange")); }); + if (app === "marketplace") { + const refreshed = page.waitForResponse(r => new URL(r.url()).pathname === "/api/capsules/catalog"); + await frame.locator("body").evaluate(() => window.dispatchEvent(new MessageEvent("message", { + origin: "null", source: window.parent, data: { type: "elastos:menu-command", cmd: "refresh" }, + }))); + await refreshed; + await frame.getByRole("button", { name: "Models", exact: true }).click(); + await frame.locator(`[data-app="model:${selectedCid}"]`).first().waitFor({ state: "attached" }); + } await frame.getByRole("button", { name: "Models", exact: true }).click(); - await frame.locator(".model-identity").filter({ hasText: selectedCid }).waitFor(); + if (app === "marketplace") await frame.getByRole("button", { name: "Details", exact: true }).click(); + if (app === "system") await frame.getByText("Content details", { exact: true }).click(); + if (app === "marketplace") await frame.getByText("Technical details", { exact: true }).click(); + await frame.locator(app === "marketplace" ? ".model-content-identity" : ".model-identity").filter({ hasText: selectedCid }).waitFor(); delayStatus = null; releaseStatus(); const deliveredStatus = await frame.locator("body").evaluate(() => window.modelStatusDelivered); assert.equal(deliveredStatus.cid, cid, "old status actually reaches the consumer after the CID switch"); assert.equal(deliveredStatus.operation_id, operation); assert.equal(await frame.getByRole("button", { name: "Cancel preparation" }).count(), 0, "late old operation cannot replace new identity"); - assert.match(await frame.locator(".model-identity").innerText(), new RegExp(selectedCid)); + assert.match(await frame.locator(app === "marketplace" ? ".model-content-identity" : ".model-identity").innerText(), new RegExp(selectedCid)); + if (app === "marketplace") { + await frame.getByRole("button", { name: "Use", exact: true }).waitFor(); + const detailCid = selectedCid; + await frame.locator("[data-model-management]").evaluate(node => { + window.fixtureOpenModelDetail = node; + window.fixturePreviousModelUse = node.querySelector('[data-model-control="use"]'); + }); + const mutationsBeforeRotation = calls.filter(c => c.method !== "content.status").length; + selectedCid = `bafybei${"c".repeat(52)}`; + await frame.getByRole("button", { name: "Refresh models" }).click(); + await frame.getByText("No verified model is available. Ask your administrator to configure a trusted catalog.", { exact: true }).waitFor(); + assert.equal(await frame.locator("[data-model-management]").evaluate(node => node === window.fixtureOpenModelDetail), true, "CID rotation leaves the same detail controller mounted"); + assert.match(await frame.locator(".model-content-identity").innerText(), new RegExp(detailCid), "the open detail keeps its original CID"); + assert.equal(await frame.locator('[data-model-control="use"], [data-model-control="keep"], [data-model-control="cancel"]').count(), 0, "the previous detail exposes no actions for the replacement CID"); + const staleMutationAttempts = await frame.locator("body").evaluate(() => { + const originalFetch = window.fetch; + let attempts = 0; + window.fetch = (...args) => { + if (args[0] === "/api/capsules/interfaces/invoke") attempts++; + return originalFetch(...args); + }; + try { window.fixturePreviousModelUse.click(); } finally { window.fetch = originalFetch; } + return attempts; + }); + assert.equal(staleMutationAttempts, 0, "even a detached previous Use control cannot dispatch after CID rotation"); + assert.equal(calls.filter(c => c.method !== "content.status").length, mutationsBeforeRotation, "CID rotation performs no model mutation"); + } assert.deepEqual(pageErrors, []); await page.close(); } assert.deepEqual(fixtureErrors, []); console.log("PASS Marketplace/System model management browser smoke"); +} catch (error) { + if (process.env.MODEL_UI_SCREENSHOTS && activePage && !activePage.isClosed()) { + await activePage.screenshot({ path: `${process.env.MODEL_UI_SCREENSHOTS}/failure.png` }); + console.error((await activePage.locator("iframe").contentFrame().locator("body").innerText()).slice(0, 8000)); + } + throw error; } finally { await browser.close(); server.closeAllConnections(); await new Promise(resolveClose => server.close(resolveClose)); } From 06bf4e0f11e602733e8eef861b835184ae683504 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 10:51:34 -0500 Subject: [PATCH 299/535] docs: record installed model and Assistant feedback checkpoint --- TASKS.md | 2 +- docs/audits/2026-09-10-execution-handover.md | 49 +++++++-------- .../2026-09-11-assistant-convergence.md | 60 ++++++++++++++++++- state.md | 4 +- 4 files changed, 85 insertions(+), 30 deletions(-) diff --git a/TASKS.md b/TASKS.md index cd45e2886..429f97184 100644 --- a/TASKS.md +++ b/TASKS.md @@ -44,7 +44,7 @@ The C1-C7 sequence in the approved plan controls integration and publication. | C1-delivery | Signed three-platform candidate installation | Functional preparation complete; final assembly after C5 freeze / delivery owner | Native inputs, bootstrap and publisher import have bounded source/artifact proof. Keep working builds for functional tests and rebuild only affected components. Complete atomic promotion, final installer stamping, platform regeneration and signed fresh installs at the final release boundary. | | J1 / C2 | Install, passkey, Recovery Kit, Profile and usable Home | Human passkey proof pending / Home and installer owners | Registration binding and atomic/conflict-safe identity persistence are integrated with 66 focused tests passing. Durable owner intent and Profile-inclusive recovery now pass source and UI checks. Real signed Mac installation and seed export pass. Source repairs for kit-first recovery, gateway shutdown and media-tool reuse are accepted. Both targets pass installed browser recovery/reload/retry, complete owned shutdown/restart, full Home setup/media reuse and Carrier cleanup/reuse. User-review account label, Recovery/Advanced spacing and cross-port session isolation are repaired on `1e320578`; shared-browser Mac/Linux proof passes. Next: AUTH-01 final user acceptance on the updated preserved Homes; concurrent-save and window/selection UI; W2-W5; rerun the smoke with System entry through the ElastOS menu, then complete the app matrix; full Linux/Mac staging. Publish J1 for review first under D1. | | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | -| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Next obtain human model admission and reply acceptance there; Linux remains waiting on disk headroom and a reviewed startup profile. Cold Carrier delivery, Marketplace handoff and retention/removal keep their Required gates. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | +| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Anders confirms Assistant works. Current `3c2f9a80` repairs pending Keep intent, normal Marketplace Models category/details, theme tokens and duplicate activity/headings. Source/rendered checks and installed Mac retention readback, Qwen reply/save/reload and human model-view observation pass, with matching receipts and user data preserved. Pending preparation itself uses source/rendered proof; the installed model was reused. Linux still needs disk headroom and a reviewed startup profile. Next required work is exact Marketplace-to-Assistant handoff, safe removal and cold Carrier delivery. Broader execution and monitoring pause at this user-feedback checkpoint; full human J3 remains required. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | | J4 / C4 | Browser works in the two agreed placements | Planned / Browser owner | Retain canonical B01-B16 on the Browser branch; Mac A/A/A and Linux Home/Exit with Mac Engine A/B/A, including reload. Full claimed-role proof remains C6. | | J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | Review PR60 then PR59 against final callers. The focused PR62 video preparation repair entered C1 after actual media dependency testing exposed its blocker. Full PR60/PR59 integration and J5 acceptance remain Required; extensions enter by readiness before freeze. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | diff --git a/docs/audits/2026-09-10-execution-handover.md b/docs/audits/2026-09-10-execution-handover.md index 50353a79c..e6c4bdf7f 100644 --- a/docs/audits/2026-09-10-execution-handover.md +++ b/docs/audits/2026-09-10-execution-handover.md @@ -30,27 +30,28 @@ Before edits, recheck HEAD/tree, dirt, fetched divergence and process ownership. Published review remains `00003b6f`; subsequent integration and Assistant work is local. Closeout documentation follows the code candidate. -The current installed Assistant candidate is -`fa297cb59a07971a472c6020586ea876fdc9a220`, tree -`4c1eaa05448ab2de64c685df4872f7a7d23b245d`. It contains Sash's UI in the canonical -Assistant capsule and protected v2 workspace adoption. The bounded Mac checkpoint -passes one identity, canonical and legacy launch, real Qwen reply/save/reload, -original draft/exact model preservation and full Runtime restart. Shutdown closed -29 owned processes and the held HTTP connection. Runtime, capsule and served -hashes match their current installation receipts. Providers, engine and the six -admitted package files are reused. See -[the convergence check](2026-09-11-assistant-convergence.md) for the acceptance -mapping and the distinction between installed data and three-store fixture proof. - -A separate fresh Mac human Home serves the same candidate and has a verified -locally cached Qwen package. The user has enrolled a passkey account. The private -checkpoint records its URL and receipt. Human review found stale release metadata -for System and Marketplace; installer guard `822c4e3d` and corrected metadata now -pass normal launches in the actual browser. Account and Profile data remain -intact. Human Qwen admission and reply acceptance are next. The earlier recovery human Homes -remain `1e320578`; existing user previews and public stage remain `d790a48e`. -The original recovery, retry, reload, media reuse and Carrier cleanup receipts -remain accepted. Full AUTH-01 human acceptance is still pending. +The current installed candidate is `3c2f9a80012ff510449998f0b423a21c47ad2a98`, +tree `6be3ed9cde01c6dea8850ebd6ac381c5f03a9480`. Runtime built/installed SHA-256 +is `1a55c86d95f5ca286bb63ee65242f3c23e19d0d1acd29888dc976b0033639666`. +It preserves the accepted one-Assistant consolidation and adds pending Keep +intent, normal Marketplace Models rows/details, shared theme tokens and distinct +activity. Both existing Mac checkpoint Homes have matching receipts and all 84 +affected browser files match served bytes. One Runtime build; providers, engine +and admitted Qwen package reused. Installed System Keep readback, actual Qwen +reply/save/reload, original draft/exact model and owned restart pass. Pending +Keep during preparation is source/rendered proof; the installed checks preserve +existing admission. See [the convergence check](2026-09-11-assistant-convergence.md) +for provenance, exact checks and acceptance limits. + +Anders confirms Assistant works. His enrolled account, Profile and conversation +data are preserved. Normal browser controls show the new System and Marketplace +Models views with Qwen available on the device. The earlier System/Marketplace +metadata repair and installer guard `822c4e3d` remain in place. Original recovery +human Homes retain `1e320578`; existing public stage remains `d790a48e`. The private +checkpoint records current process/port ownership; the old Linux preview tunnel +was absent on this resume and was left unchanged. Original recovery, retry, +reload, media reuse and Carrier cleanup receipts retain their exact earlier +verdicts. Full AUTH-01 human acceptance remains pending. The full model donor `6972e165` is merged by `b318bfda`. Sash shelf history is reconciled by `37c82d4f`; Irzhy foundation `617796a9` (PR60 replacing PR57) is @@ -60,10 +61,10 @@ work. The full old-tip merge proposal remains superseded. The [preservation check](2026-09-11-integration-preservation.md) accounts for all 13 September contributor PRs and retains each remaining feature gate. -This implementation checkpoint is closed at its bounded Mac result and waits -for human acceptance. Linux needs the recorded additional disk headroom and a +This user-feedback checkpoint closes at its bounded Mac result. Full human J3 +acceptance remains required. Linux needs the recorded additional disk headroom and a reviewed startup profile. Cold Carrier delivery, Marketplace handoff, -retention/removal, remaining UI differences, Irzhy authority cutover and follow-up, +safe removal, remaining UI differences, Irzhy authority cutover and follow-up, dedicated provider-host/J5 proof and full J1–J5 retain their dependency order and Required gates. Confirmed backend cancellation remains distinct from the accepted honest unknown result. Recurring monitoring remains paused; Jetson is deferred. diff --git a/docs/audits/2026-09-11-assistant-convergence.md b/docs/audits/2026-09-11-assistant-convergence.md index c20abb9fb..aa8007f68 100644 --- a/docs/audits/2026-09-11-assistant-convergence.md +++ b/docs/audits/2026-09-11-assistant-convergence.md @@ -30,7 +30,61 @@ run cursor and complete output history. Reload reads the same run; opening or switching modes creates no run. Chat and Studio keep separate drafts. An absent image/video offer produces the existing unavailable state. -## Installed Mac result +## Current model and activity follow-up + +Candidate `3c2f9a80012ff510449998f0b423a21c47ad2a98`, tree +`6be3ed9cde01c6dea8850ebd6ac381c5f03a9480`, is installed on both existing Mac +checkpoint Homes. Runtime built and installed SHA-256 is +`1a55c86d95f5ca286bb63ee65242f3c23e19d0d1acd29888dc976b0033639666`. +One affected Runtime build took 139 seconds. All 84 browser files in Assistant, +System and Marketplace match source, installed and served hashes on each Home; +current installation receipts bind the Runtime, components and capsule tree. +Providers, engine and admitted model files were reused. + +| Reported behavior | Repair and target proof | Verdict | +| --- | --- | --- | +| Keep during preparation | Runtime saves principal/CID retention intent independently of admission. Eleven retention tests cover replay, restart, promotion, cancellation, failure and shared ownership; rendered System/Marketplace tests cover pending Keep and admission during its response. | Pass source and rendered; installed Homes reuse existing admission | +| Saved local retention | Installed synthetic System toggles Keep, reads it back after refresh and restores the original choice. | Pass Mac installed | +| Normal Models category and consistent controls | Existing Marketplace rows/details host typed model controls; shared theme tokens style System. Installed Marketplace and actual human System/Marketplace show Qwen available locally. | Pass Mac installed and observed human browser | +| Repeated category headings | Single-category and Installed views keep one page heading; Discover keeps its section headings. Rendered category/layout checks pass. | Pass Mac rendered; human Models category observed | +| Repeated Assistant activity | Summary owns current activity; expansion contains distinct history or visible model reasoning. Rendered reasoning/history tests pass. Actual installed Qwen run shows one initial status, then returns `4`. | Pass Mac rendered and installed | +| Preserve replies, drafts and owned resources | Installed reply/save/reload preserves the original draft, exact model and legacy snapshots, with zero implicit runs. Updates close 13 synthetic and 15 human owned processes and release ports; restart succeeds. 148 synthetic and 147 human non-artifact files are byte-identical across installation. | Pass Mac installed | + +After the real Qwen run, the current candidate also closes all 29 owned processes, +closes a held HTTP connection, releases its listener and restarts with the data-root +lock released. This separate receipt binds cleanup to the new Runtime binary. + +Anders reports that Assistant works. The human Home keeps its enrolled account, +Profile and conversation data. Its new Models views were checked through normal +browser controls after the update. Keep was left at the user's existing choice. + +Model UI provenance is explicit: `b0c4c2f5` added the dedicated model screens in +the integration work. The reviewed Sash and Irzhy snapshots contain no separate +Models design. Repeated category headings are present in Sash's `f7c58625` and +were retained by the adaptation; current Marketplace files matched donor +`6972e165` before this repair. This follow-up reuses Sash's existing category and +detail layout. The bounded donor review found no existing remedy to merge. + +Preparation writes and verifies a local model closure. Its byte counter can +advance from already-local Content reads; it does not measure network traffic or +RAM use. Keep protects Runtime's prepared cache from reclamation. A pending choice +survives reconciliation and successful admission; settled cancellation, failure +or expiry clears it when no matching pending/admitted operation remains. It does +not pin IPFS or reserve model memory. The native engine starts or reuses its model +when inference begins. Cold Content/Carrier availability and delivery remain +Required; this run makes no new zero-transfer or remote-availability claim. + +Independent review accepts this bounded Mac result. First-failure diagnostics +retain a stale retired-capsule test assumption, fixture timing/selectors, and +installed script frame/selector repairs. No second Runtime build was needed. +Existing image-load diagnostics remain recorded; this is not whole-UI acceptance. +Linux candidate proof remains pending on disk headroom and its reviewed startup +profile. Marketplace-to-Assistant exact selection handoff, safe removal, cold +Carrier delivery, remaining contributor UI differences and full J1–J5/D1–D6 +keep their gates. Broader execution and monitoring remain paused. Publication, +final package stamping and signing retain their existing approval/release gates. + +## Earlier Assistant consolidation proof Candidate `fa297cb59a07971a472c6020586ea876fdc9a220`, tree `4c1eaa05448ab2de64c685df4872f7a7d23b245d`, passes the bounded installed @@ -60,12 +114,12 @@ old protected workspace bytes remain exact. The test intentionally saved new canonical messages and Home window state; readback proves their contents. A separate fresh Mac Home serves the same candidate for human passkey setup. Its Qwen package was imported from the verified local closure with zero network -download; normal user enrollment and model admission remain to be exercised there. +download. Subsequent enrollment and successful Assistant use are recorded in the current follow-up above. Existing human previews remain unchanged. Independent review accepts this bounded Mac result. Linux convergence remains pending on the recorded disk headroom and startup-profile requirements. Human -acceptance, cold Carrier delivery, Marketplace handoff, retention/removal and +acceptance of the complete journey, cold Carrier delivery, Marketplace handoff, safe removal and full cross-platform J3 retain their Required gates. ## Review and failure disposition diff --git a/state.md b/state.md index fec3d094a..b21b723fe 100644 --- a/state.md +++ b/state.md @@ -13,7 +13,7 @@ changes. The proposed full old-URUX-tip merge is superseded by review of concret remaining behavior differences. The remaining protected-content stack retains its source order and feature-preservation gates in [the integration check](docs/audits/2026-09-11-integration-preservation.md). -The approved Assistant consolidation `fa297cb5` now passes source, rendered and installed Mac checks: one identity, real Qwen reply/save/reload, original draft/model preservation and full Runtime restart. The separate human Home now has an enrolled passkey account. Human review exposed stale System/Marketplace archive metadata; guard `822c4e3d` and corrected installation receipts close that launch gap. Both apps open in the actual user browser, and Marketplace shows Qwen with Use available. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). +The approved Assistant consolidation `fa297cb5` now passes source, rendered and installed Mac checks: one identity, real Qwen reply/save/reload, original draft/model preservation and full Runtime restart. The separate human Home now has an enrolled passkey account. Human review exposed stale System/Marketplace archive metadata; guard `822c4e3d` and corrected installation receipts close that launch gap. Anders confirms Assistant works. Current candidate `3c2f9a80` adds Keep intent during preparation, ordinary Marketplace model rows/details, shared Models theme tokens and distinct Assistant activity. Both Mac Homes have matching Runtime `1a55c86d95f5ca286bb63ee65242f3c23e19d0d1acd29888dc976b0033639666` and 84 served browser artifact records. Installed System retention readback and a real Qwen reply/save/reload pass; the human Home shows Qwen available locally with account and data preserved. Pending preparation is covered by Runtime/rendered tests while installed proof reuses the existing model admission. Original UI parity beyond this slice and full cross-platform J3 acceptance remain open. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). The earlier isolated Mac J3 receipt records Runtime `fbf1a4b0` with matching build, @@ -888,7 +888,7 @@ complete. Installed proof and the atomic authority cutover remain open. saved/reloaded text and model selection, and owned shutdown/restart. Current build/installed/served receipts match. The new human Home has an enrolled passkey account and a locally cached Qwen package. System/Marketplace metadata repair `822c4e3d` passes actual - launch; human model admission and reply acceptance remain pending. Earlier Home Agent and standalone Assistant observations above are + launch. Anders now confirms successful Assistant use; current `3c2f9a80` also passes the bounded Models/activity follow-up described above. Full human J3 acceptance remains pending. Earlier Home Agent and standalone Assistant observations above are historical target receipts. Advanced tools, Library reads, search and broader Studio capabilities retain their typed-contract gates. From 1bb081ff7afce85a2723001166737af9670e8c4b Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 12:06:58 -0500 Subject: [PATCH 300/535] docs: publish integration review and retire website checkpoint --- TASKS.md | 25 +-- docs/audits/2026-09-10-execution-handover.md | 41 ++--- docs/audits/2026-09-11-contributor-review.md | 55 +++++++ docs/audits/2026-09-11-team-sync.md | 162 +++++++++++++++++++ state.md | 39 ++++- 5 files changed, 285 insertions(+), 37 deletions(-) create mode 100644 docs/audits/2026-09-11-contributor-review.md create mode 100644 docs/audits/2026-09-11-team-sync.md diff --git a/TASKS.md b/TASKS.md index 429f97184..1d83ea142 100644 --- a/TASKS.md +++ b/TASKS.md @@ -21,15 +21,18 @@ owns source, installed and public facts. Session handover: [2026-09-10 execution handover](docs/audits/2026-09-10-execution-handover.md). The combined automated installed checkpoint passes; AUTH-01 retains its full -human verdict. Anders accepted the visible review repairs and authorized continued -development on `feat/0.7.1-website-execution`, published at `00003b6f`. See -[the checkpoint report](docs/audits/2026-09-11-first-checkpoint.md). C3/J3 is now -active; recurring monitoring stays paused. Prioritize the two visible Home -assistants, the model loading failure, and then the Browser loading failure. Reproduce -on the installed artifact before donor integration. Preserve user drafts and -identity while converging on the intended single assistant. The closeout review removed five fully preserved branch names (75 to 70); -24 other merged-tip candidates retain historical reflog work. Reconcile donors -and worktrees before merging; these reported failures remain open acceptance. +human verdict. The reviewed combined source `06bf4e0f` is published through +`feat/0.7.1-integration` in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). +The website checkpoint and later donor history are protected there; the redundant +website refs and checkout are retired. The active integration checkout is named +in the private checkpoint. This documentation-only closeout preserves product +code, installed Homes and acceptance criteria. The [team report](docs/audits/2026-09-11-team-sync.md) +and [contributor review](docs/audits/2026-09-11-contributor-review.md) describe +included work and remaining gates. C3/J3 pauses after the bounded Mac checkpoint +and publication. Next: exact Marketplace-to-Assistant handoff and safe removal, +then cold Content/Carrier delivery. Browser follows its named queue. Preserve the +remaining 69 branches and 11 worktrees under their existing history/owner gates; +recurring monitoring stays paused. This is the current execution queue. Older references below are retained follow-up obligations, not permission to bypass this order or the release gate. @@ -44,9 +47,9 @@ The C1-C7 sequence in the approved plan controls integration and publication. | C1-delivery | Signed three-platform candidate installation | Functional preparation complete; final assembly after C5 freeze / delivery owner | Native inputs, bootstrap and publisher import have bounded source/artifact proof. Keep working builds for functional tests and rebuild only affected components. Complete atomic promotion, final installer stamping, platform regeneration and signed fresh installs at the final release boundary. | | J1 / C2 | Install, passkey, Recovery Kit, Profile and usable Home | Human passkey proof pending / Home and installer owners | Registration binding and atomic/conflict-safe identity persistence are integrated with 66 focused tests passing. Durable owner intent and Profile-inclusive recovery now pass source and UI checks. Real signed Mac installation and seed export pass. Source repairs for kit-first recovery, gateway shutdown and media-tool reuse are accepted. Both targets pass installed browser recovery/reload/retry, complete owned shutdown/restart, full Home setup/media reuse and Carrier cleanup/reuse. User-review account label, Recovery/Advanced spacing and cross-port session isolation are repaired on `1e320578`; shared-browser Mac/Linux proof passes. Next: AUTH-01 final user acceptance on the updated preserved Homes; concurrent-save and window/selection UI; W2-W5; rerun the smoke with System entry through the ElastOS menu, then complete the app matrix; full Linux/Mac staging. Publish J1 for review first under D1. | | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | -| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Active / coordinator owns source, builds and tests; independent review | Continue on the published website branch. The preserved human Home retains its earlier zero-offer observation and duplicate Assistant entry points; the separate Mac model test Home now has an admitted Qwen offer. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Anders confirms Assistant works. Current `3c2f9a80` repairs pending Keep intent, normal Marketplace Models category/details, theme tokens and duplicate activity/headings. Source/rendered checks and installed Mac retention readback, Qwen reply/save/reload and human model-view observation pass, with matching receipts and user data preserved. Pending preparation itself uses source/rendered proof; the installed model was reused. Linux still needs disk headroom and a reviewed startup profile. Next required work is exact Marketplace-to-Assistant handoff, safe removal and cold Carrier delivery. Broader execution and monitoring pause at this user-feedback checkpoint; full human J3 remains required. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | +| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Paused after checkpoint publication / coordinator; independent review | Continue on `feat/0.7.1-integration` after a new implementation instruction. Draft PR64 contains the complete combined checkpoint; the human Mac Home now has one Assistant and admitted Qwen. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Anders confirms Assistant works. Current `3c2f9a80` repairs pending Keep intent, normal Marketplace Models category/details, theme tokens and duplicate activity/headings. Source/rendered checks and installed Mac retention readback, Qwen reply/save/reload and human model-view observation pass, with matching receipts and user data preserved. Pending preparation itself uses source/rendered proof; the installed model was reused. Linux still needs disk headroom and a reviewed startup profile. Next required work is exact Marketplace-to-Assistant handoff, safe removal and cold Carrier delivery. Broader execution and monitoring pause at this user-feedback checkpoint; full human J3 remains required. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | | J4 / C4 | Browser works in the two agreed placements | Planned / Browser owner | Retain canonical B01-B16 on the Browser branch; Mac A/A/A and Linux Home/Exit with Mac Engine A/B/A, including reload. Full claimed-role proof remains C6. | -| J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | Review PR60 then PR59 against final callers. The focused PR62 video preparation repair entered C1 after actual media dependency testing exposed its blocker. Full PR60/PR59 integration and J5 acceptance remain Required; extensions enter by readiness before freeze. | +| J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | PR60 original history is integrated in the published candidate; coordinate its open PR disposition with Irzhy. The focused video preparation repair adopted during C1 comes from the foundation inherited by PR62, rather than its four follow-up commits. Review PR59 authority cutover against final callers, then take PR62 work by readiness. J5 installed/human acceptance remains Required; extensions enter by readiness before freeze. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | | C6 | Accept the combined signed candidate | Planned / independent reviewer and human testers | All required journeys, shared Home/app regression, Browser qualification and first-hop update proof on matching artifacts. | | C7 | Publish and verify the accepted release | Planned / release owner | C6 and explicit approvals, accepted main tree/artifact identity, then public delivery checks. | diff --git a/docs/audits/2026-09-10-execution-handover.md b/docs/audits/2026-09-10-execution-handover.md index e6c4bdf7f..bd3a6f88d 100644 --- a/docs/audits/2026-09-10-execution-handover.md +++ b/docs/audits/2026-09-10-execution-handover.md @@ -8,9 +8,10 @@ shared-browser reload/refresh/sign-out check and show the exact restored name. follow-up proof from reused lifecycle/media receipts. The two isolated Homes preserve the user's restored data and retain the full AUTH-01 human gate. Anders accepted the visible repairs, published the branch at `00003b6f`, and resumed -C3/J3 on that branch. Recurring monitoring stays paused. Full J1–J5 and D1–D6 -acceptance remains intact; the existing branch replaces the proposed extra J1/J3 -working branch names under the user's latest instruction. +C3/J3 on that branch. The combined checkpoint is now published through the +existing integration branch in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). +The website refs and checkout are retired after remote protection and ownership +checks. Recurring monitoring stays paused. Full J1–J5 and D1–D6 acceptance remains intact. ## Resume here @@ -23,12 +24,15 @@ local operator ledger and current checkpoint. Resolve their directory with `git rev-parse --path-format=absolute --git-common-dir`; read `development-loop-current.md` there. A linked worktree's `.git` is a file. -Use the explicit website checkout on `feat/0.7.1-website-execution`, based on -the user-selected `origin/upstream/0.7.1-dev` at `6c61c990`. The default checkout -contains unrelated dirty donor work. Preserve all 70 branches and 12 worktrees. -Before edits, recheck HEAD/tree, dirt, fetched divergence and process ownership. -Published review remains `00003b6f`; subsequent integration and Assistant work is -local. Closeout documentation follows the code candidate. +Use the existing integration checkout on `feat/0.7.1-integration`, based on +user-selected `origin/upstream/0.7.1-dev` at `6c61c990`. Resolve its exact local +path from the private checkpoint. Reviewed source `06bf4e0f`, tree `4483f403`, +is published in draft PR64 with this documentation-only closeout. Both website +heads remain ancestors of the verified remote integration ref. The default +checkout contains unrelated dirty donor work. Preserve the remaining 69 branches +and 11 worktrees. Before edits, recheck HEAD/tree, dirt, fetched divergence and +process ownership. [The team report](2026-09-11-team-sync.md) and +[contributor review](2026-09-11-contributor-review.md) are the shareable closeout. The current installed candidate is `3c2f9a80012ff510449998f0b423a21c47ad2a98`, tree `6be3ed9cde01c6dea8850ebd6ac381c5f03a9480`. Runtime built/installed SHA-256 @@ -70,7 +74,7 @@ Required gates. Confirmed backend cancellation remains distinct from the accepte honest unknown result. Recurring monitoring remains paused; Jetson is deferred. Further public deployment requires review and approval of the exact candidate. -The seed's existing GitHub credentials remain available for a future authorized +The private publication record owns the verified route for a future authorized push. Required installation manifests and receipts are current; final installer stamping, packaging and signing follow C5 source freeze and the release gate. @@ -119,12 +123,12 @@ obligations retain their release gates while that source intake proceeds. 2. **Deliver the public preview:** approve and deploy the reviewed storefront and seed Runtime, preserve existing accounts, then verify public artifacts. Public signed installer delivery remains a separate gate. -3. **One current Home assistant and working local AI (J3):** Anders reports two - Home assistants, one outdated, and a model that does not load. Reproduce on - the actual installed path, map both surfaces to their source/donor owners, - preserve drafts/data, and converge on the intended single experience. - Prove model Get, local reply, stop, restart/reuse and removal through the - canonical typed Model and Content/Carrier contracts. +3. **Complete the local AI journey (J3):** one Assistant and real Qwen replies + now pass the bounded Mac checkpoint. Next prove exact Marketplace-to-Assistant + handoff and safe removal, then cold Content/Carrier delivery. Preserve drafts, + model identity and run ownership. Full human/Linux acceptance, confirmed + cancellation, cold transfer integrity/retry and shared regressions keep their + existing gates; the earlier duplicate/loading diagnostics remain historical. 4. **Working Browser (J4):** Anders reports that Browser does not load. Start with that installed failure, then integrate the canonical Browser work and prove the agreed Mac-local and Linux Home/Exit with Mac Engine placements. @@ -150,7 +154,8 @@ and reflog revisions remain reachable from retained refs, with an exact restorat map in the private cleanup receipt. Another 24 merged-tip candidates retain historical reflog work and need review before removal. -All 12 worktrees remain. The active website checkout is clean after the checkpoint closeout commit; the +The authorized publication retirement leaves 11 worktrees and 69 branches. +The active integration checkout owns further work; the root documentation, Browser maturity and Home URUX worktrees contain preserved dirty work. The detached Browser build is contained in `fix/browser-maturity`. Three historical backup refs and the clean review/build worktrees retain their @@ -162,7 +167,7 @@ owning journey. Preserve semantic conflict resolutions, especially provider bounds/lifecycle, Documents `if_revision`, Profile authority and the single components cutover. Compare history as well as tree bytes before retiring a ref. The next session must not declare the whole repository clean based only on -the website worktree. Keep at least 10% free disk space. +the integration worktree. Keep at least 10% free disk space. Runtime owns authority, networking and lifecycle; signed Profile authority owns identity/name; Content/Carrier owns model delivery. Remove obsolete surfaces diff --git a/docs/audits/2026-09-11-contributor-review.md b/docs/audits/2026-09-11-contributor-review.md new file mode 100644 index 000000000..ef60633fa --- /dev/null +++ b/docs/audits/2026-09-11-contributor-review.md @@ -0,0 +1,55 @@ +# ElastOS 0.7.1: contributor branches and PRs + +**Checked 11 September 2026.** Companion to the [5 to 11 September team report](2026-09-11-team-sync.md). GitHub API metadata, local ancestry and selected patch/tree comparisons were checked against delivery candidate `06bf4e0f`, tree `4483f403a85d1ac27b4568f7b8cb4cf1a8a4e546`. + +At publication, the reviewed candidate was 79 commits ahead and 0 behind the earlier public website checkpoint at `00003b6f`; both histories are now protected by integration. Local adoption and GitHub PR state are shown separately. “Ancestor” means the original commit is in candidate history. “Adapted” means work was transferred or revised through a different commit. Neither term certifies complete installed behavior. + +The existing `feat/0.7.1-integration` branch was fast-forwarded to this candidate and published in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) into `upstream/0.7.1-dev`. Remote verification protected both website heads before the redundant local/remote website refs and checkout were removed. This companion accompanies the single documentation-only closeout; PR64 records its final commit and immutable report links. PR58's merge remains in history; PR60's original head is included in the published candidate; PR59 and PR62 remain pending. All contributor PR heads, bases and states were preserved. + +## September inventory + +The existing 13-PR inventory covers PRs created or updated since **1 September**, so it includes work older than this report's 5 to 11 September delivery window. Sash is `SashaMIT`; Irzhy is `irzhywau`. PR authorship and individual code authorship can differ, particularly in integration PRs. + +| Contributor / PR | Head branch and commit | GitHub state | Interpretation at `06bf4e0f` | +| --- | --- | --- | --- | +| Sash [19](https://github.com/Elacity/elastos-runtime/pull/19) | `chore/ci-source-home-matrix` · `1d6b49e3` | Closed, unmerged | Separate original ancestry. Current CI retains three-platform source installation, manual ref selection and installed-provider verification. macOS Clippy/tests moved to Linux jobs, and named feature push triggers became PR/manual/main/tag triggers. These differences need CI policy review. | +| Sash [23](https://github.com/Elacity/elastos-runtime/pull/23) | `feat/home-urux-on-freeze` · `5e546ef4` | Closed, unmerged | Adapted UI entered through `8b547590`, Irzhy's `7dd1780b` and merge `985fdffc`. Later Sash dock/lock changes are also ancestors. Compare remaining behaviors; the old tip is not the next merge target. | +| Sash [26](https://github.com/Elacity/elastos-runtime/pull/26) | `feat/gba-nonogram-advance` · `7c622705` | Open | Adopted through `1fd30b38`, with separate original ancestry. ROM, MIT licence and SVG blobs match exactly. Current icon metadata/assets and attribution reflect later changes. | +| Irzhy [38](https://github.com/Elacity/elastos-runtime/pull/38) | `upstream/0.7-dev` · `e481b153` | Merged | Original release-source head is an ancestor. Historical foundation with September activity, rather than new implementation this week. | +| Irzhy [51](https://github.com/Elacity/elastos-runtime/pull/51) | `upstream/0.7.1-dev` · `6c61c990` | Open | The head is the public dev baseline and an ancestor. This PR releases dev to main after candidate acceptance. | +| Irzhy [52](https://github.com/Elacity/elastos-runtime/pull/52) | `feat/protected-content-installed-provisioning` · `4d688cc5` | Merged | Original head is an ancestor. Custody/chain provisioning and startup reconciliation are included. | +| Sash [54](https://github.com/Elacity/elastos-runtime/pull/54) | `feat/home-first-run-seed-0.7.1` · `2a49ea57` | Merged | Original head is an ancestor. Includes first-run seed, lock face, Documents/Library/chrome fixes, dock behavior and capsule-owned service icons. | +| Sash [55](https://github.com/Elacity/elastos-runtime/pull/55) | `feat/home-shelf-assistant-face-0.7.1` · `923193bb` | Closed, unmerged | Adapted content arrived through integration. Local merge `37c82d4f` records original ancestry. Three patches have matching stable patch IDs; the original and adopted harness commits have the same capsule tree. | +| Irzhy [57](https://github.com/Elacity/elastos-runtime/pull/57) | `feat/protected-content-installed-e2e-proof` · `617796a9` | Closed, unmerged | Replaced by PR60 after retargeting/rebase. Both now identify the same head. Count this implementation once. | +| Irzhy [58](https://github.com/Elacity/elastos-runtime/pull/58) | `feat/0.7.1-integration` · `5ba1faa0` | Merged | Published head is an ancestor. Local merge `b318bfda` incorporates the later donor at `6972e165` too. | +| Irzhy [59](https://github.com/Elacity/elastos-runtime/pull/59) | `feat/protected-content-atomic-cutover` · `25ab205e` | Open | Head is outside candidate ancestry. Follows PR60. Runtime authority cutover and API removal need review against the current callers before integration. | +| Irzhy [60](https://github.com/Elacity/elastos-runtime/pull/60) | `feat/protected-content-installed-e2e-proof` · `617796a9` | Open | Full original history is merged locally by `dd21d8bd`. Includes provider hosting, peer seeding, availability/media repairs and installed-proof tooling. Its GitHub PR remains open. | +| Irzhy [62](https://github.com/Elacity/elastos-runtime/pull/62) | `feat/protected-content-0.7.1-followup` · `06179578` | Open, draft | Head is outside candidate ancestry and follows PR59. Its four commits contain the plan, wiring/cleanup, Creator and non-media protection/read support. Reader capsule, audio and crypto-review completion remain pending. | + +## Earlier sources needed to interpret 0.7.1 + +| Contributor / PR | Head branch and commit | GitHub state | Interpretation | +| --- | --- | --- | --- | +| Irzhy [39](https://github.com/Elacity/elastos-runtime/pull/39) | `feat/protected-content-uiux-reconstruction` · `7dd1780b` | Merged | Ancestor through `985fdffc`. Carries adapted Sash UIUX plus Archive Manager reconstruction, typed Assistant workspace, Home menu protocol and CI work. | +| Irzhy [43](https://github.com/Elacity/elastos-runtime/pull/43) | `fix/protected-content-mint-intent-adoption` · `58ebfb23` | Merged | Ancestor. Preserves completed mint work after a lost completion mark. | +| Irzhy [15](https://github.com/Elacity/elastos-runtime/pull/15) | `feat/dkms-esp-port` · `27d85c6f` | Open | Separate ancestry. The project retains an extraction ledger for protected-content behavior. Remaining parity and audit-migration decisions need explicit disposition; a wholesale merge is not implied. | +| Sash [17](https://github.com/Elacity/elastos-runtime/pull/17) | `feat/model-provider-rebase` · `9f423796` | Open | Separate ancestry. Credit the earlier model-provider work. The candidate uses a later typed-provider implementation; complete feature equivalence requires a scoped comparison. | +| Irzhy [25](https://github.com/Elacity/elastos-runtime/pull/25) | `feat/elastos-logger` · `66c8bba4` | Open | Separate ancestry. The current candidate has no `elastos-logger` crate. This independent logging proposal needs an explicit scope/disposition decision. | + +Earlier protected-content extraction work is represented in the base source and PR15 ledger. This review checks the active intake interpretation and the relevant older open tracks; it does not claim feature-by-feature equivalence across every historical PR. + +## Integration details and remaining decisions + +**Sash UI preservation.** The original shelf/composer/Agent Space series was authored on 3 September. This week recorded its original ancestry and retained its UI in the unified Assistant. Patch comparisons and capsule-tree identity support that preservation: original `239c6cb7` and adopted `77498557` share tree `c03166c4489f6773792c38c337f7ccb5e714f6ce`. The URUX adaptation chain and newer dock/lock commits are also present. Review remaining Viewer rail, pager/switcher, Mission Control, launcher motion and workflow differences against the current Runtime contracts before selecting further UI work. + +**Irzhy installed evidence.** The 5 to 7 September protected-content test used an installed client, three custody containers and an Anvil/Base fork. Its final receipt records passing mint, availability, buy, 2-of-3 release, ordered reads, close, denial/tamper/replay, restart and cleanup phases. The proof used headless principals; final Brave interaction, independent operator/hardware domains and real Base remain separate. Its receipts bind the binaries actually tested, which predate later lint/test edits. Commit `617796a9` retains a 5 September author date, an 8 September committer date, and entered the delivery branch on 10 September. + +**PR62 scope.** The diff from `25ab205e` to `06179578` contains no media-provider changes. Media code present at the PR62 tip comes from its inherited stack. The delivery branch's media repair therefore does not establish adoption of PR62 follow-up work. Creator source exists at its current head; a Reader capsule does not. Required video-journey repairs remain required under D2, while optional additions enter only with source and installed evidence before freeze. + +**Models UI ownership.** Dedicated Models screens came from integration commit `b0c4c2f5`. The bounded source review found no separate Sash/Irzhy Models design to restore. The latest repair reuses Sash's general Marketplace category/detail layout and shared theme. This separates credit for the design foundation from responsibility for the later model controls. + +Before closing contributor intake, the team needs dispositions for the older open tracks, the remaining UI differences and PR59/PR62 readiness. A clean merge or a closed PR cannot make those decisions on its own. + +## Check method and limits + +GitHub API supplied PR states, heads, branches and descriptions. Local Git supplied ancestry, commit dates, stable patch IDs, selected blob/tree identities and the PR59-to-PR62 diff. The preservation and convergence audits supplied installed-evidence scope. This review ran no builds or installed tests and changed no contributor branches, PRs or acceptance checkboxes. diff --git a/docs/audits/2026-09-11-team-sync.md b/docs/audits/2026-09-11-team-sync.md new file mode 100644 index 000000000..d4926dd9b --- /dev/null +++ b/docs/audits/2026-09-11-team-sync.md @@ -0,0 +1,162 @@ +# ElastOS 0.7.1: weekly team report + +**5 to 11 September 2026.** Source snapshot: `06bf4e0f`. Publication status: the reviewed checkpoint is published on `feat/0.7.1-integration` in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). This report accompanies the single documentation-only closeout; the PR records its final commit and immutable report links. Calendar dates use America/Panama unless a source explicitly gives UTC. Earlier contributor work is identified as work integrated or reviewed during this period. + +The period produced first-owner and recovery work, a published website/setup review branch, passing automated recovery and shutdown checks on Mac and Linux, and one installed Mac Assistant with real model replies. Irzhy also recorded a full headless protected-content journey in the local custody/chain test environment. Model, Home and protected-content contributions were brought into the delivery branch while preserving their history. Full 0.7.1 release acceptance remains open. + +The combined source is published through `feat/0.7.1-integration` for team review. It contains the website checkpoint, the complete later local integration history and the Assistant/model repairs. Remote verification protected both website heads before its local and remote branch refs and redundant checkout were removed. Irzhy's merged PR58 remains in history; PR59 and PR62 remain pending. The installed Mac Home retains its tested product code. The next implementation slice is exact Marketplace-to-Assistant handoff and safe removal, followed by cold Content/Carrier delivery. + +The remaining release work is to complete the five agreed user journeys on one frozen candidate. This report separates completed implementation, recorded tests and acceptance still required. + +## Foundations completed on 5 and 6 September + +**First-owner setup.** Commits `145fec2b`, `eb25f747` and `43b8f830` introduced durable enrollment shared by the registration routes, bound ceremonies to the correct origin/relying party, and made credential persistence atomic and conflict-safe. Public HTTPS enrollment uses a one-use operator token; the same claimant can resume after a lost response or restart. On 6 September, Anders completed Touch ID enrollment in an isolated Mac Home. The record showed one Admin, root, enrollment event and consumed operation, with owner state preserved after restart. The public HTTPS path retained its own acceptance gate. + +**Complete recovery.** An installed setup run found that a Recovery Kit could predate Profile creation. Repair `abefc7ae` made System create or retain the Profile and export its identity through one complete-kit action, with coverage recorded after successful export. Tests covered restoration, concurrent creation, failed export and retry. The later Mac/Linux kit-first recovery checks extend this work. + +**Local AI and run ownership.** The 5 to 6 September series added verified engine/model inputs (`c95cf4c9`), managed local-engine startup/recovery (`7c4fc929`), hosted-backend evidence (`fcb8fc5e`) and a Responses adapter with explicit cancellation outcomes (`0d768415`). Home Agent preserved unresolved drafts and uncertain create outcomes instead of dispatching again. Anders observed a complete Qwen reply on the isolated Mac installation. One journal-bound request completed in 111.2 seconds. Stop on another run recorded one dispatch and one unknown terminal outcome. Hosted support still needs its optional release acceptance. + +The 5 September product and identity assessments also proposed a useful document-and-AI journey and a clear separation between personal identity, device keys, permissions and capsule identity. These were design recommendations. They did not change the five-journey release scope or establish new identity/cryptographic support. + +## Work delivered this week + +| Area | Work completed | Evidence and scope | +| --- | --- | --- | +| Release planning and QA | Reconciled the release around five user journeys, six scope decisions and one execution sequence. Triaged 38 QA rows against their actual versions and environments. | The Notion plan separates required release criteria, optional additions and later work. The QA review distinguished current defects, old-channel results, test-harness issues and items outside this release. | +| Website and installation | Added the storefront and canonical `/home/` entry, preserved old Home routes with redirects, and made preview availability explicit. Improved signed bootstrap verification, platform selection, packaging and automatic Home setup. | Website/setup source is published for review at `00003b6f`. Anders used the signed private Mac preview installer. Native preparation passed on Mac, Linux x86_64 and Linux ARM, with all three input sets verified. Final regeneration, signing and fresh-install acceptance remain part of the release gate. | +| Identity and recovery | Made Recover select a kit first. Added complete Profile recovery, original DID/name restoration, retry and continuation after reload or a fresh sign-in. Fixed restored account labels, recovery layout and session-cookie isolation between local Homes. | The combined automated checkpoint passed on Mac ARM64 and Linux x86_64. Wrong-password attempts preserved account data. Anders reviewed the visible follow-up repairs. The complete human authentication test remains pending. | +| Home saves and window behavior | On 7 September, protected app saves gained concurrent-write guards (`84723559`), and capsules gained their own window and text-selection policies (`cdb0492d`, `11ab93ff`). The storage/access design was recorded in `e221ae50`. | Source changes are integrated. Final two-window save conflicts, app/window behavior and broader storage implementation retain the checks assigned in the plan. | +| Runtime and setup lifecycle | Closed held gateway connections and awaited owned process groups during shutdown. Repaired Carrier endpoint cleanup and reused verified media-tool downloads. | Matching Mac/Linux checkpoint artifacts passed recovery, shutdown/restart, media reuse/integrity and the recorded Carrier checks. Source-only fault cases retain separate verdicts. | +| Protected content and video | Irzhy's 5 to 7 September installed test completed mint, availability, purchase, open/read/close, custody/replica drills, denial/tamper/replay cases and restart cleanup. Delivery work later took media repairs from the shared protected-content stack: Runtime invocation compatibility, zero-based DASH segments and codec preservation. | The earlier finalize receipt reports all phases passing on three custody containers and an Anvil/Base fork, with headless principals. Linux amd64 CI provisioning/preflight passed on 7 September after a bind-mount ownership repair. The extracted media code passed provider/Runtime tests. Final Brave, real-Base and independent-operator claims retain separate gates. | +| Model foundation | Integrated the full model donor, including bootstrap integrity, signed catalog access, admission, preparation, startup, inventory and lifecycle work. Repaired the observed readiness and local-verifier boundaries. | Focused source tests and a real installed Mac Qwen reply/lifecycle/reuse checkpoint passed. The required delivery and management journey still has open acceptance steps, listed below. | +| One Assistant | Consolidated the public Assistant identity while retaining Sash's Agent Space, shelf composer and conversation UI. Added protected workspace adoption, complete history/draft preservation, visible conflict copies and typed Chat/Build/Studio controls. | Local code `fa297cb5` passed source, rendered and limited installed Mac checks. A real Qwen response appeared, saved, and survived reload and full Runtime restart with the original draft and exact model selection. | +| Model controls and activity | Follow-up `3c2f9a80` retains Keep intent during preparation, uses the existing Marketplace category/detail layout and gives System consistent styling. `d3d757ce` separates current activity from previous activity and visible reasoning. Source-install guard `822c4e3d` validates copied capsule metadata. | Both Mac checkpoint Homes have the follow-up installed. System Keep readback and real Qwen reply/save/reload pass; the new model views were observed in Anders's browser. Pending Keep has Runtime/rendered race tests; the installed checks reused an admitted package. | +| Contributor integration | Preserved the full model donor history, recorded Sash shelf ancestry and merged Irzhy's protected-content foundation locally. Rechecked the 13-PR September inventory and earlier sources relevant to 0.7.1. | The companion contributor review distinguishes GitHub state, local ancestry, adapted changes and outstanding behavior. The September inventory starts on 1 September; it is broader than this report's delivery window. | +| Development process | Revised the development-loop skill to version 0.4.1. Added installed-result checkpoints, effort review and stronger plan/dependency checks. | The written workflow is improved. Measured cost savings remain to be demonstrated. | + +Early-week Browser work also produced useful evidence. The 7 September investigation reproduced missing-artifact readiness errors, input problems and fragile recovery. Later records include bounded local browsing/media/input proof in run 92, approved inspection/fill/revoke actions in run 93, and Linux Home/Exit with a Mac Engine in run 96. That remote run confirmed ordinary actions and all 13 close effects, but failed its five-second viewer reload gate. These are historical results on their named artifacts; the combined release still needs Browser acceptance. + +## What the Assistant checks establish + +The consolidation has seven Runtime storage tests, ten migration tests, 26 run/recovery tests and eight Studio tests, plus source and rendered checks. Fixtures cover all three legacy stores, long Unicode histories, drafts, unknown fields and concurrent saves. Runtime preserves original objects and snapshots and prevents old writers from overwriting adopted state. + +Installed Mac adoption exercised existing Home Agent data. The consolidation check verified one visible Assistant, legacy launch routes and preservation of the selected content/offer and user work. Its parity record matched 118 affected artifact records. The later `3c2f9a80` check separately verified the current Runtime's 29 owned processes closing, connection/listener release and restart. All 84 affected Assistant/System/Marketplace browser files match source, installed and served bytes on each Mac Home. + +All-three-store migration has source and rendered proof; the installed migration used one legacy data set. Installed Studio covers mode controls and the honest absence of image/video offers. Generation and foreign-run protection have source/test evidence. Anders has since reported successful Assistant use, and the update preserved his enrolled account, Profile and chats. Full human J3 and Linux qualification remain required. + +Keep protects Runtime's prepared cache from reclamation. It does not pin Content in IPFS or reserve model memory. Preparation progress counts verified local package bytes and can use content already on disk. These checks establish local preparation and inference; ordinary cold Carrier delivery remains a separate requirement. + +## Source, installed candidate and publication + +| Surface | Recorded version | Meaning for the team | +| --- | --- | --- | +| Public `main` | `8ac18bec`, also the target of tag `v0.7.0` | Existing release source. | +| Public `upstream/0.7.1-dev` | `6c61c990` | Current public integration baseline. | +| Historical website checkpoint | `00003b6f`, tree `7d816617` | Published earlier; its full history remains reachable through integration after the website ref was retired. | +| Reviewed integration checkpoint | `06bf4e0f`, tree `4483f403` | Published in draft PR64: 134 commits ahead and 0 behind public dev at this source checkpoint. The documentation-only closeout follows without changing product code. | +| Earlier local integration donor | `6972e165` | Its complete history is included. The existing integration branch was fast-forwarded to the combined candidate and is now the continuing working/review line. | +| Installed Assistant/model candidate | Code `3c2f9a80`, tree `6be3ed9c` | Current recorded candidate on the two Mac checkpoint Homes. It includes the earlier consolidation and later metadata, model-control and activity repairs. | +| Earlier isolated recovery Homes | Code `1e320578` | Separate Mac/Linux recovery evidence and preserved user-review state. | +| Existing previews/public stage | Recorded as `d790a48e` in the handover | Earlier artifacts with their own receipts. The new Assistant checkpoint does not update their verdict. | + +The human-review Mac Home runs code `3c2f9a80`. The reviewed source head `06bf4e0f` differs only in four Markdown files. Runtime hash verification and the 84 affected browser-file comparisons passed. Publishing the source branch preserves this installed Home and its user data. Detailed audits bind the Runtime, components, capsules and served files to their tested versions. + +The result is one active `feat/0.7.1-integration` working/review branch locally and on GitHub, with [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) into `upstream/0.7.1-dev`. The website branch was removed locally and remotely after exact ancestry and remote-head verification. Its checkout was removed after tracked/untracked/ignored-file, evidence, symlink and process checks. Only rebuildable caches were discarded; the historical receipts remain available. The inventory now contains 69 local branches and 11 worktrees. Remaining donor histories and dirty worktrees keep their existing owners and cleanup gates. This publication preserves `main`, public dev, PR58's merge and installed services. Contributor PR closure or retargeting requires separate coordination. + +The website checkpoint and later combined work are now protected by the public integration history. Release PR51, the 0.7.1 tag, stable artifact promotion and public deployment follow acceptance and approval. Public refs were checked directly. The report review reused product evidence and did not qualify the public site's current bytes. + +[Published website checkpoint](https://github.com/Elacity/elastos-runtime/tree/00003b6f0771138fbb1a5d834de7c027feaee508) · [Published checkpoint evidence](https://github.com/Elacity/elastos-runtime/blob/00003b6f/docs/audits/2026-09-11-first-checkpoint.md) · [Release PR51](https://github.com/Elacity/elastos-runtime/pull/51) + +## Notion plan and current acceptance + +The governing plan is [ElastOS 0.7.1 release plan: five user journeys from start to stop](https://app.notion.com/p/wauio/ElastOS-0-7-1-release-plan-five-user-journeys-from-start-to-stop-3d6b682adcca81948f78d12abcd677b9). It was compiled on 9 September, reconciled on 10 September, and updated through the 11 September Assistant checkpoint. + +The tables below preserve all five journeys, six decisions and seven execution checkpoints. The full criteria remain in Notion. Its C3 and J3 evidence rows reflect `3c2f9a80`; some J1/source-history wording needs reconciliation with the later integration. One acceptance decision needs clarification: the J3 overview asks for confirmed cancellation, while detailed step 5 permits confirmed settlement or an honest unknown. The team must resolve that wording before accepting Stop behavior. + +| Journey | Required user outcome | Position at this checkpoint | +| --- | --- | --- | +| **J1: Install and first setup** | Install on Linux x86_64, Linux aarch64 and Mac Apple silicon; create or recover the protected Profile; save a complete Recovery Kit; reach a coherent first-run Home and lock/unlock again. | Website/setup source and Mac/Linux recovery checkpoints advanced. Full clean-install coverage, normal Chrome/Safari authentication, remaining setup/help/docs checks, shared Home behavior and an outside tester remain required. | +| **J2: Update** | Start from the actual 0.7.0 release updater, discover and apply the signed candidate, restart and preserve identity, passkeys, Profiles, content and drafts. | Updater and packaging source repairs exist. Actual first-hop and interruption tests on the frozen platform set remain open. A source-home replacement has a separate verdict. | +| **J3: Obtain and use local AI** | Discover signed model content in Marketplace, admit it through ordinary controls, select the exact offer, receive a reply, recover the run safely, reuse it and remove it with consent. | Mac replies, saved work, restart and Keep readback pass; Anders reports successful Assistant use. Exact Marketplace handoff, safe removal, cold Content/Carrier delivery, Linux proof and full human/shared acceptance remain. Earlier Stop evidence records an honest unknown; confirmed backend cancellation remains unproven. | +| **J4: Browser** | Complete open, navigation, video, audio, input, reload and close both on the Mac locally and with Linux Home/Exit using a Mac Engine. | Bounded earlier runs are recorded. Remote reload, matched profile persistence/upload proof, performance/media evidence and final qualification remain open. | +| **J5: Protected video** | A creator protects, mints and lists a video; a separate buyer buys, opens, plays and closes it in Brave on one Runtime with an Anvil/Base fork; settlement and recovery are verified. | Irzhy's earlier headless installed journey and the extracted video-preparation repair have recorded results. The authority cutover, selected follow-up work and Brave journey on the final candidate remain required. External cryptographic review needs a recorded owner, scope and date; issue #48 stays open through review disposition. | + +### Decisions retained from Notion + +| Decision | Agreed scope | +| --- | --- | +| **D1** | Publish for review journey by journey, starting with J1. J2 shares delivery work; J3 depends on J1. Execution continued on the website branch. The authorized publication consolidates that accumulated work into the existing integration branch while preserving its history and the journey acceptance gates. | +| **D2** | Take PR62 additions by readiness at source freeze. Repairs required for the video journey stay required. Optional additions need source and installed proof before freeze; unfinished additions move to 0.7.2. | +| **D3** | A model is signed content delivered through Content and Carrier. Cold delivery is required. Hosted adapters and a Codex provider are optional after the local path. Remote inference, hosted sharing and Jetson are later work. | +| **D4** | Browser uses one capsule and Runtime contract for Mac-local use and Linux Home/Exit with a Mac Engine. Both placements, including reload, require qualification. | +| **D5** | Ordinary installation and update parity are required on Linux x86_64, Linux aarch64 and Mac Apple silicon. A `.dmg` is later work. | +| **D6** | Incorporate PR63's verified five-journey frame with the current J3 queue and canonical Browser IDs. Close PR63 as superseded after its replacement is verified and the action is approved. | + +### Execution sequence retained from Notion + +| Checkpoint | Deliverable and dependency | Current position | +| --- | --- | --- | +| **C1** | Truthful storefront, canonical Home entry and isolated signed candidate delivery. Concrete seed deployment needs review and approval. | Source and private/staged delivery advanced; final delivery gates remain. | +| **C2** | Complete J1 setup work, preserve recovery/session/lifecycle behavior, carry J2 as coherent update commits and publish reviewed evidence. | Review prefix published; complete human/shared J1 acceptance remains open. | +| **C3** | Deliver required J3 on the accepted J1 behavior, with independent review and human acceptance. | Mac Assistant plus model/activity follow-up recorded; broader J3 criteria remain. | +| **C4** | Finish Browser source and installed proof for both required placements. It can run alongside C3 when target/build resources are available. Hand the tested updater slice to J2. | Browser slice and qualification remain pending. | +| **C5** | Assemble reviewed journeys into dev, take PR60 then PR59 and ready PR62 work, stamp version/changelog and freeze the exact source and optional scope. | Pending. The local PR60 foundation merge is partial intake, rather than completion of assembly. | +| **C6** | Build/sign one compatible three-platform set and run all required journey, update/interruption and shared checks on that frozen candidate. | Pending. A repair records a new candidate and repeats affected acceptance. | +| **C7** | After acceptance and approval, merge PR51, verify tree identity, tag and publish accepted artifacts, then verify canonical public delivery and close the hygiene record. | Pending. | + +J4 and J5 can proceed beside J3 only when their owners have independent source boundaries and available build/test resources. Final packaging follows source freeze. Optional work cannot postpone the freeze. + +The shared gate includes Create/Recover, Profile and kit coverage, restart, window policies, deep links, drafts, picker acknowledgements, two-window Documents saves, GBA save conflicts, Archive/Library, Chat, repeated System Save, Desktop/Terminal switching and Inbox handoff. Required source checks include whitespace, Home entropy, Rust formatting and narrow tests; installed checks bind source, built, installed and served artifacts. + +Browser qualification is a substantial remaining time commitment. Notion retains **100 cold launches, 100 warm launches, 100 lifecycle cycles, 30 minutes of continuous audio/video and eight hours of mixed use**, with matching manual UX and the objective audit, for the two claimed roles. This requirement needs planned test ownership and capacity before a release date can be credible. + +## Integration and team coordination + +Sash's shelf, Agent Space and composer were authored before this reporting period. This week's work preserved their history and integrated them into one Assistant. Irzhy's reconstruction also contributed Archive Manager, typed Assistant workspace and the Home menu protocol. Credit belongs to both the original UI work and the later adaptation. + +Sash's URUX adaptation `8b547590` entered through Irzhy's `7dd1780b`/`985fdffc`, followed by newer Sash dock/lock changes. Review the remaining behavior differences against this current implementation. Merging the old URUX tip in full would repeat already-adapted work. + +Irzhy's remaining sequence is [PR59](https://github.com/Elacity/elastos-runtime/pull/59) authority cutover, then ready work from [PR62](https://github.com/Elacity/elastos-runtime/pull/62), after the locally merged [PR60](https://github.com/Elacity/elastos-runtime/pull/60) foundation. PR60 is still open on GitHub. The published integration checkpoint includes its original history and leaves PR59/PR62 unchanged. Agree PR60's disposition with Irzhy before the next merge. PR62 is a draft: the adopted media work comes from its inherited foundation, while its four follow-up commits remain outside this candidate. The cutover must preserve conditional Documents saves, model execution contracts and Runtime lifecycle ownership. + +The [contributor branch and PR review](2026-09-11-contributor-review.md) records the checked heads, GitHub states, local/adapted status and remaining work, including relevant sources outside the 13-PR September inventory. It also lists unresolved dispositions instead of assuming every open branch belongs in this release. + +| Owner in the plan | Next coordination responsibility | +| --- | --- | +| Anders | Human acceptance, scope decisions, exact-candidate review and publication/release approvals. | +| Implementation owner | Complete the next required J3 behavior and its focused tests; preserve accepted J1 behavior. | +| Independent reviewer | Review the changed source and receipts, maintain one shared acceptance record, and coordinate target resources. | +| Sash | Review the integrated Home/Assistant experience and J1 human journey, including outside-tester coverage. | +| Browser owner | Finish the required reload slice and supply evidence for both placements and the updater slice. | +| Irzhy | Supply PR59/PR62 readiness and protected-video proof against the final callers. | +| Carrier and Content owner | Prove ordinary cold model delivery with bounded transfer, integrity, cancellation and cleanup. | + +These responsibilities need confirmed owners before the next slice starts. Notion owns acceptance and decisions, `TASKS.md` owns execution, and `state.md` owns current source, installed versions and verification results. + +## Process lessons and next meeting decisions + +The development loop caught real defects, but repeated coordination, broad reviews and large task histories consumed substantial effort. Accepted user journeys advanced more slowly than implementation. The available account-wide allowance readings cannot isolate this project's exact cost. + +Version 0.4.1 of the development-loop skill now requires one useful installed result, an early evidence checkpoint, and a route change when effort outpaces acceptance. The owner and reviewer check assumptions and dependencies before execution. After a result, they revise only the affected next goals and reuse unchanged evidence. The improvement still to demonstrate is lower effort per accepted user behavior. + +For the team sync, agree the next installed acceptance slice and its owner/reviewer, then resolve these items: + +1. Review the published integration [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) with this report and its contributor review. Website branch retirement is complete. Agree PR60's disposition with Irzhy. +2. Assign the next J3 slice, exact Marketplace-to-Assistant handoff and safe removal, with one owner and reviewer. Assign J1/J3 human checks, resolve Stop wording and reconcile Notion after the results. +3. Choose how to provide Linux model-test capacity and a reviewed Linux startup profile. The recorded gate requires about 13.4 GB additional free space before build margin. +4. Schedule the two-role Browser campaign and agree when its owned resources are available. +5. Confirm PR59/PR62 intake readiness and the external cryptographic review owner, scope and date. + +Use one build/test owner for the next installed result, review it independently, then update the dependent goals from that evidence. + +## Evidence used + +This report combines the live Notion plan, direct GitHub/Git ref checks, commit history and recorded audits. It is a document review, with no new product qualification run. Published evidence is linked above. The following audit files are included in the published reviewed checkpoint `06bf4e0f`: + +- `docs/audits/2026-09-10-execution-handover.md`: delivery history and current handover. +- `docs/audits/2026-09-11-first-checkpoint.md`: combined Mac/Linux recovery and lifecycle proof. +- `docs/audits/2026-09-11-j3-bootstrap.md`: bootstrap scope and focused source tests. +- `docs/audits/2026-09-11-integration-preservation.md`: donor history, all 13 contributor PR dispositions and remaining behavior differences. +- `docs/audits/2026-09-11-assistant-convergence.md`: one-Assistant source, rendered, installed and human-proof boundaries. +- `TASKS.md` and `state.md`: current execution and evidence records. + +The 5 September product/identity assessments, earlier enrollment/model source records, Browser analysis, QA triage and development-loop cost review also informed this report. The contributor companion links the reviewed PRs and identifies the limits of the comparison. diff --git a/state.md b/state.md index b21b723fe..17d06cad8 100644 --- a/state.md +++ b/state.md @@ -1,5 +1,31 @@ # State +## Current publication checkpoint — 2026-09-11 + +`feat/0.7.1-integration` is the active working/review branch. Reviewed source +`06bf4e0f11e602733e8eef861b835184ae683504`, tree +`4483f403a85d1ac27b4568f7b8cb4cf1a8a4e546`, is published in +[draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) against +`upstream/0.7.1-dev` at `6c61c990`. This single documentation closeout adds the +[team report](docs/audits/2026-09-11-team-sync.md) and +[contributor review](docs/audits/2026-09-11-contributor-review.md). +The PR records the final documentation commit; product code remains the reviewed +checkpoint. Remote ancestry protects both website heads (`00003b6f` and +`06bf4e0f`). The website refs and redundant checkout are removed; 69 local branches +and 11 worktrees remain with their donor/history preservation gates. Main, dev, +contributor PRs and installed services retain their prior identities. + +The Mac Home retains installed code `3c2f9a80` and Runtime SHA-256 +`1a55c86d95f5ca286bb63ee65242f3c23e19d0d1acd29888dc976b0033639666`. +This publication reused prior product evidence and ran the four basic source +gates; it performed no build, installation, restart or deployment. +Full J1 to J5 and D1 to D6 acceptance remains open under the approved plan. +The bounded publication/cleanup run stops here; broader execution and monitoring +stay paused. Next implementation: exact Marketplace-to-Assistant handoff and +safe removal, then cold Content/Carrier delivery. + +## Installed and source evidence + The September 11 integration now uses the full donor history. The combined source preserves current recovery, cookie and shutdown code while adding model admission, provider lifecycle, storage and window behavior. Focused source and rendered @@ -41,14 +67,11 @@ failures, repairs, independent review and the remaining full J3 gates. Last updated: 2026-09-11 UTC -- `feat/0.7.1-website-execution` is published for review at `00003b6f`, tree - `7d816617`, 55 ahead and zero behind development `6c61c990`. Anders chose - continued development on this branch. Notion now records the publication, - installed checkpoint and active C3/J3 intake; recurring monitoring stays paused. - The current Mac Home shows zero model offers; All models opens empty Settings, - and the app launcher lists both Assistant and Home Agent. This is an earlier - setup gap than the donor's historical preparation failure, whose cause remains - unknown. The integration donor retains its source and preservation role. +- Historical publication `00003b6f`, tree `7d816617`, was the website/recovery + review checkpoint. The later local integration donor and Assistant/model + repairs are now included in published integration. The original zero-offer + and duplicate-Assistant observations are superseded by the bounded installed + Mac results above; their original diagnostics and receipts remain historical. - The first J3 bootstrap slice integrates donor `c95cf4c9` with the current media setup call sites. Small artifact fixtures and all 56 Runtime setup From 259666222f12b21283131cc7926ba7d0c1a52e99 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 12:27:36 -0500 Subject: [PATCH 301/535] fix(build): lock the integrated provider dependency graph --- capsules/object-provider/Cargo.lock | 267 ++++++++++++++++++++++++---- elastos/Cargo.lock | 2 +- scripts/setup-source-home.sh | 14 +- 3 files changed, 243 insertions(+), 40 deletions(-) diff --git a/capsules/object-provider/Cargo.lock b/capsules/object-provider/Cargo.lock index 59687fecb..63e6c7a2e 100644 --- a/capsules/object-provider/Cargo.lock +++ b/capsules/object-provider/Cargo.lock @@ -52,7 +52,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" dependencies = [ "crypto-common 0.1.7", - "generic-array", + "generic-array 0.14.7", ] [[package]] @@ -64,6 +64,7 @@ dependencies = [ "cfg-if", "cipher", "cpufeatures 0.2.17", + "zeroize", ] [[package]] @@ -78,6 +79,7 @@ dependencies = [ "ctr", "ghash", "subtle", + "zeroize", ] [[package]] @@ -140,7 +142,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -151,7 +153,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -494,7 +496,7 @@ version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" dependencies = [ - "generic-array", + "generic-array 0.14.7", ] [[package]] @@ -832,6 +834,12 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "cpubits" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15b85f9c39137c3a891689859392b1bd49812121d0d61c9caf00d46ed5ce06ae" + [[package]] name = "cpufeatures" version = "0.2.17" @@ -1064,19 +1072,35 @@ version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" dependencies = [ - "generic-array", + "generic-array 0.14.7", "rand_core 0.6.4", "subtle", "zeroize", ] +[[package]] +name = "crypto-bigint" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a52aa3fcda4e6302a9f48734f234d35d4721b96f8fe07d073f07ce9df4f0271" +dependencies = [ + "cpubits", + "ctutils", + "hybrid-array", + "num-traits", + "rand_core 0.10.1", + "serdect", + "subtle", + "zeroize", +] + [[package]] name = "crypto-common" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" dependencies = [ - "generic-array", + "generic-array 0.14.7", "rand_core 0.6.4", "typenum", ] @@ -1087,6 +1111,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" dependencies = [ + "getrandom 0.4.3", "hybrid-array", "rand_core 0.10.1", ] @@ -1107,6 +1132,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" dependencies = [ "cmov", + "subtle", ] [[package]] @@ -1154,6 +1180,23 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "custody-provider" +version = "0.1.0" +dependencies = [ + "ed25519-dalek 2.2.0", + "elastos-protected-content-contracts", + "elastos-protected-content-custody", + "elastos-protected-content-provider-contracts", + "elastos-protected-content-rights", + "getrandom 0.2.17", + "nix", + "serde", + "serde_json", + "sha2 0.10.9", + "zeroize", +] + [[package]] name = "data-encoding" version = "2.11.1" @@ -1220,7 +1263,7 @@ dependencies = [ "asn1-rs", "displaydoc", "nom", - "num-bigint", + "num-bigint 0.4.8", "num-traits", "rusticata-macros", ] @@ -1424,7 +1467,7 @@ checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" dependencies = [ "der 0.7.10", "digest 0.10.7", - "elliptic-curve", + "elliptic-curve 0.13.8", "rfc6979", "signature 2.2.0", "spki 0.7.3", @@ -1632,6 +1675,32 @@ dependencies = [ "x-wing", ] +[[package]] +name = "elastos-protected-content-custody" +version = "0.7.0" +dependencies = [ + "aes", + "aes-gcm", + "ctr", + "ed25519-dalek 2.2.0", + "elastos-protected-content-contracts", + "elastos-protected-content-provider-contracts", + "ghash", + "hkdf", + "hpke", + "nix", + "rand 0.10.2", + "rand 0.9.5", + "rand_chacha 0.3.1", + "rand_core 0.6.4", + "sha2 0.10.9", + "subtle", + "thiserror 1.0.69", + "vsss-rs", + "x-wing", + "zeroize", +] + [[package]] name = "elastos-protected-content-provider-contracts" version = "0.7.0" @@ -1713,6 +1782,7 @@ dependencies = [ "bs58", "cid", "clap", + "custody-provider", "data-encoding", "dirs", "distributed-topic-tracker", @@ -1761,8 +1831,8 @@ dependencies = [ "tracing-subscriber", "url", "wasmparser 0.236.1", - "zip", "zeroize", + "zip", ] [[package]] @@ -1815,20 +1885,51 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" dependencies = [ "base16ct 0.2.0", - "crypto-bigint", + "crypto-bigint 0.5.5", "digest 0.10.7", - "ff", - "generic-array", - "group", + "ff 0.13.1", + "generic-array 0.14.7", + "group 0.13.0", "hkdf", "pem-rfc7468 0.7.0", "pkcs8 0.10.2", "rand_core 0.6.4", - "sec1", + "sec1 0.7.3", "subtle", "zeroize", ] +[[package]] +name = "elliptic-curve" +version = "0.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d65aa39b3a5c1c9c1b745c9a019234bb7a21b77abcb4f4d266d706e2d577d65" +dependencies = [ + "base16ct 1.0.0", + "crypto-bigint 0.7.5", + "crypto-common 0.2.2", + "ff 0.14.0", + "group 0.14.0", + "hybrid-array", + "pkcs8 0.11.0", + "rand_core 0.10.1", + "sec1 0.8.1", + "subtle", + "zeroize", +] + +[[package]] +name = "elliptic-curve-tools" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a0d5e534f103b079a71ef1d66c6e62c89413f1b62709ca11f744429b4afe5b4" +dependencies = [ + "elliptic-curve 0.14.1", + "heapless 0.9.3", + "serde", + "serdect", +] + [[package]] name = "embedded-io" version = "0.4.0" @@ -1874,7 +1975,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1899,6 +2000,16 @@ dependencies = [ "subtle", ] +[[package]] +name = "ff" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1f686ab92a9fb0eaf188f6c6c87b89490baa6fdb0db4544ba4dc47f7942489f" +dependencies = [ + "rand_core 0.10.1", + "subtle", +] + [[package]] name = "fiat-crypto" version = "0.2.9" @@ -2172,6 +2283,16 @@ dependencies = [ "zeroize", ] +[[package]] +name = "generic-array" +version = "1.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "337d46834ee672ab3e48caca2cb0c78cc174fb12b3a68d0d88f99a0519a5e36e" +dependencies = [ + "rustversion", + "typenum", +] + [[package]] name = "getrandom" version = "0.2.17" @@ -2221,6 +2342,7 @@ checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" dependencies = [ "opaque-debug", "polyval", + "zeroize", ] [[package]] @@ -2252,11 +2374,22 @@ version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" dependencies = [ - "ff", + "ff 0.13.1", "rand_core 0.6.4", "subtle", ] +[[package]] +name = "group" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fd1a1c7a5206c5b7a3f5a0d7ccd3ff85d0c8f5133d62a02680255b0004af5f4" +dependencies = [ + "ff 0.14.0", + "rand_core 0.10.1", + "subtle", +] + [[package]] name = "h2" version = "0.4.19" @@ -2296,6 +2429,15 @@ dependencies = [ "byteorder", ] +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + [[package]] name = "hashbrown" version = "0.15.5" @@ -2324,13 +2466,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cdc6457c0eb62c71aac4bc17216026d8410337c4126773b9c5daba343f17964f" dependencies = [ "atomic-polyfill", - "hash32", + "hash32 0.2.1", "rustc_version", "serde", "spin 0.9.9", "stable_deref_trait", ] +[[package]] +name = "heapless" +version = "0.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25ba4bd83f9415b58b4ed8dc5714c76e626a105be4646c02630ad730ad3b5aa4" +dependencies = [ + "hash32 0.3.1", + "stable_deref_trait", +] + [[package]] name = "heck" version = "0.5.0" @@ -2448,7 +2600,7 @@ dependencies = [ "aes-gcm", "chacha20poly1305", "digest 0.10.7", - "generic-array", + "generic-array 0.14.7", "hkdf", "hmac", "p256", @@ -2517,7 +2669,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" dependencies = [ "ctutils", + "subtle", "typenum", + "zeroize", ] [[package]] @@ -2759,7 +2913,7 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" dependencies = [ - "generic-array", + "generic-array 0.14.7", ] [[package]] @@ -3168,7 +3322,7 @@ checksum = "f6e3919bbaa2945715f0bb6d3934a173d1e9a59ac23767fbaaef277265a7411b" dependencies = [ "cfg-if", "ecdsa", - "elliptic-curve", + "elliptic-curve 0.13.8", "once_cell", "sha2 0.10.9", "signature 2.2.0", @@ -3435,6 +3589,7 @@ dependencies = [ "module-lattice", "rand_core 0.10.1", "sha3 0.11.0", + "zeroize", ] [[package]] @@ -3446,6 +3601,7 @@ dependencies = [ "ctutils", "hybrid-array", "num-traits", + "zeroize", ] [[package]] @@ -3775,7 +3931,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -3788,6 +3944,16 @@ dependencies = [ "num-traits", ] +[[package]] +name = "num-bigint" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93e7820bc0a80a0238e650327316f929ba18d5be054b647490a3a6a339f3e7c0" +dependencies = [ + "num-integer", + "num-traits", +] + [[package]] name = "num-conv" version = "0.2.2" @@ -3995,7 +4161,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" dependencies = [ "ecdsa", - "elliptic-curve", + "elliptic-curve 0.13.8", "primeorder", "sha2 0.10.9", ] @@ -4235,7 +4401,7 @@ dependencies = [ "cobs", "embedded-io 0.4.0", "embedded-io 0.6.1", - "heapless", + "heapless 0.7.17", "postcard-derive", "serde", ] @@ -4292,7 +4458,7 @@ version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" dependencies = [ - "elliptic-curve", + "elliptic-curve 0.13.8", ] [[package]] @@ -4403,7 +4569,7 @@ dependencies = [ "once_cell", "socket2", "tracing", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4747,7 +4913,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4806,7 +4972,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4877,12 +5043,26 @@ checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" dependencies = [ "base16ct 0.2.0", "der 0.7.10", - "generic-array", + "generic-array 0.14.7", "pkcs8 0.10.2", "subtle", "zeroize", ] +[[package]] +name = "sec1" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56d437c2f19203ce5f7122e507831de96f3d2d4d3be5af44a0b0a09d8a80e4d" +dependencies = [ + "base16ct 1.0.0", + "ctutils", + "der 0.8.1", + "hybrid-array", + "subtle", + "zeroize", +] + [[package]] name = "security-framework" version = "3.7.0" @@ -5214,7 +5394,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -5422,10 +5602,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.3.4", + "getrandom 0.4.3", "once_cell", "rustix", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -5988,6 +6168,27 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" +[[package]] +name = "vsss-rs" +version = "6.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bfc736cfd88115aedb95ba84bc2d428fe351e92a56f69fce090af301402d91" +dependencies = [ + "crypto-bigint 0.7.5", + "elliptic-curve 0.14.1", + "elliptic-curve-tools", + "ff 0.14.0", + "generic-array 1.4.5", + "hex", + "hybrid-array", + "num-bigint 0.5.1", + "num-traits", + "rand_core 0.10.1", + "serde", + "subtle", + "zeroize", +] + [[package]] name = "walkdir" version = "2.5.0" @@ -6514,7 +6715,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -7045,6 +7246,7 @@ dependencies = [ "sha3 0.12.0", "shake", "x25519-dalek 3.0.0", + "zeroize", ] [[package]] @@ -7067,6 +7269,7 @@ checksum = "e7e8131a03190127fb2263afc72b322ecadae46b6ff8c6f399ff5d02f5559af6" dependencies = [ "curve25519-dalek 5.0.0", "rand_core 0.10.1", + "zeroize", ] [[package]] diff --git a/elastos/Cargo.lock b/elastos/Cargo.lock index de29af314..94d344740 100644 --- a/elastos/Cargo.lock +++ b/elastos/Cargo.lock @@ -1871,8 +1871,8 @@ dependencies = [ "wasm-encoder 0.236.1", "wasmparser 0.236.1", "x-wing", - "zip", "zeroize", + "zip", ] [[package]] diff --git a/scripts/setup-source-home.sh b/scripts/setup-source-home.sh index 7a0fcce8c..65e521426 100755 --- a/scripts/setup-source-home.sh +++ b/scripts/setup-source-home.sh @@ -329,14 +329,14 @@ build_browser_vm_guest_helper() { echo "rust-lld not found in $rust_sysroot; Browser VM guest helper cross-build is unavailable" >&2 exit 1 fi - env "$linker_env=$linker" "$CARGO_BIN" build --quiet \ + env "$linker_env=$linker" "$CARGO_BIN" build --locked --quiet \ --manifest-path "$manifest" \ --target "$rust_target" \ --release return fi - "$CARGO_BIN" build --quiet \ + "$CARGO_BIN" build --locked --quiet \ --manifest-path "$manifest" \ --target "$rust_target" \ --release @@ -1805,11 +1805,11 @@ if ! grep -Eq '^[[:space:]]*trusted_keys[[:space:]]*=' "${CONFIG_TOML}"; then fi echo "[setup-source-home] build runtime server" -"$CARGO_BIN" build --manifest-path "${ROOT}/elastos/Cargo.toml" --release -p elastos-server +"$CARGO_BIN" build --locked --manifest-path "${ROOT}/elastos/Cargo.toml" --release -p elastos-server verify_collaboration_startup_config_input if [[ "$PLATFORM" == "darwin-arm64" ]]; then echo "[setup-source-home] build Browser VZ engine supervisor" - "$CARGO_BIN" build --manifest-path "${ROOT}/elastos/Cargo.toml" --release -p elastos-vz --bin browser-vz-engine-supervisor + "$CARGO_BIN" build --locked --manifest-path "${ROOT}/elastos/Cargo.toml" --release -p elastos-vz --bin browser-vz-engine-supervisor fi build_browser_vm_guest_helpers @@ -1831,13 +1831,13 @@ source_home_binary_manifest_path() { } echo "[setup-source-home] build native provider binaries" -"$CARGO_BIN" build --manifest-path "${ROOT}/elastos/capsules/shell/Cargo.toml" --release +"$CARGO_BIN" build --locked --manifest-path "${ROOT}/elastos/capsules/shell/Cargo.toml" --release source_home_binary_names | while IFS= read -r provider; do - "$CARGO_BIN" build --manifest-path "$(source_home_binary_manifest_path "${provider}")" --release + "$CARGO_BIN" build --locked --manifest-path "$(source_home_binary_manifest_path "${provider}")" --release done echo "[setup-source-home] build Home CLI native renderer" -"$CARGO_BIN" build --manifest-path "${ROOT}/capsules/home-cli/Cargo.toml" --release --bin home-cli +"$CARGO_BIN" build --locked --manifest-path "${ROOT}/capsules/home-cli/Cargo.toml" --release --bin home-cli echo "[setup-source-home] build app WASM capsules" for capsule in "${APP_CAPSULES[@]}"; do From 093b51ca3e12bf2e10abf3f35bb166bfe617a9e7 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 12:51:44 -0500 Subject: [PATCH 302/535] docs: record September work and accepted public preview --- TASKS.md | 15 +-- docs/audits/2026-09-10-execution-handover.md | 26 ++-- docs/audits/2026-09-11-contributor-review.md | 30 ++--- docs/audits/2026-09-11-public-preview.md | 79 ++++++++++++ docs/audits/2026-09-11-team-sync.md | 119 +++++++++---------- state.md | 55 +++++---- 6 files changed, 208 insertions(+), 116 deletions(-) create mode 100644 docs/audits/2026-09-11-public-preview.md diff --git a/TASKS.md b/TASKS.md index 1d83ea142..34699a38c 100644 --- a/TASKS.md +++ b/TASKS.md @@ -25,8 +25,9 @@ human verdict. The reviewed combined source `06bf4e0f` is published through `feat/0.7.1-integration` in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). The website checkpoint and later donor history are protected there; the redundant website refs and checkout are retired. The active integration checkout is named -in the private checkpoint. This documentation-only closeout preserves product -code, installed Homes and acceptance criteria. The [team report](docs/audits/2026-09-11-team-sync.md) +in the private checkpoint. Public preview `25966622` is now deployed with +verified artifacts and preserved account/user data; Anders confirmed existing-account +sign-in and saved work. Mac Homes retain their tested candidate. The [team report](docs/audits/2026-09-11-team-sync.md) and [contributor review](docs/audits/2026-09-11-contributor-review.md) describe included work and remaining gates. C3/J3 pauses after the bounded Mac checkpoint and publication. Next: exact Marketplace-to-Assistant handoff and safe removal, @@ -40,10 +41,10 @@ The C1-C7 sequence in the approved plan controls integration and publication. | ID | Outcome | Status / owner | Required input and next proof | | --- | --- | --- | --- | -| C1-site | A useful, truthful public storefront | Revised design in user review / coordinator | Centered introduction, platform tabs, first-use cards, product explanation and feature status follow the design reference. Inside Home explains local AI, Browser and protected publishing. A single 0.7.1 availability label replaces old release and Mac preparation copy. Copy feedback and permission fallback pass isolated browser checks; the control remains disabled until the 0.7.1 installer is served and verified. Desktop/mobile checks pass; user review and public approval remain open. | -| C1-home | Human Home entry at `/home/` | Source accepted / coordinator | Route, assets, auth and manifest checks pass. W2's empty-migration receipt and private-directory repairs pass Mac/Linux source smokes. Installed sign-out and shortcut migration remain C1-seed. | +| C1-site | A useful, truthful public storefront | Public preview deployed / coordinator | New storefront and canonical Home link pass public HTTPS hash and browser checks at `25966622`. Confirm team review. Installer control stays disabled until signed 0.7.1 delivery is served and verified. | +| C1-home | Human Home entry at `/home/` | Public route verified / coordinator | Public `/home/` and legacy redirect pass; the new sign-in screen renders without console errors. Anders confirmed existing-account sign-in and saved work. Full J1 sign-out/shortcut acceptance remains open. | | C1-proof | The execution loop handles a real reviewed slice | Accepted for source review and monitoring / coordinator | Independent source/rendered review, nine process cases and actual recurring monitor delivery pass. Continue through installed and public proof under C1-seed. | -| C1-seed | Seed serves the reviewed local candidate and storefront | Private working build verified / coordinator | Runtime d790a48e and storefront pass artifact parity, account/sign-in/System and actual Profile-inclusive Recovery Kit export. Clean private preview is available. Public stage is prepared; explicit approval, existing-account proof and full journey acceptance remain open. | +| C1-seed | Seed serves the reviewed local candidate and storefront | Bounded deployment accepted / coordinator | Authorized `25966622` deployment passes integrity, 573 artifact comparisons, running/served hashes and zero-migration guards. Account/user-file hashes are preserved. Anders confirmed existing-account sign-in and saved work; the temporary stage and deployment rollback are removed. Full journeys, Browser/protected-video target configuration and signed delivery retain their gates. | | C1-delivery | Signed three-platform candidate installation | Functional preparation complete; final assembly after C5 freeze / delivery owner | Native inputs, bootstrap and publisher import have bounded source/artifact proof. Keep working builds for functional tests and rebuild only affected components. Complete atomic promotion, final installer stamping, platform regeneration and signed fresh installs at the final release boundary. | | J1 / C2 | Install, passkey, Recovery Kit, Profile and usable Home | Human passkey proof pending / Home and installer owners | Registration binding and atomic/conflict-safe identity persistence are integrated with 66 focused tests passing. Durable owner intent and Profile-inclusive recovery now pass source and UI checks. Real signed Mac installation and seed export pass. Source repairs for kit-first recovery, gateway shutdown and media-tool reuse are accepted. Both targets pass installed browser recovery/reload/retry, complete owned shutdown/restart, full Home setup/media reuse and Carrier cleanup/reuse. User-review account label, Recovery/Advanced spacing and cross-port session isolation are repaired on `1e320578`; shared-browser Mac/Linux proof passes. Next: AUTH-01 final user acceptance on the updated preserved Homes; concurrent-save and window/selection UI; W2-W5; rerun the smoke with System entry through the ElastOS menu, then complete the app matrix; full Linux/Mac staging. Publish J1 for review first under D1. | | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | @@ -63,8 +64,8 @@ Immediate preview delivery: the fixed d790 candidate passed Mac installation and seed export; the user also installed it. The observed shutdown, duplicate-download and kit-first recovery repairs now pass source checks. The combined automated installed checks pass. AUTH-01 needs a user-operated -passkey on the prepared matching artifacts; preserve its pending verdict. Preserve all user test Homes. The public deployment -package is prepared and awaits approval plus target checks. Jetson is deferred +passkey on the prepared matching artifacts; preserve its pending verdict. Preserve all user test Homes. The public preview is deployed and Anders confirmed existing-account +sign-in and saved work. Jetson is deferred at the user's request. Wider J2–J5 work and final three-platform assembly remain required for release; unchanged artifacts retain their evidence. diff --git a/docs/audits/2026-09-10-execution-handover.md b/docs/audits/2026-09-10-execution-handover.md index bd3a6f88d..eebe698f6 100644 --- a/docs/audits/2026-09-10-execution-handover.md +++ b/docs/audits/2026-09-10-execution-handover.md @@ -27,14 +27,15 @@ local operator ledger and current checkpoint. Resolve their directory with Use the existing integration checkout on `feat/0.7.1-integration`, based on user-selected `origin/upstream/0.7.1-dev` at `6c61c990`. Resolve its exact local path from the private checkpoint. Reviewed source `06bf4e0f`, tree `4483f403`, -is published in draft PR64 with this documentation-only closeout. Both website +is published in draft PR64. Public deployment uses descendant `25966622`, tree +`419bdba6`, with reconciled dependency locks and locked setup builds. Both website heads remain ancestors of the verified remote integration ref. The default checkout contains unrelated dirty donor work. Preserve the remaining 69 branches and 11 worktrees. Before edits, recheck HEAD/tree, dirt, fetched divergence and process ownership. [The team report](2026-09-11-team-sync.md) and [contributor review](2026-09-11-contributor-review.md) are the shareable closeout. -The current installed candidate is `3c2f9a80012ff510449998f0b423a21c47ad2a98`, +The current Mac installed candidate is `3c2f9a80012ff510449998f0b423a21c47ad2a98`, tree `6be3ed9cde01c6dea8850ebd6ac381c5f03a9480`. Runtime built/installed SHA-256 is `1a55c86d95f5ca286bb63ee65242f3c23e19d0d1acd29888dc976b0033639666`. It preserves the accepted one-Assistant consolidation and adds pending Keep @@ -51,7 +52,7 @@ Anders confirms Assistant works. His enrolled account, Profile and conversation data are preserved. Normal browser controls show the new System and Marketplace Models views with Qwen available on the device. The earlier System/Marketplace metadata repair and installer guard `822c4e3d` remain in place. Original recovery -human Homes retain `1e320578`; existing public stage remains `d790a48e`. The private +human Homes retain `1e320578`; the older private stage retains `d790a48e`. The private checkpoint records current process/port ownership; the old Linux preview tunnel was absent on this resume and was left unchanged. Original recovery, retry, reload, media reuse and Carrier cleanup receipts retain their exact earlier @@ -73,9 +74,12 @@ dedicated provider-host/J5 proof and full J1–J5 retain their dependency order Required gates. Confirmed backend cancellation remains distinct from the accepted honest unknown result. Recurring monitoring remains paused; Jetson is deferred. -Further public deployment requires review and approval of the exact candidate. -The private publication record owns the verified route for a future authorized -push. Required installation manifests and receipts are current; final installer +The user-authorized public preview deployment is complete at `25966622`. +[Its proof](2026-09-11-public-preview.md) records preserved account/user data, +zero migrated roots/objects, installed/running parity and public HTTPS hashes. +Anders confirmed existing-account sign-in and saved work. The temporary stage +and bounded deployment rollback are removed. Browser Engine and custody configuration +remain separate J4/J5 work. Required installation manifests and receipts are current; final installer stamping, packaging and signing follow C5 source freeze and the release gate. ## Earlier preview evidence @@ -93,8 +97,8 @@ stamping, packaging and signing follow C5 source freeze and the release gate. - Recovery export smoke `49a62db1` uses the loaded foreground System window. Earlier failures were retained: static controls and restored covered windows made the test click the wrong surface. The final actual export passes. -- Public deployment has a concrete staged package and preservation plan. - Public live remains unchanged; deployment requires explicit approval. +- The earlier staged package supported the later authorized public deployment + recorded above. Its older source and test receipts remain historical evidence. ## Current repairs and acceptance limits @@ -120,9 +124,9 @@ obligations retain their release gates while that source intake proceeds. 1. **Finish J1:** finish the checkpoint’s user-operated passkey proof; prove restored Profile/name, returning sign-in, Desktop/Terminal and shared app behavior. Complete remaining setup, save-conflict and window work. -2. **Deliver the public preview:** approve and deploy the reviewed storefront - and seed Runtime, preserve existing accounts, then verify public artifacts. - Public signed installer delivery remains a separate gate. +2. **Complete public delivery:** preview deployment and its bounded human + sign-in/saved-work check are complete. Public signed installer delivery + follows the frozen-candidate release gate. 3. **Complete the local AI journey (J3):** one Assistant and real Qwen replies now pass the bounded Mac checkpoint. Next prove exact Marketplace-to-Assistant handoff and safe removal, then cold Content/Carrier delivery. Preserve drafts, diff --git a/docs/audits/2026-09-11-contributor-review.md b/docs/audits/2026-09-11-contributor-review.md index ef60633fa..f11cf470a 100644 --- a/docs/audits/2026-09-11-contributor-review.md +++ b/docs/audits/2026-09-11-contributor-review.md @@ -1,21 +1,21 @@ # ElastOS 0.7.1: contributor branches and PRs -**Checked 11 September 2026.** Companion to the [5 to 11 September team report](2026-09-11-team-sync.md). GitHub API metadata, local ancestry and selected patch/tree comparisons were checked against delivery candidate `06bf4e0f`, tree `4483f403a85d1ac27b4568f7b8cb4cf1a8a4e546`. +**Checked 11 September 2026.** Companion to the [5 to 11 September team report](2026-09-11-team-sync.md). Reviewed source: `06bf4e0f`, tree `4483f403a85d1ac27b4568f7b8cb4cf1a8a4e546`, published in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) into `upstream/0.7.1-dev`. -At publication, the reviewed candidate was 79 commits ahead and 0 behind the earlier public website checkpoint at `00003b6f`; both histories are now protected by integration. Local adoption and GitHub PR state are shown separately. “Ancestor” means the original commit is in candidate history. “Adapted” means work was transferred or revised through a different commit. Neither term certifies complete installed behavior. +PR58 remains in history; PR60 is included; PR59 then ready PR62 work remain pending. Contributor PR states remain separate from adoption. "Ancestor" means original history is included; "adapted" means transferred or revised work. Both require separate installed acceptance. -The existing `feat/0.7.1-integration` branch was fast-forwarded to this candidate and published in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) into `upstream/0.7.1-dev`. Remote verification protected both website heads before the redundant local/remote website refs and checkout were removed. This companion accompanies the single documentation-only closeout; PR64 records its final commit and immutable report links. PR58's merge remains in history; PR60's original head is included in the published candidate; PR59 and PR62 remain pending. All contributor PR heads, bases and states were preserved. +The public preview runs `25966622`, which adds a build-lock repair to this checkpoint; contributor ancestry is unchanged. Review the remaining UI differences and PR dispositions below against this candidate. The team report records branch cleanup, source divergence and the next acceptance steps; PR64 records the current review head. ## September inventory -The existing 13-PR inventory covers PRs created or updated since **1 September**, so it includes work older than this report's 5 to 11 September delivery window. Sash is `SashaMIT`; Irzhy is `irzhywau`. PR authorship and individual code authorship can differ, particularly in integration PRs. +These 13 PRs were created or updated since **1 September**, wider than the delivery window. Sash is `SashaMIT`; Irzhy is `irzhywau`. PR authorship can differ from individual code authorship. | Contributor / PR | Head branch and commit | GitHub state | Interpretation at `06bf4e0f` | | --- | --- | --- | --- | | Sash [19](https://github.com/Elacity/elastos-runtime/pull/19) | `chore/ci-source-home-matrix` · `1d6b49e3` | Closed, unmerged | Separate original ancestry. Current CI retains three-platform source installation, manual ref selection and installed-provider verification. macOS Clippy/tests moved to Linux jobs, and named feature push triggers became PR/manual/main/tag triggers. These differences need CI policy review. | -| Sash [23](https://github.com/Elacity/elastos-runtime/pull/23) | `feat/home-urux-on-freeze` · `5e546ef4` | Closed, unmerged | Adapted UI entered through `8b547590`, Irzhy's `7dd1780b` and merge `985fdffc`. Later Sash dock/lock changes are also ancestors. Compare remaining behaviors; the old tip is not the next merge target. | +| Sash [23](https://github.com/Elacity/elastos-runtime/pull/23) | `feat/home-urux-on-freeze` · `5e546ef4` | Closed, unmerged | Adapted UI entered through `8b547590`, Irzhy's `7dd1780b` and merge `985fdffc`. Later Sash dock/lock changes are also ancestors. Select remaining behaviors against the adapted implementation. | | Sash [26](https://github.com/Elacity/elastos-runtime/pull/26) | `feat/gba-nonogram-advance` · `7c622705` | Open | Adopted through `1fd30b38`, with separate original ancestry. ROM, MIT licence and SVG blobs match exactly. Current icon metadata/assets and attribution reflect later changes. | -| Irzhy [38](https://github.com/Elacity/elastos-runtime/pull/38) | `upstream/0.7-dev` · `e481b153` | Merged | Original release-source head is an ancestor. Historical foundation with September activity, rather than new implementation this week. | +| Irzhy [38](https://github.com/Elacity/elastos-runtime/pull/38) | `upstream/0.7-dev` · `e481b153` | Merged | Original release-source head is an ancestor. Earlier release foundation with September activity. | | Irzhy [51](https://github.com/Elacity/elastos-runtime/pull/51) | `upstream/0.7.1-dev` · `6c61c990` | Open | The head is the public dev baseline and an ancestor. This PR releases dev to main after candidate acceptance. | | Irzhy [52](https://github.com/Elacity/elastos-runtime/pull/52) | `feat/protected-content-installed-provisioning` · `4d688cc5` | Merged | Original head is an ancestor. Custody/chain provisioning and startup reconciliation are included. | | Sash [54](https://github.com/Elacity/elastos-runtime/pull/54) | `feat/home-first-run-seed-0.7.1` · `2a49ea57` | Merged | Original head is an ancestor. Includes first-run seed, lock face, Documents/Library/chrome fixes, dock behavior and capsule-owned service icons. | @@ -23,7 +23,7 @@ The existing 13-PR inventory covers PRs created or updated since **1 September** | Irzhy [57](https://github.com/Elacity/elastos-runtime/pull/57) | `feat/protected-content-installed-e2e-proof` · `617796a9` | Closed, unmerged | Replaced by PR60 after retargeting/rebase. Both now identify the same head. Count this implementation once. | | Irzhy [58](https://github.com/Elacity/elastos-runtime/pull/58) | `feat/0.7.1-integration` · `5ba1faa0` | Merged | Published head is an ancestor. Local merge `b318bfda` incorporates the later donor at `6972e165` too. | | Irzhy [59](https://github.com/Elacity/elastos-runtime/pull/59) | `feat/protected-content-atomic-cutover` · `25ab205e` | Open | Head is outside candidate ancestry. Follows PR60. Runtime authority cutover and API removal need review against the current callers before integration. | -| Irzhy [60](https://github.com/Elacity/elastos-runtime/pull/60) | `feat/protected-content-installed-e2e-proof` · `617796a9` | Open | Full original history is merged locally by `dd21d8bd`. Includes provider hosting, peer seeding, availability/media repairs and installed-proof tooling. Its GitHub PR remains open. | +| Irzhy [60](https://github.com/Elacity/elastos-runtime/pull/60) | `feat/protected-content-installed-e2e-proof` · `617796a9` | Open | Full original history is included through `dd21d8bd`. Includes provider hosting, peer seeding, availability/media repairs and installed-proof tooling. Its GitHub PR remains open. | | Irzhy [62](https://github.com/Elacity/elastos-runtime/pull/62) | `feat/protected-content-0.7.1-followup` · `06179578` | Open, draft | Head is outside candidate ancestry and follows PR59. Its four commits contain the plan, wiring/cleanup, Creator and non-media protection/read support. Reader capsule, audio and crypto-review completion remain pending. | ## Earlier sources needed to interpret 0.7.1 @@ -32,24 +32,24 @@ The existing 13-PR inventory covers PRs created or updated since **1 September** | --- | --- | --- | --- | | Irzhy [39](https://github.com/Elacity/elastos-runtime/pull/39) | `feat/protected-content-uiux-reconstruction` · `7dd1780b` | Merged | Ancestor through `985fdffc`. Carries adapted Sash UIUX plus Archive Manager reconstruction, typed Assistant workspace, Home menu protocol and CI work. | | Irzhy [43](https://github.com/Elacity/elastos-runtime/pull/43) | `fix/protected-content-mint-intent-adoption` · `58ebfb23` | Merged | Ancestor. Preserves completed mint work after a lost completion mark. | -| Irzhy [15](https://github.com/Elacity/elastos-runtime/pull/15) | `feat/dkms-esp-port` · `27d85c6f` | Open | Separate ancestry. The project retains an extraction ledger for protected-content behavior. Remaining parity and audit-migration decisions need explicit disposition; a wholesale merge is not implied. | +| Irzhy [15](https://github.com/Elacity/elastos-runtime/pull/15) | `feat/dkms-esp-port` · `27d85c6f` | Open | Separate ancestry. The project retains an extraction ledger for protected-content behavior. Decide remaining behavior parity and audit migration before selecting further changes. | | Sash [17](https://github.com/Elacity/elastos-runtime/pull/17) | `feat/model-provider-rebase` · `9f423796` | Open | Separate ancestry. Credit the earlier model-provider work. The candidate uses a later typed-provider implementation; complete feature equivalence requires a scoped comparison. | -| Irzhy [25](https://github.com/Elacity/elastos-runtime/pull/25) | `feat/elastos-logger` · `66c8bba4` | Open | Separate ancestry. The current candidate has no `elastos-logger` crate. This independent logging proposal needs an explicit scope/disposition decision. | +| Irzhy [25](https://github.com/Elacity/elastos-runtime/pull/25) | `feat/elastos-logger` · `66c8bba4` | Open | Separate ancestry. The proposed `elastos-logger` crate remains outside the candidate; decide its scope and disposition. | -Earlier protected-content extraction work is represented in the base source and PR15 ledger. This review checks the active intake interpretation and the relevant older open tracks; it does not claim feature-by-feature equivalence across every historical PR. +The base source and PR15 ledger record earlier protected-content extraction. This review covers intake and relevant older tracks; full historical feature equivalence needs a scoped comparison. ## Integration details and remaining decisions -**Sash UI preservation.** The original shelf/composer/Agent Space series was authored on 3 September. This week recorded its original ancestry and retained its UI in the unified Assistant. Patch comparisons and capsule-tree identity support that preservation: original `239c6cb7` and adopted `77498557` share tree `c03166c4489f6773792c38c337f7ccb5e714f6ce`. The URUX adaptation chain and newer dock/lock commits are also present. Review remaining Viewer rail, pager/switcher, Mission Control, launcher motion and workflow differences against the current Runtime contracts before selecting further UI work. +**Sash UI preservation.** Sash authored the shelf/composer/Agent Space series on 3 September. This week retained its UI in one Assistant and recorded original ancestry. Original `239c6cb7` and adopted `77498557` share capsule tree `c03166c4489f6773792c38c337f7ccb5e714f6ce`; matching patches support preservation. The URUX adaptation and newer dock/lock commits are included. Review Viewer rail, pager/switcher, Mission Control, launcher motion and workflow differences against current Runtime contracts before selecting further work. **Irzhy installed evidence.** The 5 to 7 September protected-content test used an installed client, three custody containers and an Anvil/Base fork. Its final receipt records passing mint, availability, buy, 2-of-3 release, ordered reads, close, denial/tamper/replay, restart and cleanup phases. The proof used headless principals; final Brave interaction, independent operator/hardware domains and real Base remain separate. Its receipts bind the binaries actually tested, which predate later lint/test edits. Commit `617796a9` retains a 5 September author date, an 8 September committer date, and entered the delivery branch on 10 September. -**PR62 scope.** The diff from `25ab205e` to `06179578` contains no media-provider changes. Media code present at the PR62 tip comes from its inherited stack. The delivery branch's media repair therefore does not establish adoption of PR62 follow-up work. Creator source exists at its current head; a Reader capsule does not. Required video-journey repairs remain required under D2, while optional additions enter only with source and installed evidence before freeze. +**PR62 scope.** Media-provider code at `06179578` comes from the inherited stack. The four follow-up commits after `25ab205e` remain outside this candidate. Creator source exists; Reader capsule work remains pending. Under [D2](2026-09-11-team-sync.md#decisions-retained-from-notion), video-journey repairs remain required; optional additions need source and installed evidence before freeze. -**Models UI ownership.** Dedicated Models screens came from integration commit `b0c4c2f5`. The bounded source review found no separate Sash/Irzhy Models design to restore. The latest repair reuses Sash's general Marketplace category/detail layout and shared theme. This separates credit for the design foundation from responsibility for the later model controls. +**Models UI ownership.** Dedicated Models screens came from `b0c4c2f5`. The reviewed design foundation is Sash's general Marketplace category/detail layout and shared theme, reused by the later model controls. The bounded source comparison traced the reused contributor design to that Marketplace foundation. -Before closing contributor intake, the team needs dispositions for the older open tracks, the remaining UI differences and PR59/PR62 readiness. A clean merge or a closed PR cannot make those decisions on its own. +Before closing intake, Sash reviews remaining UI differences, Irzhy establishes PR59/PR62 readiness, and the team decides which older open tracks belong in this release. ## Check method and limits -GitHub API supplied PR states, heads, branches and descriptions. Local Git supplied ancestry, commit dates, stable patch IDs, selected blob/tree identities and the PR59-to-PR62 diff. The preservation and convergence audits supplied installed-evidence scope. This review ran no builds or installed tests and changed no contributor branches, PRs or acceptance checkboxes. +GitHub API supplied PR states, heads, branches and descriptions. Git supplied ancestry, dates, patch IDs, selected blob/tree identities and the PR59-to-PR62 diff. The [preservation](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-11-integration-preservation.md) and [convergence](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-11-assistant-convergence.md) audits bound installed evidence. This document review reused those results and preserved contributor branches, PRs and acceptance checkboxes. diff --git a/docs/audits/2026-09-11-public-preview.md b/docs/audits/2026-09-11-public-preview.md new file mode 100644 index 000000000..8004f2203 --- /dev/null +++ b/docs/audits/2026-09-11-public-preview.md @@ -0,0 +1,79 @@ +# Public preview deployment, 11 September 2026 + +The user-authorized preview is available at [ElastOS](https://elastos.elacitylabs.com/) +and [Home](https://elastos.elacitylabs.com/home/). The public Runtime restarted +at 17:42 UTC from `259666222f12b21283131cc7926ba7d0c1a52e99`, tree +`419bdba6a09ed37e8176ddfda9c22c1a270f6054`, published in +[draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). + +## Build and adoption + +The first Linux build found an outdated standalone Object provider lockfile. +Its required packages already existed, with matching versions and checksums, in +the reviewed workspace lockfile. Commit `25966622` reconciles those inputs and +makes setup use `--locked`. All 25 required Linux manifests passed locked offline +metadata checks. The resumed build reused the compiled Runtime/providers and +completed source-home setup. Application source remains the reviewed `06bf4e0f` behavior. + +The staged installation passed component integrity and Home/Services readiness. +Public adoption replaced the selected capsules, binaries, helpers, website and +manifests. Browser helper paths were bound to the stable public installation. +Verified media tools were reused through Runtime setup. Public Kubo, provider +settings, accounts, keys and user files stayed in place. + +The initial check rejected an older group-writable identity directory. Its mode +was changed from `0775` to `0700`; Runtime created an empty owner-only identity +lock. Device keys and credentials retained their bytes. Before and after capsule +adoption, the migration guard reported `already_ready`, zero roots, zero objects +and an empty roots list. The restart migration receipt repeated that result. +Account/authentication records, user files and preserved configuration matched +the pre-start checks after startup. + +## Verification + +| Surface | SHA-256 | +| --- | --- | +| Built, installed and running Runtime | `b6634092055116c85556426f200c177fc6bf0ad2159af4025d48bb7bf3e89cff` | +| Installed components manifest | `7252c7363a264e16753907bcee98d36b96256865d3bd30b6cbd87db176606c8e` | +| Public website index | `d096d41c2e32beca1a9f2bb89ca7b69db343bd49739dbdcdbf61fc1c5af6c987` | +| Public Home index | `d2f5742661700f67814e6945cf4e6f6db4bed547cd949230be316f26dea3e34e` | + +The installed integrity checker passed. All 573 selected artifact files matched +their staged inputs, including the intentional helper-path substitutions. +Five public HTTPS route checks passed: `/`, `/home/`, `/apps/services/` and +`/apps/assistant/` returned 200 with matching installed hashes; `/apps/home/` +redirected to `/home/` and returned the same bytes. A fresh +browser rendered the storefront and Home sign-in screen without console errors. +The target source was clean at verification. The private operator receipt holds +exact paths, commands, process identities and the full artifact inventory. + +Recheck the public hashes with: + +```bash +curl -fsSL https://elastos.elacitylabs.com/ | shasum -a 256 +curl -fsSL https://elastos.elacitylabs.com/home/ | shasum -a 256 +``` + +From the candidate checkout, the installed check uses `ELASTOS_DATA_DIR` set +to the verified installation directory: + +```bash +python3 scripts/components-release-integrity-check.py \ + --manifest "$ELASTOS_DATA_DIR/components.json" --platform linux-amd64 \ + --profile source-home --source-root . \ + --source-home-data-dir "$ELASTOS_DATA_DIR" +``` + +## Remaining checks + +Anders confirmed that existing-account sign-in and saved work look correct. +This closes the bounded deployment acceptance gate. The temporary stage and +146 MiB deployment rollback are removed. Earlier historical rollback sets retain +their separate reconciliation obligations. + +Startup diagnostics report an unavailable configured Browser Engine and an +unconfigured inactive custody provider. J4/J5 acceptance remains open on the +required test targets. Full J1-J5 acceptance, Linux model qualification and +the signed three-platform installer release remain open. This deployment makes +the reviewed preview available for team testing; the download control correctly +states that the 0.7.1 installer is coming soon. diff --git a/docs/audits/2026-09-11-team-sync.md b/docs/audits/2026-09-11-team-sync.md index d4926dd9b..6e58c2ddc 100644 --- a/docs/audits/2026-09-11-team-sync.md +++ b/docs/audits/2026-09-11-team-sync.md @@ -1,12 +1,10 @@ # ElastOS 0.7.1: weekly team report -**5 to 11 September 2026.** Source snapshot: `06bf4e0f`. Publication status: the reviewed checkpoint is published on `feat/0.7.1-integration` in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). This report accompanies the single documentation-only closeout; the PR records its final commit and immutable report links. Calendar dates use America/Panama unless a source explicitly gives UTC. Earlier contributor work is identified as work integrated or reviewed during this period. +**5 to 11 September 2026.** Reviewed source: `06bf4e0f`, published in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) into `upstream/0.7.1-dev`. PR64 records the current review head. Dates use America/Panama unless a source gives UTC; earlier contributions are identified as work integrated or reviewed this week. -The period produced first-owner and recovery work, a published website/setup review branch, passing automated recovery and shutdown checks on Mac and Linux, and one installed Mac Assistant with real model replies. Irzhy also recorded a full headless protected-content journey in the local custody/chain test environment. Model, Home and protected-content contributions were brought into the delivery branch while preserving their history. Full 0.7.1 release acceptance remains open. +The team delivered first-owner setup and recovery work, automated Mac/Linux recovery and shutdown checks, and one installed Mac Assistant with real model replies. Irzhy recorded a full headless protected-content journey in the local custody/chain environment. Integration preserved the model, Home and protected-content histories, including merged PR58 and PR60's foundation. PR59 and PR62 remain pending. -The combined source is published through `feat/0.7.1-integration` for team review. It contains the website checkpoint, the complete later local integration history and the Assistant/model repairs. Remote verification protected both website heads before its local and remote branch refs and redundant checkout were removed. Irzhy's merged PR58 remains in history; PR59 and PR62 remain pending. The installed Mac Home retains its tested product code. The next implementation slice is exact Marketplace-to-Assistant handoff and safe removal, followed by cold Content/Carrier delivery. - -The remaining release work is to complete the five agreed user journeys on one frozen candidate. This report separates completed implementation, recorded tests and acceptance still required. +Sash and Irzhy can now review the combined checkpoint. The next implementation slice is exact Marketplace-to-Assistant handoff and safe removal, followed by cold Content/Carrier delivery. The team needs one owner and independent reviewer for that slice, plus J1/J3 human checks. All five journeys still need complete acceptance on one frozen candidate. The [public preview](https://elastos.elacitylabs.com/) now runs `25966622`; [open Home](https://elastos.elacitylabs.com/home/) to test it. Anders confirmed existing-account sign-in and saved work. Full journey acceptance remains open. ## Foundations completed on 5 and 6 September @@ -16,7 +14,7 @@ The remaining release work is to complete the five agreed user journeys on one f **Local AI and run ownership.** The 5 to 6 September series added verified engine/model inputs (`c95cf4c9`), managed local-engine startup/recovery (`7c4fc929`), hosted-backend evidence (`fcb8fc5e`) and a Responses adapter with explicit cancellation outcomes (`0d768415`). Home Agent preserved unresolved drafts and uncertain create outcomes instead of dispatching again. Anders observed a complete Qwen reply on the isolated Mac installation. One journal-bound request completed in 111.2 seconds. Stop on another run recorded one dispatch and one unknown terminal outcome. Hosted support still needs its optional release acceptance. -The 5 September product and identity assessments also proposed a useful document-and-AI journey and a clear separation between personal identity, device keys, permissions and capsule identity. These were design recommendations. They did not change the five-journey release scope or establish new identity/cryptographic support. +The 5 September assessments proposed a document-and-AI journey and separate personal identity, device keys, permissions and capsule identity. These design recommendations retained the five-journey scope; added identity or cryptographic support requires its own proof. ## Work delivered this week @@ -31,10 +29,10 @@ The 5 September product and identity assessments also proposed a useful document | Model foundation | Integrated the full model donor, including bootstrap integrity, signed catalog access, admission, preparation, startup, inventory and lifecycle work. Repaired the observed readiness and local-verifier boundaries. | Focused source tests and a real installed Mac Qwen reply/lifecycle/reuse checkpoint passed. The required delivery and management journey still has open acceptance steps, listed below. | | One Assistant | Consolidated the public Assistant identity while retaining Sash's Agent Space, shelf composer and conversation UI. Added protected workspace adoption, complete history/draft preservation, visible conflict copies and typed Chat/Build/Studio controls. | Local code `fa297cb5` passed source, rendered and limited installed Mac checks. A real Qwen response appeared, saved, and survived reload and full Runtime restart with the original draft and exact model selection. | | Model controls and activity | Follow-up `3c2f9a80` retains Keep intent during preparation, uses the existing Marketplace category/detail layout and gives System consistent styling. `d3d757ce` separates current activity from previous activity and visible reasoning. Source-install guard `822c4e3d` validates copied capsule metadata. | Both Mac checkpoint Homes have the follow-up installed. System Keep readback and real Qwen reply/save/reload pass; the new model views were observed in Anders's browser. Pending Keep has Runtime/rendered race tests; the installed checks reused an admitted package. | -| Contributor integration | Preserved the full model donor history, recorded Sash shelf ancestry and merged Irzhy's protected-content foundation locally. Rechecked the 13-PR September inventory and earlier sources relevant to 0.7.1. | The companion contributor review distinguishes GitHub state, local ancestry, adapted changes and outstanding behavior. The September inventory starts on 1 September; it is broader than this report's delivery window. | -| Development process | Revised the development-loop skill to version 0.4.1. Added installed-result checkpoints, effort review and stronger plan/dependency checks. | The written workflow is improved. Measured cost savings remain to be demonstrated. | +| Contributor integration | Preserved model donor and Sash shelf history and included Irzhy's protected-content foundation. Rechecked the 13-PR September inventory and relevant earlier sources. | The companion contributor review distinguishes GitHub state, local ancestry, adapted changes and outstanding behavior. The September inventory starts on 1 September; it is broader than this report's delivery window. | +| Development process | Revised the development-loop skill to version 0.4.1. Added installed-result checkpoints, effort review and stronger plan/dependency checks. | The written workflow is improved. Measured cost savings remain to be demonstrated. The Linux deployment also showed that locked dependency checks should precede compilation. | -Early-week Browser work also produced useful evidence. The 7 September investigation reproduced missing-artifact readiness errors, input problems and fragile recovery. Later records include bounded local browsing/media/input proof in run 92, approved inspection/fill/revoke actions in run 93, and Linux Home/Exit with a Mac Engine in run 96. That remote run confirmed ordinary actions and all 13 close effects, but failed its five-second viewer reload gate. These are historical results on their named artifacts; the combined release still needs Browser acceptance. +The 7 September Browser investigation reproduced missing-artifact readiness errors, input problems and fragile recovery. Run 92 later recorded bounded local browsing/media/input proof; run 93 recorded approved inspection/fill/revoke actions. Run 96 used Linux Home/Exit with a Mac Engine: ordinary actions and all 13 close effects passed, while the five-second viewer reload gate failed. These results apply to their named artifacts; the release candidate still needs Browser acceptance. ## What the Assistant checks establish @@ -44,34 +42,39 @@ Installed Mac adoption exercised existing Home Agent data. The consolidation che All-three-store migration has source and rendered proof; the installed migration used one legacy data set. Installed Studio covers mode controls and the honest absence of image/video offers. Generation and foreign-run protection have source/test evidence. Anders has since reported successful Assistant use, and the update preserved his enrolled account, Profile and chats. Full human J3 and Linux qualification remain required. -Keep protects Runtime's prepared cache from reclamation. It does not pin Content in IPFS or reserve model memory. Preparation progress counts verified local package bytes and can use content already on disk. These checks establish local preparation and inference; ordinary cold Carrier delivery remains a separate requirement. +Keep protects Runtime's prepared cache from reclamation; IPFS pinning and model-memory reservation are outside its scope. Preparation counts verified local package bytes, including content already on disk. Ordinary cold Carrier delivery still needs proof. ## Source, installed candidate and publication | Surface | Recorded version | Meaning for the team | | --- | --- | --- | -| Public `main` | `8ac18bec`, also the target of tag `v0.7.0` | Existing release source. | -| Public `upstream/0.7.1-dev` | `6c61c990` | Current public integration baseline. | -| Historical website checkpoint | `00003b6f`, tree `7d816617` | Published earlier; its full history remains reachable through integration after the website ref was retired. | -| Reviewed integration checkpoint | `06bf4e0f`, tree `4483f403` | Published in draft PR64: 134 commits ahead and 0 behind public dev at this source checkpoint. The documentation-only closeout follows without changing product code. | -| Earlier local integration donor | `6972e165` | Its complete history is included. The existing integration branch was fast-forwarded to the combined candidate and is now the continuing working/review line. | -| Installed Assistant/model candidate | Code `3c2f9a80`, tree `6be3ed9c` | Current recorded candidate on the two Mac checkpoint Homes. It includes the earlier consolidation and later metadata, model-control and activity repairs. | -| Earlier isolated recovery Homes | Code `1e320578` | Separate Mac/Linux recovery evidence and preserved user-review state. | -| Existing previews/public stage | Recorded as `d790a48e` in the handover | Earlier artifacts with their own receipts. The new Assistant checkpoint does not update their verdict. | +| Public `main` | `8ac18bec`, tag `v0.7.0` | Existing release source; [PR51](https://github.com/Elacity/elastos-runtime/pull/51) remains the release path. | +| Public `upstream/0.7.1-dev` | `6c61c990` | Public dev baseline. | +| Historical website checkpoint | `00003b6f`, tree `7d816617` | Its full history remains reachable through integration. | +| Reviewed integration checkpoint | `06bf4e0f`, tree `4483f403` | Published in PR64; 134 ahead/0 behind public dev and 79 ahead/0 behind the earlier public website checkpoint. Counts include merged history. | +| Publication closeout | `1bb081ff`, tree `810ff8bc` | The original Markdown-only closeout records PR64, branch cleanup and the team reports. | +| Linux build repair | `25966622`, tree `419bdba6` | Reconciles the Object provider lockfile with dependencies already in the reviewed workspace and makes all setup builds use `--locked`. Application source remains the reviewed checkpoint. | +| Earlier integration donor | `6972e165` | Its complete history is included in the continuing integration branch. | +| Installed Assistant/model candidate | `3c2f9a80`, tree `6be3ed9c` | Two Mac checkpoint Homes retain the tested consolidation, metadata, model-control and activity repairs. | +| Earlier isolated recovery Homes | `1e320578` | Separate Mac/Linux recovery evidence and preserved user-review state. | +| Earlier previews/stage record | `d790a48e` in the handover | Earlier artifacts retain their own receipts. | +| Public website and Home | `25966622`, tree `419bdba6` | Deployed on 11 September at 17:42 UTC. Runtime, 573 installed artifact files and public route hashes pass. Account and user-file checks pass; Anders confirmed sign-in and saved work. | + +Product checkpoint `06bf4e0f` differs from installed Mac code only in four Markdown files. Runtime hash checks and all 84 affected browser-file comparisons passed; the audits bind Runtime, components, capsules and served files to their tested versions. -The human-review Mac Home runs code `3c2f9a80`. The reviewed source head `06bf4e0f` differs only in four Markdown files. Runtime hash verification and the 84 affected browser-file comparisons passed. Publishing the source branch preserves this installed Home and its user data. Detailed audits bind the Runtime, components, capsules and served files to their tested versions. +`feat/0.7.1-integration` is the continuing local/public review branch. Verified integration history protects both website heads; the redundant website refs and checkout were removed after file, evidence and process checks. Contributor PR disposition remains separate. Release PR51, the 0.7.1 tag and stable artifact promotion follow candidate acceptance and approval. -The result is one active `feat/0.7.1-integration` working/review branch locally and on GitHub, with [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) into `upstream/0.7.1-dev`. The website branch was removed locally and remotely after exact ancestry and remote-head verification. Its checkout was removed after tracked/untracked/ignored-file, evidence, symlink and process checks. Only rebuildable caches were discarded; the historical receipts remain available. The inventory now contains 69 local branches and 11 worktrees. Remaining donor histories and dirty worktrees keep their existing owners and cleanup gates. This publication preserves `main`, public dev, PR58's merge and installed services. Contributor PR closure or retargeting requires separate coordination. +## Public preview verification -The website checkpoint and later combined work are now protected by the public integration history. Release PR51, the 0.7.1 tag, stable artifact promotion and public deployment follow acceptance and approval. Public refs were checked directly. The report review reused product evidence and did not qualify the public site's current bytes. +The public website, Home, Services and Assistant return HTTP 200 with matching installed bytes; `/apps/home/` redirects to `/home/`. A fresh browser shows the new storefront and Home sign-in screen without console errors. The cutover preserved account records, keys, user files and provider settings. Both migration guards and the restart gate reported zero roots and zero objects to migrate. One older identity-directory permission needed correction. -[Published website checkpoint](https://github.com/Elacity/elastos-runtime/tree/00003b6f0771138fbb1a5d834de7c027feaee508) · [Published checkpoint evidence](https://github.com/Elacity/elastos-runtime/blob/00003b6f/docs/audits/2026-09-11-first-checkpoint.md) · [Release PR51](https://github.com/Elacity/elastos-runtime/pull/51) +The server still reports an unavailable configured Browser Engine and an unconfigured inactive custody provider. Browser and protected-video acceptance remain with J4/J5. Public 0.7.1 installers still show “coming soon”; final signed delivery follows source freeze. Anders confirmed existing-account sign-in and saved work; the deployment rollback gate is closed. The [deployment record](2026-09-11-public-preview.md) contains the source, hashes and verification limits. ## Notion plan and current acceptance The governing plan is [ElastOS 0.7.1 release plan: five user journeys from start to stop](https://app.notion.com/p/wauio/ElastOS-0-7-1-release-plan-five-user-journeys-from-start-to-stop-3d6b682adcca81948f78d12abcd677b9). It was compiled on 9 September, reconciled on 10 September, and updated through the 11 September Assistant checkpoint. -The tables below preserve all five journeys, six decisions and seven execution checkpoints. The full criteria remain in Notion. Its C3 and J3 evidence rows reflect `3c2f9a80`; some J1/source-history wording needs reconciliation with the later integration. One acceptance decision needs clarification: the J3 overview asks for confirmed cancellation, while detailed step 5 permits confirmed settlement or an honest unknown. The team must resolve that wording before accepting Stop behavior. +Notion holds the full criteria. Its publication status links PR64, and C3/J3 evidence reflects `3c2f9a80`. Before accepting Stop, resolve the difference between the J3 overview's confirmed cancellation and detailed step 5's confirmed settlement or honest unknown. | Journey | Required user outcome | Position at this checkpoint | | --- | --- | --- | @@ -85,7 +88,7 @@ The tables below preserve all five journeys, six decisions and seven execution c | Decision | Agreed scope | | --- | --- | -| **D1** | Publish for review journey by journey, starting with J1. J2 shares delivery work; J3 depends on J1. Execution continued on the website branch. The authorized publication consolidates that accumulated work into the existing integration branch while preserving its history and the journey acceptance gates. | +| **D1** | Publish for review journey by journey, starting with J1. J2 shares delivery work; J3 depends on J1. The combined integration checkpoint retains those acceptance gates. | | **D2** | Take PR62 additions by readiness at source freeze. Repairs required for the video journey stay required. Optional additions need source and installed proof before freeze; unfinished additions move to 0.7.2. | | **D3** | A model is signed content delivered through Content and Carrier. Cold delivery is required. Hosted adapters and a Codex provider are optional after the local path. Remote inference, hosted sharing and Jetson are later work. | | **D4** | Browser uses one capsule and Runtime contract for Mac-local use and Linux Home/Exit with a Mac Engine. Both placements, including reload, require qualification. | @@ -96,67 +99,61 @@ The tables below preserve all five journeys, six decisions and seven execution c | Checkpoint | Deliverable and dependency | Current position | | --- | --- | --- | -| **C1** | Truthful storefront, canonical Home entry and isolated signed candidate delivery. Concrete seed deployment needs review and approval. | Source and private/staged delivery advanced; final delivery gates remain. | +| **C1** | Truthful storefront, canonical Home entry and isolated signed candidate delivery. Concrete seed deployment needs review and approval. | Public storefront and Home are deployed and artifact-verified. Anders confirmed existing-account use; full J1 and signed installer delivery remain. | | **C2** | Complete J1 setup work, preserve recovery/session/lifecycle behavior, carry J2 as coherent update commits and publish reviewed evidence. | Review prefix published; complete human/shared J1 acceptance remains open. | | **C3** | Deliver required J3 on the accepted J1 behavior, with independent review and human acceptance. | Mac Assistant plus model/activity follow-up recorded; broader J3 criteria remain. | | **C4** | Finish Browser source and installed proof for both required placements. It can run alongside C3 when target/build resources are available. Hand the tested updater slice to J2. | Browser slice and qualification remain pending. | -| **C5** | Assemble reviewed journeys into dev, take PR60 then PR59 and ready PR62 work, stamp version/changelog and freeze the exact source and optional scope. | Pending. The local PR60 foundation merge is partial intake, rather than completion of assembly. | +| **C5** | Assemble reviewed journeys into dev, take PR60 then PR59 and ready PR62 work, stamp version/changelog and freeze the exact source and optional scope. | Pending. PR60 is included in the review candidate; dev assembly and source freeze remain open. | | **C6** | Build/sign one compatible three-platform set and run all required journey, update/interruption and shared checks on that frozen candidate. | Pending. A repair records a new candidate and repeats affected acceptance. | | **C7** | After acceptance and approval, merge PR51, verify tree identity, tag and publish accepted artifacts, then verify canonical public delivery and close the hygiene record. | Pending. | -J4 and J5 can proceed beside J3 only when their owners have independent source boundaries and available build/test resources. Final packaging follows source freeze. Optional work cannot postpone the freeze. +J4 and J5 can proceed beside J3 with separate source ownership and available build/test resources. Final packaging follows source freeze; unfinished optional work moves to 0.7.2. The shared gate includes Create/Recover, Profile and kit coverage, restart, window policies, deep links, drafts, picker acknowledgements, two-window Documents saves, GBA save conflicts, Archive/Library, Chat, repeated System Save, Desktop/Terminal switching and Inbox handoff. Required source checks include whitespace, Home entropy, Rust formatting and narrow tests; installed checks bind source, built, installed and served artifacts. -Browser qualification is a substantial remaining time commitment. Notion retains **100 cold launches, 100 warm launches, 100 lifecycle cycles, 30 minutes of continuous audio/video and eight hours of mixed use**, with matching manual UX and the objective audit, for the two claimed roles. This requirement needs planned test ownership and capacity before a release date can be credible. +Browser qualification requires **100 cold launches, 100 warm launches, 100 lifecycle cycles, 30 minutes of continuous audio/video and eight hours of mixed use**, with matching manual UX and the objective audit, for both claimed roles. Assign test ownership and capacity before setting a release date. ## Integration and team coordination -Sash's shelf, Agent Space and composer were authored before this reporting period. This week's work preserved their history and integrated them into one Assistant. Irzhy's reconstruction also contributed Archive Manager, typed Assistant workspace and the Home menu protocol. Credit belongs to both the original UI work and the later adaptation. - -Sash's URUX adaptation `8b547590` entered through Irzhy's `7dd1780b`/`985fdffc`, followed by newer Sash dock/lock changes. Review the remaining behavior differences against this current implementation. Merging the old URUX tip in full would repeat already-adapted work. +Sash authored the shelf, Agent Space and composer before this period. This week preserved their history in one Assistant. Irzhy's adaptation also contributed Archive Manager, typed Assistant workspace and the Home menu protocol. The [contributor review](2026-09-11-contributor-review.md) records the adaptation chain, exact PR heads, earlier sources and remaining UI differences. Select further UI work against that integrated behavior. -Irzhy's remaining sequence is [PR59](https://github.com/Elacity/elastos-runtime/pull/59) authority cutover, then ready work from [PR62](https://github.com/Elacity/elastos-runtime/pull/62), after the locally merged [PR60](https://github.com/Elacity/elastos-runtime/pull/60) foundation. PR60 is still open on GitHub. The published integration checkpoint includes its original history and leaves PR59/PR62 unchanged. Agree PR60's disposition with Irzhy before the next merge. PR62 is a draft: the adopted media work comes from its inherited foundation, while its four follow-up commits remain outside this candidate. The cutover must preserve conditional Documents saves, model execution contracts and Runtime lifecycle ownership. +The remaining intake order is [PR59](https://github.com/Elacity/elastos-runtime/pull/59), then ready [PR62](https://github.com/Elacity/elastos-runtime/pull/62) work. [PR60](https://github.com/Elacity/elastos-runtime/pull/60) is included in the published candidate and remains open on GitHub; agree its disposition with Irzhy. PR62's four follow-up commits remain outside the candidate. Adopted media repairs came from its inherited foundation. The authority cutover must preserve conditional Documents saves, model execution contracts and Runtime lifecycle ownership. -The [contributor branch and PR review](2026-09-11-contributor-review.md) records the checked heads, GitHub states, local/adapted status and remaining work, including relevant sources outside the 13-PR September inventory. It also lists unresolved dispositions instead of assuming every open branch belongs in this release. - -| Owner in the plan | Next coordination responsibility | +| Owner in the plan | Next responsibility and proof | | --- | --- | -| Anders | Human acceptance, scope decisions, exact-candidate review and publication/release approvals. | -| Implementation owner | Complete the next required J3 behavior and its focused tests; preserve accepted J1 behavior. | -| Independent reviewer | Review the changed source and receipts, maintain one shared acceptance record, and coordinate target resources. | -| Sash | Review the integrated Home/Assistant experience and J1 human journey, including outside-tester coverage. | -| Browser owner | Finish the required reload slice and supply evidence for both placements and the updater slice. | -| Irzhy | Supply PR59/PR62 readiness and protected-video proof against the final callers. | -| Carrier and Content owner | Prove ordinary cold model delivery with bounded transfer, integrity, cancellation and cleanup. | +| Anders | Review the exact candidate, make scope and release decisions, and complete assigned human acceptance. | +| Implementation owner, to confirm | Deliver exact Marketplace handoff and safe removal; run focused source and installed checks while preserving accepted J1 behavior. | +| Independent reviewer, to confirm | Check source and artifact receipts against that result; record acceptance and coordinate target resources. | +| Sash | Review integrated Home/Assistant and the J1 human journey, with outside-tester coverage. | +| Browser owner, to confirm | Repair reload, qualify both placements, and provide the tested updater slice to J2. | +| Irzhy | Establish PR59/PR62 readiness against current callers and supply final-candidate protected-video proof. | +| Carrier and Content owner, to confirm | Prove cold model delivery, including bounded transfer, integrity, cancellation and cleanup. | -These responsibilities need confirmed owners before the next slice starts. Notion owns acceptance and decisions, `TASKS.md` owns execution, and `state.md` owns current source, installed versions and verification results. +Confirm the proposed assignments before execution. Notion owns acceptance and decisions, `TASKS.md` owns execution, and `state.md` owns source, installed versions and verification results. ## Process lessons and next meeting decisions -The development loop caught real defects, but repeated coordination, broad reviews and large task histories consumed substantial effort. Accepted user journeys advanced more slowly than implementation. The available account-wide allowance readings cannot isolate this project's exact cost. - -Version 0.4.1 of the development-loop skill now requires one useful installed result, an early evidence checkpoint, and a route change when effort outpaces acceptance. The owner and reviewer check assumptions and dependencies before execution. After a result, they revise only the affected next goals and reuse unchanged evidence. The improvement still to demonstrate is lower effort per accepted user behavior. +Repeated coordination, broad reviews and large task histories consumed substantial effort. The loop caught real defects, but accepted journeys advanced more slowly than implementation. Account-wide allowance readings cannot isolate this project's cost. -For the team sync, agree the next installed acceptance slice and its owner/reviewer, then resolve these items: +Development-loop skill 0.4.1 now requires one useful installed result, an early evidence checkpoint, and a route change when effort outpaces acceptance. The owner and reviewer check dependencies first, revise affected goals after each result and reuse unchanged evidence. Lower effort per accepted behavior still needs measurement. -1. Review the published integration [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) with this report and its contributor review. Website branch retirement is complete. Agree PR60's disposition with Irzhy. -2. Assign the next J3 slice, exact Marketplace-to-Assistant handoff and safe removal, with one owner and reviewer. Assign J1/J3 human checks, resolve Stop wording and reconcile Notion after the results. -3. Choose how to provide Linux model-test capacity and a reviewed Linux startup profile. The recorded gate requires about 13.4 GB additional free space before build margin. -4. Schedule the two-role Browser campaign and agree when its owned resources are available. -5. Confirm PR59/PR62 intake readiness and the external cryptographic review owner, scope and date. +1. Review PR64 with this report and its contributor review. Update Notion C1 status from the deployment evidence and Anders's sign-in confirmation, and agree PR60's disposition with Irzhy. +2. Confirm one J3 implementation owner and reviewer for Marketplace handoff and safe removal, then cold delivery. Assign J1/J3 human checks, resolve Stop criteria and reconcile Notion from the results. +3. Assign Linux model-test capacity and review its startup profile. The recorded capacity gate calls for about 13.4 GB additional free space before build margin. Allocate model-test capacity separately from the staged Runtime build. +4. Schedule the Browser campaign for both placements with owned build/test resources. +5. Confirm PR59/PR62 readiness and the external cryptographic review owner, scope and date. -Use one build/test owner for the next installed result, review it independently, then update the dependent goals from that evidence. +After independent review of the next installed result, update dependent goals from its evidence. Freeze the source before final three-platform packaging and shared acceptance. ## Evidence used -This report combines the live Notion plan, direct GitHub/Git ref checks, commit history and recorded audits. It is a document review, with no new product qualification run. Published evidence is linked above. The following audit files are included in the published reviewed checkpoint `06bf4e0f`: +This document review uses Notion, GitHub/Git checks, commit history and recorded product tests. The published source checkpoint `06bf4e0f` contains these evidence records: -- `docs/audits/2026-09-10-execution-handover.md`: delivery history and current handover. -- `docs/audits/2026-09-11-first-checkpoint.md`: combined Mac/Linux recovery and lifecycle proof. -- `docs/audits/2026-09-11-j3-bootstrap.md`: bootstrap scope and focused source tests. -- `docs/audits/2026-09-11-integration-preservation.md`: donor history, all 13 contributor PR dispositions and remaining behavior differences. -- `docs/audits/2026-09-11-assistant-convergence.md`: one-Assistant source, rendered, installed and human-proof boundaries. -- `TASKS.md` and `state.md`: current execution and evidence records. +- [Execution handover](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-10-execution-handover.md): delivery history. +- [First checkpoint](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-11-first-checkpoint.md): Mac/Linux recovery and lifecycle proof. +- [J3 bootstrap](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-11-j3-bootstrap.md): scope and focused tests. +- [Integration preservation](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-11-integration-preservation.md): donor history, 13 PR dispositions and behavior differences. +- [Assistant convergence](https://github.com/Elacity/elastos-runtime/blob/06bf4e0f/docs/audits/2026-09-11-assistant-convergence.md): source, rendered, installed and human-proof scope. +- [TASKS.md](https://github.com/Elacity/elastos-runtime/blob/1bb081ff/TASKS.md) and [state.md](https://github.com/Elacity/elastos-runtime/blob/1bb081ff/state.md): execution and evidence at the publication closeout. -The 5 September product/identity assessments, earlier enrollment/model source records, Browser analysis, QA triage and development-loop cost review also informed this report. The contributor companion links the reviewed PRs and identifies the limits of the comparison. +The 5 September product/identity assessments, enrollment/model records, Browser analysis, QA triage and process-cost review also informed this report. Qualification remains bounded by each recorded artifact and test scope. diff --git a/state.md b/state.md index 17d06cad8..0518a36a6 100644 --- a/state.md +++ b/state.md @@ -1,28 +1,39 @@ # State -## Current publication checkpoint — 2026-09-11 - -`feat/0.7.1-integration` is the active working/review branch. Reviewed source -`06bf4e0f11e602733e8eef861b835184ae683504`, tree -`4483f403a85d1ac27b4568f7b8cb4cf1a8a4e546`, is published in -[draft PR64](https://github.com/Elacity/elastos-runtime/pull/64) against -`upstream/0.7.1-dev` at `6c61c990`. This single documentation closeout adds the -[team report](docs/audits/2026-09-11-team-sync.md) and -[contributor review](docs/audits/2026-09-11-contributor-review.md). -The PR records the final documentation commit; product code remains the reviewed -checkpoint. Remote ancestry protects both website heads (`00003b6f` and -`06bf4e0f`). The website refs and redundant checkout are removed; 69 local branches -and 11 worktrees remain with their donor/history preservation gates. Main, dev, -contributor PRs and installed services retain their prior identities. - -The Mac Home retains installed code `3c2f9a80` and Runtime SHA-256 +## Current public preview and review checkpoint, 2026-09-11 + +`feat/0.7.1-integration` is the active working/review branch in +[draft PR64](https://github.com/Elacity/elastos-runtime/pull/64), based on +`upstream/0.7.1-dev` at `6c61c990`. Product checkpoint `06bf4e0f` preserves +both website heads and the reviewed contributor histories. The redundant website +refs and checkout are removed; the remaining donor/history gates still apply. + +Public website and Home now run `259666222f12b21283131cc7926ba7d0c1a52e99`, +tree `419bdba6a09ed37e8176ddfda9c22c1a270f6054`. This extends the publication +closeout with reconciled dependency locks and `--locked` setup builds. +All 25 required Linux manifests passed locked offline checks, and the cached +Linux source-home build completed. Application source retains the reviewed behavior. + +The authorized deployment at 17:42 UTC passed installed integrity, 573 artifact +file comparisons, running Runtime parity and public website/Home/Services/Assistant +hash checks. `/apps/home/` redirects to `/home/`. Account records, keys, user files +and provider settings match the preserved state. Migration and restart receipts +report zero migrated roots/objects. An owner-only identity directory and Runtime's +empty identity lock are the recorded metadata changes. Anders confirmed existing-account sign-in and saved work. The temporary stage +and bounded deployment rollback are removed. +The configured Browser Engine is unavailable and inactive custody remains unconfigured. +See [public preview proof](docs/audits/2026-09-11-public-preview.md). + +Mac checkpoint Homes retain installed code `3c2f9a80` and Runtime SHA-256 `1a55c86d95f5ca286bb63ee65242f3c23e19d0d1acd29888dc976b0033639666`. -This publication reused prior product evidence and ran the four basic source -gates; it performed no build, installation, restart or deployment. -Full J1 to J5 and D1 to D6 acceptance remains open under the approved plan. -The bounded publication/cleanup run stops here; broader execution and monitoring -stay paused. Next implementation: exact Marketplace-to-Assistant handoff and -safe removal, then cold Content/Carrier delivery. +Main/dev and contributor PRs retain their prior identities. Full J1 to J5 and +D1 to D6 acceptance remains open. The [team report](docs/audits/2026-09-11-team-sync.md) +and [contributor review](docs/audits/2026-09-11-contributor-review.md) cover +5 to 11 September. Next implementation: exact Marketplace-to-Assistant handoff +and safe removal, then cold Content/Carrier delivery. Recurring monitoring stays paused. + +Earlier dated records below retain their original artifacts and verdicts. This +checkpoint owns the current public installation status. ## Installed and source evidence From bf846cd8e1cd984374b988340c25210033938939 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 15:01:37 -0500 Subject: [PATCH 303/535] fix(browser): start viewer status and display attachment together on reload Reload scheduling: the viewer starts the Runtime status read and the display attachment together and requires both before media connection and public readiness. Remote run 96 spent 1019 ms on status and started attachment 3464 ms after reload, leaving 1536 ms of the five-second budget. Reload evidence: the journey harness records per-substep timings for the reload observer and rejects the old serial ordering. Verification: node --test scripts/browser-viewer-restore.test.mjs node --test scripts/lib/browser-journey-viewer-reload.test.mjs node --test scripts/lib/browser-journey-harness.test.mjs node --check scripts/home-passkey-virtual-auth-smoke.mjs node scripts/browser-entropy-check.mjs node scripts/home-entropy-check.mjs --- capsules/browser/browser/browser.js | 11 ++-- scripts/browser-viewer-restore.test.mjs | 47 +++++++++++++++- scripts/home-passkey-virtual-auth-smoke.mjs | 54 +++++++++++++++---- scripts/lib/browser-journey-harness.test.mjs | 53 +++++++++++++++++- scripts/lib/browser-journey-viewer-reload.mjs | 7 +-- .../browser-journey-viewer-reload.test.mjs | 14 +++++ 6 files changed, 165 insertions(+), 21 deletions(-) diff --git a/capsules/browser/browser/browser.js b/capsules/browser/browser/browser.js index da979e167..091bc823a 100644 --- a/capsules/browser/browser/browser.js +++ b/capsules/browser/browser/browser.js @@ -2420,19 +2420,20 @@ async function restoreRuntimePageViewer(summary) { showStatus("Restoring the Browser display...", { sticky: true }); startPageHeartbeat(); try { - await fetchPageStatus({ history: "replace", forceAddress: true }); - if (!runtimeViewerOwnerActive(owner)) return true; - publishRuntimePageForHost(currentPage); // Page status is diagnostic; attach replaces offers within retained Runtime authority. if (currentPage.display_session?.mode !== "webrtc_remote_display") { throw new Error("Runtime could not restore the Browser display."); } - const display = await attachRecoveredDisplay(summary, owner); - if (!display || !runtimeViewerOwnerActive(owner)) return true; + const [status, display] = await Promise.all([ + fetchPageStatus({ history: "replace", forceAddress: true }), + attachRecoveredDisplay(summary, owner), + ]); + if (!status || !display || !runtimeViewerOwnerActive(owner)) return true; if (display.mode !== "webrtc_remote_display") { throw new Error("Runtime could not restore the Browser display."); } currentPage = { ...currentPage, display_session: display }; + publishRuntimePageForHost(currentPage); currentDisplayMode = display.mode; syncDisplayInputFromSession(display); currentView = viewFromDisplaySession(display) || currentPage.view; diff --git a/scripts/browser-viewer-restore.test.mjs b/scripts/browser-viewer-restore.test.mjs index 04926c0eb..54af73e24 100644 --- a/scripts/browser-viewer-restore.test.mjs +++ b/scripts/browser-viewer-restore.test.mjs @@ -322,7 +322,7 @@ for (const stage of ['status', 'attachment']) for (const closeState of ['in-flig const before = h.calls.length; if (outcome === 'failure') wait.reject(new Error('late close-time failure')); else wait.resolve(); assert.equal(await pending, true); - assert.equal(attaches, stage === 'attachment' ? 1 : 0); + assert.equal(attaches, 1); // Both requests started before close began. assert.equal(h.calls.length, before); assert.deepEqual(h.failures, []); assert.equal(h.state.currentPage.display_session.display_generation, 'display:' + '1'.repeat(32)); assert.equal(h.state.currentPage.runtime_cleanup.id, 'cleanup-one'); @@ -334,3 +334,48 @@ test('cleanup for a different owner leaves this viewer eligible', async () => { h.state.pendingHomeWindowCloseDelivery = { owner: { page_id: 'other', generation: 7, runtime_cleanup: { schema: 'elastos.browser.cleanup-handle/v1', id: 'old-cleanup' } } }; await h.restore(summary()); assert.equal(h.calls.some(row => row[0] === 'connect'), true); }); + +test('restore starts attachment while fresh status is pending and publishes only after both validate', async () => { + const h = harness(), status = deferred(), attachment = deferred(), fetch = h.state.fetchJson, started = []; + h.state.fetchJson = async (path, options) => { + started.push(path.endsWith('/status') ? 'status' : 'attachment'); + await (path.endsWith('/status') ? status.promise : attachment.promise); + return fetch(path, options); + }; + const pending = h.restore(summary()); + try { + assert.deepEqual(started, ['status', 'attachment']); + attachment.resolve(); + await new Promise(resolve => setImmediate(resolve)); + assert.equal(h.calls.some(row => ['publish', 'connect'].includes(row[0])), false); + } finally { status.resolve(); attachment.resolve(); await pending; } + assert.equal(h.calls.filter(row => row[0] === 'connect').length, 1); +}); + +for (const field of ['schema', 'page_id', 'direct_network']) test(`invalid concurrent status ${field} cannot publish or connect`, async () => { + const h = harness(), fetch = h.state.fetchJson; + h.state.fetchJson = async (path, options) => { + const result = await fetch(path, options); + if (path.endsWith('/status')) result[field] = 'invalid'; + return result; + }; + await assert.rejects(h.restore(summary())); + assert.equal(h.calls.some(row => ['publish', 'connect'].includes(row[0])), false); + assert.equal(h.state.currentPage.runtime_cleanup.id, 'cleanup-one'); +}); + +test('attachment failure while status is pending cannot publish after late valid status', async () => { + const h = harness(), status = deferred(), fetch = h.state.fetchJson; + h.state.fetchJson = async (path, options) => { + if (path.endsWith('/webrtc')) throw new Error('attachment failed'); + await status.promise; + return fetch(path, options); + }; + const pending = h.restore(summary()); + // Release in the next turn so both old and new scheduling finish without a test hang. + setImmediate(() => status.resolve()); + await assert.rejects(pending, /attachment failed/); + await new Promise(resolve => setImmediate(resolve)); + assert.equal(h.calls.some(row => ['publish', 'connect'].includes(row[0])), false); + assert.equal(h.state.currentPage.display_session.display_generation, 'display:' + '1'.repeat(32)); +}); diff --git a/scripts/home-passkey-virtual-auth-smoke.mjs b/scripts/home-passkey-virtual-auth-smoke.mjs index 3848a6507..74da7e6d0 100644 --- a/scripts/home-passkey-virtual-auth-smoke.mjs +++ b/scripts/home-passkey-virtual-auth-smoke.mjs @@ -2408,6 +2408,7 @@ async function observeControlledBrowserRequests(page, appFrame, token, record, : /^\/api\/apps\/browser\/open(?:\/|$)/.test(url.pathname) && req.method() === "POST" ? "opening" : /\/pages\/[^/]+\/close$/.test(url.pathname) ? "closing" : /\/pages\/[^/]+\/status$/.test(url.pathname) ? "status" + : recordNavigation && url.pathname === "/api/apps/browser/summary" && url.searchParams.get("browser_instance") === instance ? "summary" : /\/pages\/[^/]+\/heartbeat$/.test(url.pathname) ? "heartbeat" : recordNavigation && /\/pages\/[^/]+\/webrtc$/.test(url.pathname) ? "signaling" : null; if (!kind) return; @@ -2427,6 +2428,7 @@ async function observeControlledBrowserRequests(page, appFrame, token, record, record({ ...event, phase: "response", status: res.status() }); return; } + record({ ...event, phase: "headers", status: res.status() }); const pending = res.json().then(body => browserViewerSignalMetadata(body), () => ({})) .then(details => record({ ...event, ...details, phase: "response", status: res.status() })) .finally(() => pendingResponses.delete(pending)); @@ -2570,26 +2572,54 @@ async function runControlledBrowserViewerReload(page, appFrame, token, readRecei const instance = new URL(appFrame.url()).searchParams.get("browser_instance"); const runtimeOrigin = new URL(appFrame.url()).origin; assert(instance, "Viewer reload requires the current Browser instance"); + const stateObservation = { clock: "performance.now", steps: [], dropped_steps: 0 }; + let sampleId = 0; try { - return await diagnoseBrowserViewerReload({ + const evidence = await diagnoseBrowserViewerReload({ expectedUrl, readReceipt, - readState: async ({ signal }) => { + readState: async ({ signal, deadlineMs }) => { + const sample = ++sampleId; + const step = async (name, action) => { + const row = { sample, step: name, start_ms: performance.now(), deadline_ms: deadlineMs }; + if (stateObservation.steps.length < 64) stateObservation.steps.push(row); + else stateObservation.dropped_steps++; + const finish = outcome => { + if (row.end_ms === undefined) { row.end_ms = performance.now(); row.outcome = outcome; } + }; + const aborted = () => finish("aborted"); + signal.addEventListener("abort", aborted, { once: true }); + try { + if (signal.aborted) { aborted(); throw new Error("Viewer reload observation canceled"); } + const result = await action(() => { if (row.end_ms === undefined) row.headers_ms = performance.now(); }); + if (signal.aborted) throw new Error("Viewer reload observation canceled"); + finish("complete"); + return result; + } catch (error) { finish(signal.aborted ? "aborted" : "failed"); throw error; } + finally { signal.removeEventListener("abort", aborted); } + }; const headers = { Origin: "null", "x-elastos-home-token": token }; - const response = await fetch(new URL(`/api/apps/browser/summary?browser_instance=${encodeURIComponent(instance)}`, runtimeOrigin), - { headers, signal }); - assert(response.ok, "Viewer reload Runtime summary failed"); - const { sessions } = await response.json(); + const { sessions } = await step("runtime_summary", async headersReceived => { + const response = await fetch(new URL(`/api/apps/browser/summary?browser_instance=${encodeURIComponent(instance)}`, runtimeOrigin), + { headers, signal }); + headersReceived(); + assert(response.ok, "Viewer reload Runtime summary failed"); + return response.json(); + }); const pageId = sessions?.recoverable_page?.page_id; let page_status = null; if (pageId) { - const status = await fetch(new URL(`/api/apps/browser/pages/${encodeURIComponent(pageId)}/status`, runtimeOrigin), { headers, signal }); - if (status.ok) page_status = await status.json(); - else assert(status.status === 404, "Viewer reload Runtime page status failed"); + page_status = await step("remote_page_status", async headersReceived => { + const status = await fetch(new URL(`/api/apps/browser/pages/${encodeURIComponent(pageId)}/status`, runtimeOrigin), { headers, signal }); + headersReceived(); + if (status.ok) return status.json(); + assert(status.status === 404, "Viewer reload Runtime page status failed"); + return null; + }); } let visible = { viewer: null, video: null }; try { // Read the document identity and media in one execution context. - visible = await appFrame.evaluate(readBrowserViewerReloadDocument); + visible = await step("viewer_document_metrics", () => appFrame.evaluate(readBrowserViewerReloadDocument)); } catch (error) { if (!/Execution context was destroyed|Cannot find context with specified id/.test(String(error.message))) throw error; } @@ -2605,8 +2635,10 @@ async function runControlledBrowserViewerReload(page, appFrame, token, readRecei .pressSequentially(suffix, { timeout: Math.ceil(timeoutMs) }), observeRequests: (record, { signal }) => observeControlledBrowserRequests(page, appFrame, token, record, { recordNavigation: true, signal }), }); + return { ...evidence, state_observation: stateObservation }; } catch (error) { - error.details = { ...error.details, viewer_reload: error.evidence || { ok: false, failure: "probe_setup_or_observation_failed" } }; + error.details = { ...error.details, viewer_reload: { ...(error.evidence || { ok: false, failure: "probe_setup_or_observation_failed" }), + state_observation: stateObservation } }; throw error; } } diff --git a/scripts/lib/browser-journey-harness.test.mjs b/scripts/lib/browser-journey-harness.test.mjs index 3f0de02d4..9bffa3d58 100644 --- a/scripts/lib/browser-journey-harness.test.mjs +++ b/scripts/lib/browser-journey-harness.test.mjs @@ -929,6 +929,12 @@ test("recovery observer binds requests to the viewer and keeps their start gener assert.deepEqual(records.map(value => [value.kind, value.phase, value.document_generation]), [["status", "request", 0], ["navigation", "commit", 1], ["status", "response", 0], ["closing", "request", 1]]); assert.equal(records[0].request_id, records[2].request_id); + const summary = request("/api/apps/browser/summary?browser_instance=instance-one"); + emit("request", request("/api/apps/browser/summary?browser_instance=foreign")); + emit("request", summary); + emit("response", { request: () => summary, status: () => 200 }); + assert.deepEqual(records.slice(-2).map(value => [value.kind, value.phase]), [["summary", "request"], ["summary", "response"]]); + const event = { source: frameWindow, origin: "null", data: { type: "elastos.home.browser-authority-renew.request/v1", homeToken: "token-one", browserInstance: "instance-one", requestId: "renew-one" } }; @@ -998,7 +1004,12 @@ for (const observation of ["ready", "document-transition", "unexpected-viewer-er await assert.rejects(run, error => error.details.viewer_reload.failure === "probe_setup_or_observation_failed"); assert.deepEqual(actions, []); } else { - assert.equal((await run).ok, true); + const outcome = await run; + assert.equal(outcome.ok, true); + assert.deepEqual(Array.from(outcome.state_observation.steps, row => row.step), + ["runtime_summary", "remote_page_status", "viewer_document_metrics"]); + assert.ok(outcome.state_observation.steps.every(row => row.end_ms >= row.start_ms)); + assert.equal(outcome.state_observation.steps[0].outcome, "complete"); assert.deepEqual(actions, ["wait-commit", "reload-frame", "input"]); assert.match(requests[1].url, /\/pages\/runtime-owner\/status$/); } @@ -1103,3 +1114,43 @@ test("observer cancellation disposes a listener handle returned after its setup assert.equal(disposed, 1); assert.ok([...listeners.values()].every(value => value.size === 0)); }); + +for (const mode of ["body-abort", "overflow"]) test(`reload substep evidence is bounded and retains ${mode}`, async () => { + const controller = new AbortController(); + let release, inFlight, count = 0; + const body = new Promise(resolve => { release = resolve; }); + const frame = { url: () => "http://localhost:61510/apps/browser/?browser_instance=instance-one", + evaluate: async () => ({ viewer: null, video: null }) }; + const wrapper = harnessFunction("runControlledBrowserViewerReload", { + readBrowserViewerReloadDocument, markStage: () => {}, assert: (value, message) => assert.ok(value, message), + fetch: async () => { count++; return { ok: true, json: () => mode === "body-abort" ? body : { sessions: {} } }; }, + diagnoseBrowserViewerReload: async callbacks => { + if (mode === "overflow") { + for (let i = 0; i < 40; i++) await callbacks.readState({ signal: controller.signal, deadlineMs: 5000 }); + return { ok: true }; + } + inFlight = callbacks.readState({ signal: controller.signal, deadlineMs: 5000 }).catch(() => {}); + await new Promise(resolve => setImmediate(resolve)); + controller.abort(); + throw Object.assign(new Error("state deadline"), { evidence: { ok: false, failure: "state_deadline" } }); + }, + }); + const run = wrapper({}, frame, "private-token", async () => {}, "http://private.invalid"); + if (mode === "overflow") { + const result = await run; + assert.equal(result.state_observation.steps.length, 64); + assert.equal(result.state_observation.dropped_steps, 16); + assert.equal(count, 40); + } else { + let evidence; + await assert.rejects(run, error => { evidence = error.details.viewer_reload; return evidence.failure === "state_deadline"; }); + assert.equal(evidence.state_observation.steps.length, 1); + const row = evidence.state_observation.steps[0]; + assert.equal(row.step, "runtime_summary"); assert.equal(row.outcome, "aborted"); + assert.ok(row.start_ms <= row.headers_ms && row.headers_ms <= row.end_ms); + const frozen = JSON.stringify(evidence); + release({ sessions: { recoverable_page: { page_id: "private-page" } } }); await inFlight; + assert.equal(JSON.stringify(evidence), frozen); assert.equal(count, 1); + assert.ok(!frozen.includes("private")); + } +}); diff --git a/scripts/lib/browser-journey-viewer-reload.mjs b/scripts/lib/browser-journey-viewer-reload.mjs index c7181d77b..1488bb72f 100644 --- a/scripts/lib/browser-journey-viewer-reload.mjs +++ b/scripts/lib/browser-journey-viewer-reload.mjs @@ -38,7 +38,8 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re }) { const started = clock.now(); const evidence = { schema: "elastos.browser.journey-viewer-reload/v1", ok: false, - requests: [], samples: [], dropped_requests: 0, observer: { stopped: false } }; + requests: [], samples: [], dropped_requests: 0, observer: { stopped: false }, + started_monotonic_ms: started }; let phase = "baseline", stop, failure, forbidden = false, original, initialViewer; let newDocument, newVideo, bytesRequired, initialReceipt, input, receiptSequence; const interruptions = new Set(); @@ -77,8 +78,8 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re function record(event) { if (event?.source_matches !== true) return; const navigation = event.kind === "navigation" && event.phase === "commit"; - if (!navigation && (!["opening", "closing", "renewal", "probe", "status", "heartbeat", "signaling"].includes(event.kind) || - !["request", "response", "failed"].includes(event.phase))) return; + if (!navigation && (!["opening", "closing", "renewal", "probe", "status", "summary", "heartbeat", "signaling"].includes(event.kind) || + !["request", "headers", "response", "failed"].includes(event.phase))) return; if (["opening", "closing"].includes(event.kind)) { forbidden = true; for (const interrupt of interruptions) interrupt(); diff --git a/scripts/lib/browser-journey-viewer-reload.test.mjs b/scripts/lib/browser-journey-viewer-reload.test.mjs index 6ce94dae5..6ca6febff 100644 --- a/scripts/lib/browser-journey-viewer-reload.test.mjs +++ b/scripts/lib/browser-journey-viewer-reload.test.mjs @@ -359,3 +359,17 @@ test("failed Runtime predicate is retained before any pending viewer check", asy assert.ok(evidence.binding_failure.failed_checks.includes(check)); assert.ok(evidence.binding_failure.at_ms >= evidence.reload_started_ms); }); + +test("reload retains summary and attachment header timing without accepting bodies or changing its gate", async () => { + const f = fixture(), reload = f.args.reloadViewer; + f.args.reloadViewer = async budget => { + await reload(budget); + f.event("summary", "request"); f.event("summary", "response", { status: 200 }); + f.event("signaling", "headers", { signal_type: "display_attach", status: 200 }); + }; + const evidence = await f.run(); + assert.equal(evidence.started_monotonic_ms, 0); + assert.equal(evidence.requests.filter(row => row.kind === "summary").length, 2); + assert.ok(evidence.requests.some(row => row.kind === "signaling" && row.event === "headers" && row.status === 200)); + assert.ok(evidence.reload_ms <= 5000); assert.ok(!JSON.stringify(evidence).includes(secret)); +}); From ca87f0224fc4f3caac53bd4e6f46c294bebfe96d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 15:16:00 -0500 Subject: [PATCH 304/535] feat(services): authorize remote model runs on the owning Runtime Add the `remote_model` service kind beside Exit and Engine. A contact can request a person's local AI model; the owner approves in Inbox; the signed decision carries a `remote_model_grant` with the Carrier ticket, the `local_engines` offer scope, the five model operations and a six-hour expiry. The consumer stores the grant on its request record. Carrier `provider_invoke` with target `model` dispatches to `gateway_model_service::invoke` with the Runtime data dir and verified network. The owning Runtime verifies the grant from the source endpoint key, requires an `assistant` or `home-agent` consumer capsule, rebuilds the request from the typed fields only, and binds the run to a deterministic remote principal derived from the source DID and the seed principal. Offers cross Carrier only when they are local engines; `runs_create` is denied for any other offer; provider errors cross as bounded classes. Every created run is indexed by grant so a denial cancels the runs still open and reads stay possible for a day after a run settles. Verification: cargo check --locked -p elastos-server --all-targets cargo fmt --all -- --check cargo test -p elastos-server --lib -- gateway_model_service provider_resource node scripts/home-entropy-check.mjs git diff --check --- .../crates/elastos-server/src/api/gateway.rs | 7 + .../src/api/gateway_home_runtime.rs | 42 + .../src/api/gateway_home_system.rs | 180 ++++- .../elastos-server/src/api/gateway_inbox.rs | 34 +- .../src/api/gateway_model_service.rs | 765 ++++++++++++++++++ .../src/api/gateway_provider_proxy.rs | 2 +- elastos/crates/elastos-server/src/carrier.rs | 13 +- .../elastos-server/src/provider_resource.rs | 4 +- scripts/home-entropy-check.mjs | 14 +- 9 files changed, 1046 insertions(+), 15 deletions(-) create mode 100644 elastos/crates/elastos-server/src/api/gateway_model_service.rs diff --git a/elastos/crates/elastos-server/src/api/gateway.rs b/elastos/crates/elastos-server/src/api/gateway.rs index a3dab6268..4c03dc7d7 100644 --- a/elastos/crates/elastos-server/src/api/gateway.rs +++ b/elastos/crates/elastos-server/src/api/gateway.rs @@ -76,6 +76,13 @@ mod gateway_inbox; mod gateway_inspect_actions; #[path = "gateway_marketplace.rs"] mod gateway_marketplace; +#[path = "gateway_model_service.rs"] +mod gateway_model_service; +pub(crate) use gateway_model_service::{ + cancel_grant_runs as cancel_remote_model_grant_runs, invoke as invoke_remote_model, + model_grant_id, ModelServiceGrant, MODEL_GRANT_SCHEMA, MODEL_GRANT_SCOPE, MODEL_GRANT_TTL_SECS, + MODEL_LOCAL_OFFER, MODEL_OPERATIONS, MODEL_SERVICE_KIND, MODEL_SERVICE_URI, +}; #[path = "gateway_origin.rs"] mod gateway_origin; #[path = "gateway_passkey_step_up.rs"] diff --git a/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs b/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs index 38209e51b..6ac1fd59f 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs @@ -1147,6 +1147,26 @@ pub(super) fn home_service_offers_for_people_contact( source: "people_contact".to_string(), runtime_contract: None, contact_id: Some(contact.contact_id.clone()), capsule_hint: Some("browser".to_string()), route: None, }); + offers.push(HomeServiceOfferSummary { + schema: "elastos.service.offer/v1".to_string(), + offer_id: format!("offer:{}:model", contact.contact_id), + service_uri: super::MODEL_SERVICE_URI.to_string(), + service_kind: super::MODEL_SERVICE_KIND.to_string(), + display_name: format!("{}'s AI model", contact.display_name), + provider_uri: Some("elastos://model/*".to_string()), + provider_label: "Remote model".to_string(), + policy_summary: "Ask this person to run their local AI model for you. Their Runtime keeps the model and decides every request; your conversation stays on your Home.".to_string(), + status: "requestable".to_string(), + enabled: false, + grant_required: true, + grant_scope: super::MODEL_GRANT_SCOPE.to_string(), + capsule_contract: "assistant -> Runtime service grant -> owner-bound model run".to_string(), + source: "people_contact".to_string(), + runtime_contract: None, + contact_id: Some(contact.contact_id.clone()), + capsule_hint: Some("assistant".to_string()), + route: None, + }); offers } @@ -1468,6 +1488,28 @@ fn home_local_service_offers( route: Some("/apps/browser/".to_string()), }); } + if data_dir.join("bin/model-provider").is_file() { + offers.push(HomeServiceOfferSummary { + schema: "elastos.service.offer/v1".to_string(), + offer_id: super::MODEL_LOCAL_OFFER.to_string(), + service_uri: "elastos://model/offers".to_string(), + service_kind: super::MODEL_SERVICE_KIND.to_string(), + display_name: "AI model".to_string(), + provider_uri: Some("elastos://model/*".to_string()), + provider_label: "Model provider".to_string(), + policy_summary: "Share your local AI model with accepted contacts. Each request needs your approval in Inbox; hosted models stay private; this Runtime decides every run and can revoke access.".to_string(), + status: "configured".to_string(), + enabled: true, + grant_required: true, + grant_scope: super::MODEL_GRANT_SCOPE.to_string(), + capsule_contract: "assistant -> model capability -> model provider".to_string(), + source: "local_provider".to_string(), + runtime_contract: None, + contact_id: None, + capsule_hint: Some("assistant".to_string()), + route: Some("/apps/assistant/".to_string()), + }); + } if data_dir.join("bin/ipfs-provider").is_file() { offers.push(HomeServiceOfferSummary { schema: "elastos.service.offer/v1".to_string(), diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index b9603dc6a..98767d4a4 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -224,6 +224,8 @@ struct HomeServicesPendingAccessDecision { remote_exit: Option, #[serde(default, skip_serializing_if = "Option::is_none")] remote_engine: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + remote_model: Option, } #[derive(Debug, Clone, Default, Serialize, Deserialize)] @@ -244,6 +246,8 @@ struct HomeServicesRemoteOfferRequestRecord { #[serde(default, skip_serializing_if = "Option::is_none")] remote_engine_grant: Option, #[serde(default, skip_serializing_if = "Option::is_none")] + remote_model_grant: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] pending_access_decision: Option, #[serde(default, skip_serializing_if = "Option::is_none")] access_decision_sha256: Option, @@ -2129,6 +2133,7 @@ pub(super) async fn services_offer_update( status: "requested".to_string(), installed_remote_exit_id: None, remote_engine_grant: None, + remote_model_grant: None, pending_access_decision: None, access_decision_sha256: None, }, @@ -2433,6 +2438,16 @@ fn home_services_supported_request(kind: &str, uri: &str) -> bool { (kind == HOME_REMOTE_EXIT_SERVICE_KIND && uri == HOME_BROWSER_EXIT_PEER_SERVICE_URI) || (kind == crate::carrier::ENGINE_SERVICE_KIND && uri == crate::carrier::ENGINE_SERVICE_URI) + || (kind == super::MODEL_SERVICE_KIND && uri == super::MODEL_SERVICE_URI) +} + +fn home_services_local_model_shared( + data_dir: &std::path::Path, + context: &HomeLaunchTokenContext, +) -> anyhow::Result { + Ok(home_services_selection_state(data_dir, context)? + .local_offer_ids + .contains(super::MODEL_LOCAL_OFFER)) } fn home_services_local_engine_shared( @@ -2453,6 +2468,10 @@ fn home_services_request_shared( && request.service_uri == crate::carrier::ENGINE_SERVICE_URI { home_services_local_engine_shared(data_dir, context) + } else if request.service_kind == super::MODEL_SERVICE_KIND + && request.service_uri == super::MODEL_SERVICE_URI + { + home_services_local_model_shared(data_dir, context) } else if request.service_kind == HOME_REMOTE_EXIT_SERVICE_KIND && request.service_uri == HOME_BROWSER_EXIT_PEER_SERVICE_URI { @@ -2471,7 +2490,7 @@ fn home_services_send_access_request( offer: &HomeServiceOfferSummary, ) -> anyhow::Result { if !home_services_supported_request(&offer.service_kind, &offer.service_uri) { - anyhow::bail!("only Browser Engine and Exit service requests are supported"); + anyhow::bail!("only Browser Engine, Exit and model service requests are supported"); } let contact = home_services_peer_contact_record_for_offer(data_dir, context, discovery_service, offer)?; @@ -2625,6 +2644,20 @@ fn home_services_send_access_decision( "expires_at":request.grant_expires_at, }); } + if decision == "approved" + && request.service_kind == super::MODEL_SERVICE_KIND + && request.service_uri == super::MODEL_SERVICE_URI + { + payload["remote_model_grant"] = serde_json::json!({ + "schema": super::MODEL_GRANT_SCHEMA, + "grant_id": super::model_grant_id(&request.request_id), + "peer_did": runtime.peer_id, + "connect_ticket": runtime.connect_ticket, + "offer_scope": "local_engines", + "operations": super::MODEL_OPERATIONS, + "expires_at": request.grant_expires_at, + }); + } crate::carrier::sign_service_message(data_dir, &mut payload)?; let delivery = services_peer_provider_request_blocking( &runtime.transport, @@ -2827,6 +2860,12 @@ fn home_services_merge_access_decision( } else { None }, + remote_model: if decision == "approved" && record.service_kind == super::MODEL_SERVICE_KIND + { + Some(home_services_remote_model_grant(context, record, payload)?) + } else { + None + }, remote_exit: if decision == "approved" && record.service_kind == HOME_REMOTE_EXIT_SERVICE_KIND { @@ -2901,6 +2940,7 @@ fn home_services_activate_pending_decision( .and_then(|exit| exit["id"].as_str()) .map(str::to_string); next_record.remote_engine_grant = pending.remote_engine.clone(); + next_record.remote_model_grant = pending.remote_model.clone(); next_record.pending_access_decision = None; next.updated_at = now_ts(); if pending.remote_exit.is_none() && record.installed_remote_exit_id.is_none() { @@ -3408,6 +3448,136 @@ fn home_services_remote_engine_grant( ) } +fn home_services_remote_model_grant( + context: &HomeLaunchTokenContext, + record: &HomeServicesRemoteOfferRequestRecord, + payload: &serde_json::Value, +) -> anyhow::Result { + let grant = &payload["remote_model_grant"]; + let revision = payload["created_at"] + .as_u64() + .ok_or_else(|| anyhow::anyhow!("model grant revision required"))?; + let expiry = grant["expires_at"] + .as_u64() + .ok_or_else(|| anyhow::anyhow!("model grant expiry required"))?; + let ticket = home_services_payload_text( + grant, + "connect_ticket", + HOME_SERVICES_REMOTE_EXIT_TICKET_MAX_BYTES, + ) + .ok_or_else(|| anyhow::anyhow!("model Carrier ticket required"))?; + anyhow::ensure!( + grant["schema"] == super::MODEL_GRANT_SCHEMA + && grant["grant_id"].as_str() + == Some(super::model_grant_id(&record.request_id).as_str()) + && grant["peer_did"].as_str() == Some(&record.target_peer_id) + && grant["offer_scope"] == "local_engines" + && grant["operations"] == serde_json::json!(super::MODEL_OPERATIONS) + && expiry > now_ts() + && expiry > revision + && expiry - revision <= super::MODEL_GRANT_TTL_SECS, + "model grant is expired or does not match the requested service" + ); + record.target_peer_id.parse::()?; + Ok(serde_json::json!({ + "schema": super::MODEL_GRANT_SCHEMA, + "id": format!("remote-model-{}", hex::encode(&Sha256::digest(record.request_id.as_bytes())[..8])), + "grant_id": grant["grant_id"], + "peer_did": record.target_peer_id, + "connect_ticket": ticket, + "principal_id": context.principal_id, + "offer_scope": "local_engines", + "operations": grant["operations"], + "expires_at": expiry, + })) +} + +/// Destination authority for a remote model operation. The grant must have +/// been issued by this Runtime, stay approved and shared, and the requester +/// must still be an accepted contact whose device key matches the Carrier +/// source endpoint. +pub(crate) fn authorize_home_service_model( + data_dir: &std::path::Path, + network: &crate::collaboration_network::VerifiedCollaborationNetworkProfile, + source: &iroh::PublicKey, + grant_id: &str, + requester_principal_id: &str, + now: u64, +) -> anyhow::Result { + let principals = crate::auth::active_passkey_principals(data_dir)?; + anyhow::ensure!( + principals.len() <= 64, + "model authority scope is unavailable" + ); + let local_did = crate::collaboration_profile_authority::load_existing_device_did(data_dir)? + .ok_or_else(|| anyhow::anyhow!("model signing identity unavailable"))?; + let mut authorized = None; + for principal in principals { + let context = HomeLaunchTokenContext { + principal_id: principal.principal_id, + proof_binding_id: Some(principal.proof_binding_id), + session_id: String::new(), + grant_id: String::new(), + }; + let state = home_services_requests_state(data_dir, &context)?; + for record in state + .requests + .values() + .filter(|record| super::model_grant_id(&record.request_id) == grant_id) + { + anyhow::ensure!( + record.authenticated_request + && record.status == "approved" + && record.service_uri == super::MODEL_SERVICE_URI + && record.service_kind == super::MODEL_SERVICE_KIND + && home_services_local_model_shared(data_dir, &context)?, + "model grant is not active" + ); + let profile = load_profile_authority_for_context(data_dir, &context)? + .ok_or_else(|| anyhow::anyhow!("model profile authority unavailable"))?; + let store = crate::collaboration_contact_store::CollaborationContactStore::new( + data_dir, + &context.principal_id, + &home_browser_localhost_root(&context), + network.clone(), + &profile, + &local_did, + )?; + anyhow::ensure!( + store.snapshot()?.contacts().iter().any( + |contact| crate::carrier::did_to_public_key( + contact.remote_presence_device_did() + ) == Some(*source) + && record.requester_did.as_deref() + == Some(contact.remote_presence_device_did()) + ), + "model requester is no longer an accepted contact" + ); + let grant = super::ModelServiceGrant { + provider_principal_id: context.principal_id.clone(), + requester_principal_id: record + .requester_principal_id + .clone() + .ok_or_else(|| anyhow::anyhow!("model requester principal required"))?, + requester_endpoint: record + .requester_peer_id + .parse::() + .ok() + .ok_or_else(|| anyhow::anyhow!("model requester identity invalid"))?, + grant_id: grant_id.to_string(), + revision: record.updated_at, + expires_at: record + .grant_expires_at + .ok_or_else(|| anyhow::anyhow!("model grant expiry required"))?, + }; + grant.validate(source, requester_principal_id, now)?; + anyhow::ensure!(authorized.is_none(), "model grant ownership is ambiguous"); + authorized = Some(grant); + } + } + authorized.ok_or_else(|| anyhow::anyhow!("model grant was not issued by this Runtime")) +} + pub(in crate::api::gateway) fn home_services_remote_engine_grants( data_dir: &std::path::Path, context: &HomeLaunchTokenContext, @@ -3818,8 +3988,12 @@ fn home_services_mark_access_request( let mut request = request; request.status = status.to_string(); request.updated_at = revision; - request.grant_expires_at = (status == "approved") - .then(|| revision.saturating_add(crate::carrier::EXIT_GRANT_TTL_SECS)); + let grant_ttl = if request.service_kind == super::MODEL_SERVICE_KIND { + super::MODEL_GRANT_TTL_SECS + } else { + crate::carrier::EXIT_GRANT_TTL_SECS + }; + request.grant_expires_at = (status == "approved").then(|| revision.saturating_add(grant_ttl)); if request.service_uri == HOME_BROWSER_EXIT_PEER_SERVICE_URI { let limit = (status == "approved").then_some(crate::carrier::EXIT_GRANT_MAX_STREAMS); request.exit_max_active_streams = limit; diff --git a/elastos/crates/elastos-server/src/api/gateway_inbox.rs b/elastos/crates/elastos-server/src/api/gateway_inbox.rs index ade5962ab..5f9169356 100644 --- a/elastos/crates/elastos-server/src/api/gateway_inbox.rs +++ b/elastos/crates/elastos-server/src/api/gateway_inbox.rs @@ -383,16 +383,34 @@ async fn dispatch_inbox_action( let data_dir = data_dir.clone(); let context = context.clone(); let request_id = request_id.to_string(); - return tokio::task::spawn_blocking(move || { - deny_home_service_access_request( - &data_dir, - &context, - discovery_service.as_ref(), - &request_id, - ) + let grant_id = super::model_grant_id(&request_id); + let denied = tokio::task::spawn_blocking({ + let data_dir = data_dir.clone(); + move || { + deny_home_service_access_request( + &data_dir, + &context, + discovery_service.as_ref(), + &request_id, + ) + } }) .await - .map_err(|err| anyhow::anyhow!(err))?; + .map_err(|err| anyhow::anyhow!(err))??; + // Revocation closes future authority and settles the runs it had opened. + if let Some(registry) = state.provider_registry.clone() { + match super::cancel_remote_model_grant_runs(registry, &data_dir, &grant_id).await { + Ok(cancelled) if !cancelled.is_empty() => tracing::info!( + "revoked model grant {grant_id}: cancel requested for {} run(s)", + cancelled.len() + ), + Ok(_) => {} + Err(err) => { + tracing::warn!("revoked model grant {grant_id}: cancel sweep failed: {err}") + } + } + } + return Ok(denied); } if let Some(request_id) = action_id.strip_prefix("inspect-approve-request:") { let Some(step_up_token) = action.step_up_token.as_deref() else { diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs new file mode 100644 index 000000000..297f8bb60 --- /dev/null +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -0,0 +1,765 @@ +//! Remote model service: the destination side of a `remote_model` grant. +//! +//! A seed Runtime routes typed model operations over Carrier `provider_invoke` +//! with target `model`. This Runtime owns the decision. It verifies the grant +//! from the source endpoint key, rewrites the request into a destination-owned +//! binding, filters offers to local engines, calls its own model provider, and +//! records every run it created for that grant so revocation can find them. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::Duration; + +use anyhow::Context as _; +use elastos_runtime::provider::ProviderRegistry; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; + +use super::gateway_home_token::HomeLaunchTokenContext; +use super::gateway_provider_proxy::normalize_model_provider_request; + +pub(crate) const MODEL_SERVICE_KIND: &str = "remote_model"; +pub(crate) const MODEL_SERVICE_URI: &str = "elastos://peer/model"; +pub(crate) const MODEL_LOCAL_OFFER: &str = "local:provider:model"; +pub(crate) const MODEL_GRANT_SCHEMA: &str = "elastos.service.remote-model-grant/v1"; +pub(crate) const MODEL_GRANT_SCOPE: &str = "principal_scoped_remote_model_grant"; +pub(crate) const MODEL_GRANT_TTL_SECS: u64 = 6 * 3600; +pub(crate) const MODEL_RUN_RETENTION_SECS: u64 = 24 * 3600; +pub(crate) const MODEL_OPERATIONS: [&str; 5] = [ + "offers_list", + "runs_create", + "runs_get", + "runs_events", + "runs_cancel", +]; +const MODEL_CONSUMER_CAPSULES: [&str; 2] = ["assistant", "home-agent"]; +const MODEL_RUN_INDEX_SCHEMA: &str = "elastos.services.model-runs/v1"; +const MODEL_RUN_INDEX_MAX: usize = 512; +const MODEL_REQUEST_MAX_BYTES: usize = 256 * 1024; +const AUTHORITY_DEADLINE: Duration = Duration::from_secs(1); + +pub(crate) fn model_grant_id(request_id: &str) -> String { + let digest = Sha256::digest(request_id.as_bytes()); + format!("services-remote-model-grant-{}", hex::encode(&digest[..8])) +} + +/// The consumer principal's identity on this Runtime. Deterministic per source +/// Runtime and seed principal, so retries and restarts keep run ownership. +pub(crate) fn remote_principal_id( + source_endpoint_did: &str, + requester_principal_id: &str, +) -> String { + let digest = + Sha256::digest(format!("{source_endpoint_did}:{requester_principal_id}").as_bytes()); + format!("remote:{}", hex::encode(&digest[..16])) +} + +/// A grant this Runtime issued for its shared local model offers. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ModelServiceGrant { + pub provider_principal_id: String, + pub requester_principal_id: String, + pub requester_endpoint: iroh::PublicKey, + pub grant_id: String, + pub revision: u64, + pub expires_at: u64, +} + +impl ModelServiceGrant { + pub fn validate( + &self, + source: &iroh::PublicKey, + requester_principal_id: &str, + now: u64, + ) -> anyhow::Result<()> { + anyhow::ensure!( + self.requester_endpoint == *source + && requester_principal_id == self.requester_principal_id, + "model grant requester mismatch" + ); + anyhow::ensure!( + self.revision > 0 + && self.revision <= now.saturating_add(30) + && self.expires_at > now + && self.expires_at > self.revision + && self.expires_at - self.revision <= MODEL_GRANT_TTL_SECS, + "model grant expired or invalid" + ); + Ok(()) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct RemoteModelRunRecord { + pub grant_id: String, + pub source_endpoint_did: String, + pub requester_principal_id: String, + pub remote_principal_id: String, + pub capsule_id: String, + pub offer_id: String, + pub request_id: String, + pub created_at: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub terminal_at: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +struct RemoteModelRunIndex { + schema: String, + #[serde(default)] + runs: BTreeMap, +} + +fn run_index_path(data_dir: &Path) -> PathBuf { + data_dir.join("services-model-runs.json") +} + +fn run_index_lock() -> &'static std::sync::Mutex<()> { + static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + &LOCK +} + +fn read_run_index(data_dir: &Path) -> anyhow::Result { + let path = run_index_path(data_dir); + let bytes = match std::fs::read(&path) { + Ok(bytes) => bytes, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => { + return Ok(RemoteModelRunIndex { + schema: MODEL_RUN_INDEX_SCHEMA.to_string(), + runs: BTreeMap::new(), + }); + } + Err(err) => return Err(err).context("model run index read"), + }; + let index: RemoteModelRunIndex = + serde_json::from_slice(&bytes).context("model run index parse")?; + anyhow::ensure!( + index.schema == MODEL_RUN_INDEX_SCHEMA, + "model run index schema mismatch" + ); + Ok(index) +} + +fn write_run_index(data_dir: &Path, index: &RemoteModelRunIndex) -> anyhow::Result<()> { + let path = run_index_path(data_dir); + let temp = path.with_extension("json.tmp"); + let bytes = serde_json::to_vec_pretty(index)?; + std::fs::write(&temp, bytes).context("model run index write")?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(&temp, std::fs::Permissions::from_mode(0o600))?; + } + std::fs::rename(&temp, &path).context("model run index commit")?; + Ok(()) +} + +fn update_run_index( + data_dir: &Path, + now: u64, + update: impl FnOnce(&mut RemoteModelRunIndex) -> anyhow::Result, +) -> anyhow::Result { + let _guard = run_index_lock() + .lock() + .map_err(|_| anyhow::anyhow!("model run index unavailable"))?; + let mut index = read_run_index(data_dir)?; + index.runs.retain(|_, record| { + record + .terminal_at + .is_none_or(|terminal_at| now.saturating_sub(terminal_at) <= MODEL_RUN_RETENTION_SECS) + }); + let result = update(&mut index)?; + anyhow::ensure!( + index.runs.len() <= MODEL_RUN_INDEX_MAX, + "model run index is full" + ); + write_run_index(data_dir, &index)?; + Ok(result) +} + +pub(crate) fn remote_model_run( + data_dir: &Path, + run_id: &str, +) -> anyhow::Result> { + let _guard = run_index_lock() + .lock() + .map_err(|_| anyhow::anyhow!("model run index unavailable"))?; + Ok(read_run_index(data_dir)?.runs.get(run_id).cloned()) +} + +pub(crate) fn remote_model_runs_for_grant( + data_dir: &Path, + grant_id: &str, +) -> anyhow::Result> { + let _guard = run_index_lock() + .lock() + .map_err(|_| anyhow::anyhow!("model run index unavailable"))?; + Ok(read_run_index(data_dir)? + .runs + .into_iter() + .filter(|(_, record)| record.grant_id == grant_id) + .collect()) +} + +fn safe_id(value: &str, max: usize) -> bool { + !value.is_empty() + && value.len() <= max + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_' | b':' | b'.')) +} + +fn denied(code: &str, message: &str) -> Value { + json!({ "ok": false, "code": code, "error": message }) +} + +/// Bounded error classes cross Carrier; raw provider text stays on this Runtime. +fn redact_provider_error(err: &str) -> &'static str { + let lower = err.to_ascii_lowercase(); + if lower.contains("rate") && lower.contains("limit") || lower.contains("concurrency") { + "rate_limited" + } else if lower.contains("offer") + && (lower.contains("unknown") + || lower.contains("not found") + || lower.contains("unavailable")) + { + "offer_unavailable" + } else if lower.contains("owner") || lower.contains("denied") || lower.contains("unauthorized") + { + "denied" + } else { + "provider_failure" + } +} + +fn provider_status_error(result: &Value) -> Option { + (result.get("status").and_then(Value::as_str) == Some("error")).then(|| { + result + .get("message") + .and_then(Value::as_str) + .or_else(|| result.get("error").and_then(Value::as_str)) + .unwrap_or("provider error") + .to_string() + }) +} + +fn remote_context(record_principal: &str, grant_id: &str) -> HomeLaunchTokenContext { + HomeLaunchTokenContext { + principal_id: record_principal.to_string(), + proof_binding_id: None, + session_id: grant_id.to_string(), + grant_id: grant_id.to_string(), + } +} + +/// Local engine offers are the shareable set. Hosted adapters stay private to +/// this Runtime's owner. +fn shareable_offers(result: &Value) -> Vec { + result + .pointer("/data/offers") + .or_else(|| result.get("offers")) + .and_then(Value::as_array) + .map(|offers| { + offers + .iter() + .filter(|offer| offer.get("hosted").is_none_or(Value::is_null)) + .cloned() + .collect() + }) + .unwrap_or_default() +} + +fn with_offers(mut result: Value, offers: Vec) -> Value { + if result.pointer("/data/offers").is_some() { + result["data"]["offers"] = Value::Array(offers); + } else { + result["offers"] = Value::Array(offers); + } + result +} + +async fn read_authority( + data_dir: &Path, + network: &crate::collaboration_network::VerifiedCollaborationNetworkProfile, + source: iroh::PublicKey, + grant_id: &str, + requester_principal_id: &str, +) -> anyhow::Result { + let data_dir = data_dir.to_path_buf(); + let network = network.clone(); + let grant_id = grant_id.to_string(); + let requester_principal_id = requester_principal_id.to_string(); + tokio::time::timeout( + AUTHORITY_DEADLINE, + tokio::task::spawn_blocking(move || { + super::gateway_home_system::authorize_home_service_model( + &data_dir, + &network, + &source, + &grant_id, + &requester_principal_id, + crate::auth::now_ts(), + ) + }), + ) + .await + .context("model authority deadline")?? +} + +fn run_terminal(result: &Value) -> bool { + let status = result + .pointer("/data/run/status") + .or_else(|| result.pointer("/data/status")) + .or_else(|| result.pointer("/run/status")) + .or_else(|| result.get("status")) + .and_then(Value::as_str) + .unwrap_or_default(); + matches!( + status, + "completed" | "failed" | "cancelled" | "settlement_unknown" + ) +} + +/// Serve one model operation for a remote consumer. Every path returns a +/// Carrier `provider_invoke` response object. +pub(crate) async fn invoke( + registry: Arc, + data_dir: &Path, + network: Option, + source: iroh::PublicKey, + data: &Value, +) -> Value { + let operation = data["operation"].as_str().unwrap_or_default(); + if !MODEL_OPERATIONS.contains(&operation) { + return denied( + "invalid_provider_invocation", + "model operation is not part of the service contract", + ); + } + let Some(request) = data.get("request").filter(|request| request.is_object()) else { + return denied( + "invalid_provider_invocation", + "model request must be an object", + ); + }; + if serde_json::to_vec(request) + .map(|bytes| bytes.len() > MODEL_REQUEST_MAX_BYTES) + .unwrap_or(true) + { + return denied( + "invalid_provider_invocation", + "model request exceeds the service size bound", + ); + } + if request.get("op").and_then(Value::as_str) != Some(operation) { + return denied( + "invalid_provider_invocation", + "model request op must match the invocation operation", + ); + } + let remote = &request["remote_model"]; + let (Some(grant_id), Some(requester_principal_id), Some(capsule_id)) = ( + remote["grant_id"] + .as_str() + .filter(|value| safe_id(value, 128)), + remote["principal_id"] + .as_str() + .filter(|value| safe_id(value, 128)), + remote["capsule_id"] + .as_str() + .filter(|value| MODEL_CONSUMER_CAPSULES.contains(value)), + ) else { + return denied( + "denied", + "model request requires a remote grant, principal and consumer capsule", + ); + }; + let Some(network) = network else { + return denied( + "denied", + "model service network is not configured on this Runtime", + ); + }; + let source_did = match crate::carrier::public_key_to_did(&source) { + Ok(did) => did, + Err(_) => return denied("denied", "model requester identity is invalid"), + }; + let now = crate::auth::now_ts(); + + // A fresh local request: only the fields the typed contract accepts. + let mut local = request.clone(); + let local_object = local.as_object_mut().expect("request is an object"); + local_object.remove("remote_model"); + local_object.remove("_runtime_invocation"); + + let (context, indexed_run) = match operation { + "offers_list" | "runs_create" => { + let grant = + match read_authority(data_dir, &network, source, grant_id, requester_principal_id) + .await + { + Ok(grant) => grant, + Err(err) => { + tracing::info!("remote model authority denied: {err}"); + return denied("denied", "model grant is not active for this requester"); + } + }; + if let Err(err) = grant.validate(&source, requester_principal_id, now) { + tracing::info!("remote model grant rejected: {err}"); + return denied("denied", "model grant is not active for this requester"); + } + ( + remote_context( + &remote_principal_id(&source_did, requester_principal_id), + &grant.grant_id, + ), + None, + ) + } + _ => { + let Some(run_id) = local_object + .get("run_id") + .and_then(Value::as_str) + .map(str::to_string) + else { + return denied( + "invalid_provider_invocation", + "model run operation requires run_id", + ); + }; + let record = match remote_model_run(data_dir, &run_id) { + Ok(Some(record)) => record, + Ok(None) => return denied("denied", "model run is not owned by this grant"), + Err(err) => { + tracing::warn!("remote model run index unavailable: {err}"); + return denied("provider_failure", "model run index is unavailable"); + } + }; + if record.grant_id != grant_id + || record.source_endpoint_did != source_did + || record.requester_principal_id != requester_principal_id + || record.capsule_id != capsule_id + { + return denied("denied", "model run is not owned by this grant"); + } + ( + remote_context(&record.remote_principal_id, grant_id), + Some((run_id, record)), + ) + } + }; + + let normalized = match normalize_model_provider_request(operation, &local, &context, capsule_id) + { + Ok(normalized) => normalized, + Err((_, message)) => { + tracing::info!("remote model request rejected: {message}"); + return denied( + "invalid_provider_invocation", + "model request did not match the typed contract", + ); + } + }; + + if operation == "runs_create" { + let offers = match registry + .send_raw("model", &json!({ "op": "offers_list" })) + .await + { + Ok(result) => shareable_offers(&result), + Err(err) => { + tracing::warn!("remote model offers unavailable: {err}"); + return denied( + "offer_unavailable", + "model offers are unavailable on this Runtime", + ); + } + }; + let offer_id = normalized["offer_id"].as_str().unwrap_or_default(); + if !offers + .iter() + .any(|offer| offer["id"].as_str() == Some(offer_id)) + { + return denied( + "offer_unavailable", + "model offer is not shared through this grant", + ); + } + } + + let result = match registry.send_raw("model", &normalized).await { + Ok(result) => result, + Err(err) => { + let class = redact_provider_error(&err.to_string()); + tracing::info!("remote model provider error ({class}): {err}"); + return denied(class, "model provider rejected the operation"); + } + }; + if let Some(message) = provider_status_error(&result) { + let class = redact_provider_error(&message); + tracing::info!("remote model provider status error ({class}): {message}"); + return denied(class, "model provider rejected the operation"); + } + + let result = match operation { + "offers_list" => with_offers(result.clone(), shareable_offers(&result)), + "runs_create" => { + let run_id = result + .pointer("/data/run/id") + .or_else(|| result.pointer("/data/run_id")) + .or_else(|| result.pointer("/run/id")) + .or_else(|| result.get("run_id")) + .and_then(Value::as_str) + .map(str::to_string); + if let Some(run_id) = run_id { + let record = RemoteModelRunRecord { + grant_id: grant_id.to_string(), + source_endpoint_did: source_did.clone(), + requester_principal_id: requester_principal_id.to_string(), + remote_principal_id: context.principal_id.clone(), + capsule_id: capsule_id.to_string(), + offer_id: normalized["offer_id"] + .as_str() + .unwrap_or_default() + .to_string(), + request_id: normalized + .pointer("/runtime_binding/request_id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(), + created_at: now, + terminal_at: run_terminal(&result).then_some(now), + }; + if let Err(err) = update_run_index(data_dir, now, |index| { + index.runs.entry(run_id.clone()).or_insert(record); + Ok(()) + }) { + tracing::warn!("remote model run index write failed: {err}"); + return denied( + "provider_failure", + "model run could not be recorded on this Runtime", + ); + } + } + result + } + _ => { + if let Some((run_id, record)) = indexed_run { + if record.terminal_at.is_none() && run_terminal(&result) { + let _ = update_run_index(data_dir, now, |index| { + if let Some(entry) = index.runs.get_mut(&run_id) { + entry.terminal_at = Some(now); + } + Ok(()) + }); + } + } + result + } + }; + json!({ "ok": true, "result": result }) +} + +/// Cancel every open run created under a grant, after revoke or denial. +/// Returns the run ids that received a cancel request. +pub(crate) async fn cancel_grant_runs( + registry: Arc, + data_dir: &Path, + grant_id: &str, +) -> anyhow::Result> { + let mut cancelled = Vec::new(); + for (run_id, record) in remote_model_runs_for_grant(data_dir, grant_id)? { + if record.terminal_at.is_some() { + continue; + } + let context = remote_context(&record.remote_principal_id, grant_id); + let request = json!({ + "op": "runs_cancel", + "run_id": run_id, + "request_id": format!("revoke:{}", record.request_id), + }); + let Ok(normalized) = + normalize_model_provider_request("runs_cancel", &request, &context, &record.capsule_id) + else { + continue; + }; + match registry.send_raw("model", &normalized).await { + Ok(result) if provider_status_error(&result).is_none() => cancelled.push(run_id), + Ok(result) => tracing::info!( + "revoke cancel for {run_id} returned {}", + provider_status_error(&result).unwrap_or_default() + ), + Err(err) => tracing::info!("revoke cancel for {run_id} failed: {err}"), + } + } + Ok(cancelled) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn remote_principal_id_is_deterministic_and_per_principal() { + let a1 = remote_principal_id("did:key:z6Mksource", "principal-a"); + let a2 = remote_principal_id("did:key:z6Mksource", "principal-a"); + let b = remote_principal_id("did:key:z6Mksource", "principal-b"); + let other_runtime = remote_principal_id("did:key:z6Mkother", "principal-a"); + assert_eq!(a1, a2); + assert_ne!(a1, b); + assert_ne!(a1, other_runtime); + assert!(a1.starts_with("remote:")); + assert_eq!(a1.len(), "remote:".len() + 32); + } + + #[test] + fn model_grant_id_is_stable_and_distinct_from_engine_ids() { + let id = model_grant_id("request-1"); + assert_eq!(id, model_grant_id("request-1")); + assert!(id.starts_with("services-remote-model-grant-")); + assert_ne!(id, crate::carrier::engine_grant_id("request-1")); + } + + #[test] + fn grant_validation_checks_requester_expiry_and_ttl() { + let key = iroh::SecretKey::from_bytes(&[71; 32]).public(); + let other = iroh::SecretKey::from_bytes(&[72; 32]).public(); + let grant = ModelServiceGrant { + provider_principal_id: "owner".into(), + requester_principal_id: "seed-principal".into(), + requester_endpoint: key, + grant_id: model_grant_id("r"), + revision: 1_000, + expires_at: 1_000 + MODEL_GRANT_TTL_SECS, + }; + assert!(grant.validate(&key, "seed-principal", 1_500).is_ok()); + assert!(grant.validate(&other, "seed-principal", 1_500).is_err()); + assert!(grant.validate(&key, "another-principal", 1_500).is_err()); + assert!(grant + .validate(&key, "seed-principal", 1_000 + MODEL_GRANT_TTL_SECS) + .is_err()); + let too_long = ModelServiceGrant { + expires_at: 1_000 + MODEL_GRANT_TTL_SECS + 1, + ..grant.clone() + }; + assert!(too_long.validate(&key, "seed-principal", 1_500).is_err()); + } + + #[test] + fn shareable_offers_exclude_hosted_adapters() { + let result = json!({ "status": "ok", "data": { "offers": [ + { "id": "qwen", "hosted": null }, + { "id": "gpt", "hosted": { "placement": "hosted" } }, + { "id": "local-2" } + ] } }); + let ids: Vec<_> = shareable_offers(&result) + .into_iter() + .map(|offer| offer["id"].as_str().unwrap().to_string()) + .collect(); + assert_eq!(ids, vec!["qwen", "local-2"]); + let filtered = with_offers(result.clone(), shareable_offers(&result)); + assert_eq!(filtered["data"]["offers"].as_array().unwrap().len(), 2); + } + + #[test] + fn provider_errors_cross_carrier_only_as_bounded_classes() { + assert_eq!( + redact_provider_error("run concurrency limit reached for offer"), + "rate_limited" + ); + assert_eq!( + redact_provider_error("unknown offer 'x' at /Users/anders/models"), + "offer_unavailable" + ); + assert_eq!(redact_provider_error("run owner mismatch"), "denied"); + assert_eq!( + redact_provider_error("llama-server exited with code 137 at /tmp/x"), + "provider_failure" + ); + } + + #[test] + fn run_index_round_trips_and_prunes_old_terminal_runs() { + let dir = tempfile::tempdir().unwrap(); + let record = RemoteModelRunRecord { + grant_id: "g".into(), + source_endpoint_did: "did:key:z6Mksource".into(), + requester_principal_id: "p".into(), + remote_principal_id: remote_principal_id("did:key:z6Mksource", "p"), + capsule_id: "assistant".into(), + offer_id: "qwen".into(), + request_id: "req-1".into(), + created_at: 100, + terminal_at: None, + }; + update_run_index(dir.path(), 100, |index| { + index.runs.insert("run:sha256:aaaa".into(), record.clone()); + index.runs.insert( + "run:sha256:bbbb".into(), + RemoteModelRunRecord { + terminal_at: Some(50), + request_id: "req-0".into(), + ..record.clone() + }, + ); + Ok(()) + }) + .unwrap(); + assert_eq!( + remote_model_run(dir.path(), "run:sha256:aaaa").unwrap(), + Some(record.clone()) + ); + assert_eq!( + remote_model_runs_for_grant(dir.path(), "g").unwrap().len(), + 2 + ); + update_run_index(dir.path(), 50 + MODEL_RUN_RETENTION_SECS + 1, |_| Ok(())).unwrap(); + assert_eq!( + remote_model_runs_for_grant(dir.path(), "g").unwrap().len(), + 1 + ); + assert!(remote_model_run(dir.path(), "run:sha256:bbbb") + .unwrap() + .is_none()); + } + + #[tokio::test] + async fn invoke_denies_requests_without_a_grant_or_with_a_foreign_capsule() { + let dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let source = iroh::SecretKey::from_bytes(&[73; 32]).public(); + let network = None; + let missing = invoke( + registry.clone(), + dir.path(), + network.clone(), + source, + &json!({ "operation": "offers_list", "request": { "op": "offers_list" } }), + ) + .await; + assert_eq!(missing["ok"], false); + assert_eq!(missing["code"], "denied"); + let foreign_capsule = invoke( + registry.clone(), + dir.path(), + network.clone(), + source, + &json!({ "operation": "offers_list", "request": { "op": "offers_list", + "remote_model": { "grant_id": "g", "principal_id": "p", "capsule_id": "browser" } } }), + ) + .await; + assert_eq!(foreign_capsule["code"], "denied"); + let wrong_op = invoke( + registry, + dir.path(), + network, + source, + &json!({ "operation": "runs_delete", "request": { "op": "runs_delete" } }), + ) + .await; + assert_eq!(wrong_op["code"], "invalid_provider_invocation"); + assert!(!run_index_path(dir.path()).exists()); + } +} diff --git a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs index 5d7ab0a50..d45288e8b 100644 --- a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs +++ b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs @@ -3841,7 +3841,7 @@ pub(super) fn provider_proxy_runtime_metadata_field(request: &serde_json::Value) .map(String::as_str) } -fn normalize_model_provider_request( +pub(super) fn normalize_model_provider_request( op: &str, request: &serde_json::Value, context: &HomeLaunchTokenContext, diff --git a/elastos/crates/elastos-server/src/carrier.rs b/elastos/crates/elastos-server/src/carrier.rs index 0ee5e1284..515d38487 100644 --- a/elastos/crates/elastos-server/src/carrier.rs +++ b/elastos/crates/elastos-server/src/carrier.rs @@ -1407,7 +1407,17 @@ async fn handle_file_stream( .await?; return Ok(()); }; - let response = if msg.data["target"] == "browser-engine" { + let response = if msg.data["target"] == "model" { + let network = gossip_state.lock().await.browser_exit_network.clone(); + crate::api::gateway::invoke_remote_model( + registry.clone(), + data_dir, + network, + source_endpoint_id, + &msg.data, + ) + .await + } else if msg.data["target"] == "browser-engine" { let (network, slots, endpoint) = { let state = gossip_state.lock().await; ( @@ -2000,6 +2010,7 @@ fn carrier_provider_target_allowed(target: &str) -> bool { matches!( target, "content" + | "model" | "availability" | "custody" | "rights" diff --git a/elastos/crates/elastos-server/src/provider_resource.rs b/elastos/crates/elastos-server/src/provider_resource.rs index 5909b6841..ac64afe79 100644 --- a/elastos/crates/elastos-server/src/provider_resource.rs +++ b/elastos/crates/elastos-server/src/provider_resource.rs @@ -150,7 +150,9 @@ pub fn provider_operation_action(scheme: &str, op: &str) -> Option { _ => None, }, "browser-engine" => match op { - "status" | "readiness" | "page_status" | "diagnostics" | "inspect" => Some(Action::Read), + "status" | "readiness" | "page_status" | "diagnostics" | "inspect" => { + Some(Action::Read) + } "launch" | "attach_stream" | "input" | "webrtc_signal" => Some(Action::Write), "close_page" => Some(Action::Delete), _ => None, diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index 412a68abc..f71a6d63f 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -1550,6 +1550,9 @@ const carrierBridge = read( "elastos/crates/elastos-server/src/resource_bridge.rs", ); const carrierRuntime = read("elastos/crates/elastos-server/src/carrier.rs"); +const modelServiceRuntime = read( + "elastos/crates/elastos-server/src/api/gateway_model_service.rs", +); const runtimeCore = read("elastos/crates/elastos-server/src/runtime.rs"); const runtimeControl = read( "elastos/crates/elastos-server/src/runtime_control.rs", @@ -6828,9 +6831,18 @@ assert( carrierRuntime.includes('"transfer": "stream"') && carrierRuntime.includes("ProviderTransfer::Stream") && carrierRuntime.includes('"carrier_provider_invoke"') && - /"content"\s*\|\s*"availability"\s*\|\s*"custody"\s*\|\s*"rights"\s*\|\s*"key"\s*\|\s*"decrypt"\s*\|\s*"drm"\s*\|\s*"collaboration"\s*\|\s*"collaboration-direct"\s*\|\s*"collaboration-profile"/.test( + /"content"\s*\|\s*"model"\s*\|\s*"availability"\s*\|\s*"custody"\s*\|\s*"rights"\s*\|\s*"key"\s*\|\s*"decrypt"\s*\|\s*"drm"\s*\|\s*"collaboration"\s*\|\s*"collaboration-direct"\s*\|\s*"collaboration-profile"/.test( carrierRuntime, ) && + // The model target is admitted only through the destination-owned grant + // check; the generic registry path must never serve it. + carrierRuntime.includes('if msg.data["target"] == "model" {') && + carrierRuntime.includes("crate::api::gateway::invoke_remote_model(") && + modelServiceRuntime.includes("authorize_home_service_model(") && + modelServiceRuntime.includes("fn remote_principal_id(") && + modelServiceRuntime.includes('local_object.remove("_runtime_invocation");') && + modelServiceRuntime.includes("fn shareable_offers(") && + modelServiceRuntime.includes("fn redact_provider_error(") && carrierRuntime.includes("CarrierProviderInvoker") && carrierRuntime.includes("ProviderCarrierInvoker for CarrierProviderInvoker") && carrierRuntime.includes("with_carrier_endpoint_and_registry") && From d361c2678b04806f1a8a4a32ef01ff6bb550ed78 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 16:21:53 -0500 Subject: [PATCH 305/535] feat(model): route granted remote offers and runs through Carrier A principal with an approved `remote_model` grant sees the contact's shared offers in `offers_list` under `remote::` ids with a `remote_service` block, beside the local offers. `runs_create` on such an offer, and `runs_get`, `runs_events` and `runs_cancel` for a run this Runtime routed before, travel over Carrier `provider_invoke` to the granting Runtime with the typed fields plus the grant, principal and consumer capsule. The seed keeps a principal-scoped run route index so canonical run ids stay unchanged for the Assistant. A destination decision (`denied`, `rate_limited`, `offer_unavailable`, `provider_failure`) is returned as a provider error. A failed or timed-out Carrier call is returned as `transport_interrupted` with HTTP 503 and leaves the run state unknown here; only the granting Runtime settles a run. Unreachable grants contribute a `remote_services` status entry and no offers. Verification: cargo check --locked -p elastos-server --all-targets cargo fmt --all -- --check cargo test -p elastos-server --lib -- gateway_model_remote gateway_model_service gateway_tests::model node scripts/home-entropy-check.mjs git diff --check --- .../crates/elastos-server/src/api/gateway.rs | 2 + .../src/api/gateway_home_system.rs | 46 ++ .../src/api/gateway_model_remote.rs | 607 ++++++++++++++++++ .../src/api/gateway_provider_proxy.rs | 60 ++ 4 files changed, 715 insertions(+) create mode 100644 elastos/crates/elastos-server/src/api/gateway_model_remote.rs diff --git a/elastos/crates/elastos-server/src/api/gateway.rs b/elastos/crates/elastos-server/src/api/gateway.rs index 4c03dc7d7..6411260f2 100644 --- a/elastos/crates/elastos-server/src/api/gateway.rs +++ b/elastos/crates/elastos-server/src/api/gateway.rs @@ -76,6 +76,8 @@ mod gateway_inbox; mod gateway_inspect_actions; #[path = "gateway_marketplace.rs"] mod gateway_marketplace; +#[path = "gateway_model_remote.rs"] +mod gateway_model_remote; #[path = "gateway_model_service.rs"] mod gateway_model_service; pub(crate) use gateway_model_service::{ diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index 98767d4a4..d25d8b157 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -3492,6 +3492,52 @@ fn home_services_remote_model_grant( })) } +/// Approved remote model grants this principal may route model operations +/// through. The Runtime that issued a grant still decides every operation. +pub(in crate::api::gateway) fn home_services_remote_model_grants( + data_dir: &std::path::Path, + context: &HomeLaunchTokenContext, + discovery_service: Option< + &crate::collaboration_discovery_runtime::CollaborationDiscoveryService, + >, +) -> anyhow::Result> { + let state = home_services_selection_state(data_dir, context)?; + let contacts = home_services_peer_contacts_state(data_dir, context, discovery_service)?; + let mut grants = Vec::new(); + for record in state.remote_offer_requests.values() { + if record.status == "approved" + && record.service_kind == super::MODEL_SERVICE_KIND + && state.remote_offer_ids.contains(&record.offer_id) + && contacts + .contacts + .values() + .any(|contact| contact.peer_id == record.target_peer_id) + && record + .pending_access_decision + .as_ref() + .is_none_or(|pending| pending.decision == "approved") + { + if let Some(grant) = &record.remote_model_grant { + anyhow::ensure!( + grant["principal_id"].as_str() == Some(&context.principal_id), + "model grant owner changed" + ); + if grant["expires_at"] + .as_u64() + .is_some_and(|expiry| expiry > now_ts()) + { + grants.push(grant.clone()); + } + } + } + } + anyhow::ensure!( + grants.len() <= 4, + "Too many approved model services for a bounded observation" + ); + Ok(grants) +} + /// Destination authority for a remote model operation. The grant must have /// been issued by this Runtime, stay approved and shared, and the requester /// must still be an accepted contact whose device key matches the Carrier diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs new file mode 100644 index 000000000..84f1bf1e1 --- /dev/null +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -0,0 +1,607 @@ +//! Remote model service: the consumer side of a `remote_model` grant. +//! +//! The Assistant keeps using the local typed model contract. When a principal +//! holds an approved grant from a contact, that contact's shared offers appear +//! beside the local ones under a `remote:` prefixed id. A run created on such +//! an offer is executed by the granting Runtime; this Runtime only routes the +//! typed operations over Carrier and remembers which grant each run belongs +//! to. A failed Carrier call is reported as `transport_interrupted`, never as +//! a run outcome. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use anyhow::Context as _; +use elastos_runtime::provider::{ + ProviderCarrierRoute, ProviderInvocation, ProviderInvocationTransport, ProviderRegistry, + ProviderTransfer, +}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; + +use super::gateway_home_token::HomeLaunchTokenContext; + +const REMOTE_OFFER_PREFIX: &str = "remote:"; +const REMOTE_RUN_INDEX_SCHEMA: &str = "elastos.services.model-remote-runs/v1"; +const REMOTE_RUN_INDEX_MAX: usize = 1024; +const REMOTE_RUN_RETENTION_SECS: u64 = 7 * 24 * 3600; +const CARRIER_TIMEOUT_MS: u64 = 5_000; +pub(crate) const TRANSPORT_INTERRUPTED: &str = "transport_interrupted"; + +pub(crate) fn remote_offer_id(grant_id: &str, offer_id: &str) -> String { + format!("{REMOTE_OFFER_PREFIX}{grant_id}:{offer_id}") +} + +/// `remote::`; grant ids never contain `:`. +pub(crate) fn parse_remote_offer_id(value: &str) -> Option<(&str, &str)> { + let rest = value.strip_prefix(REMOTE_OFFER_PREFIX)?; + let (grant_id, offer_id) = rest.split_once(':')?; + (!grant_id.is_empty() && !offer_id.is_empty()).then_some((grant_id, offer_id)) +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct RemoteRunRoute { + pub principal_id: String, + pub grant_id: String, + pub peer_did: String, + pub offer_id: String, + pub request_id: String, + pub created_at: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub terminal_at: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +struct RemoteRunIndex { + schema: String, + #[serde(default)] + runs: BTreeMap, +} + +fn index_path(data_dir: &Path) -> PathBuf { + data_dir.join("services-model-remote-runs.json") +} + +fn index_lock() -> &'static std::sync::Mutex<()> { + static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + &LOCK +} + +fn read_index(data_dir: &Path) -> anyhow::Result { + match std::fs::read(index_path(data_dir)) { + Ok(bytes) => { + let index: RemoteRunIndex = + serde_json::from_slice(&bytes).context("remote run index parse")?; + anyhow::ensure!( + index.schema == REMOTE_RUN_INDEX_SCHEMA, + "remote run index schema mismatch" + ); + Ok(index) + } + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(RemoteRunIndex { + schema: REMOTE_RUN_INDEX_SCHEMA.to_string(), + runs: BTreeMap::new(), + }), + Err(err) => Err(err).context("remote run index read"), + } +} + +fn update_index( + data_dir: &Path, + now: u64, + update: impl FnOnce(&mut RemoteRunIndex) -> anyhow::Result, +) -> anyhow::Result { + let _guard = index_lock() + .lock() + .map_err(|_| anyhow::anyhow!("remote run index unavailable"))?; + let mut index = read_index(data_dir)?; + index.runs.retain(|_, route| { + route + .terminal_at + .is_none_or(|terminal_at| now.saturating_sub(terminal_at) <= REMOTE_RUN_RETENTION_SECS) + }); + let result = update(&mut index)?; + anyhow::ensure!( + index.runs.len() <= REMOTE_RUN_INDEX_MAX, + "remote run index is full" + ); + let path = index_path(data_dir); + let temp = path.with_extension("json.tmp"); + std::fs::write(&temp, serde_json::to_vec_pretty(&index)?).context("remote run index write")?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(&temp, std::fs::Permissions::from_mode(0o600))?; + } + std::fs::rename(&temp, &path).context("remote run index commit")?; + Ok(result) +} + +pub(crate) fn remote_run_route( + data_dir: &Path, + principal_id: &str, + run_id: &str, +) -> anyhow::Result> { + let _guard = index_lock() + .lock() + .map_err(|_| anyhow::anyhow!("remote run index unavailable"))?; + Ok(read_index(data_dir)? + .runs + .get(run_id) + .filter(|route| route.principal_id == principal_id) + .cloned()) +} + +/// One approved grant, as stored on the consumer request record. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ConsumerModelGrant { + pub grant_id: String, + pub peer_did: String, + pub connect_ticket: String, + pub expires_at: u64, +} + +impl ConsumerModelGrant { + fn from_record(grant: &Value) -> Option { + Some(Self { + grant_id: grant["grant_id"].as_str()?.to_string(), + peer_did: grant["peer_did"].as_str()?.to_string(), + connect_ticket: grant["connect_ticket"].as_str()?.to_string(), + expires_at: grant["expires_at"].as_u64()?, + }) + } +} + +pub(crate) fn consumer_grants( + data_dir: &Path, + context: &HomeLaunchTokenContext, + discovery_service: Option< + &crate::collaboration_discovery_runtime::CollaborationDiscoveryService, + >, +) -> anyhow::Result> { + Ok( + super::gateway_home_system::home_services_remote_model_grants( + data_dir, + context, + discovery_service, + )? + .iter() + .filter_map(ConsumerModelGrant::from_record) + .collect(), + ) +} + +#[derive(Debug)] +pub(crate) enum RemoteRouteError { + /// The granting Runtime answered with a bounded class such as `denied`. + Rejected { code: String }, + /// Carrier could not complete the call; the run state is unknown here. + Transport(String), + /// The request or local index is invalid. + Invalid(String), +} + +impl std::fmt::Display for RemoteRouteError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Rejected { code } => { + write!(f, "remote model service rejected the operation: {code}") + } + Self::Transport(message) => write!(f, "{TRANSPORT_INTERRUPTED}: {message}"), + Self::Invalid(message) => write!(f, "{message}"), + } + } +} + +fn typed_request( + normalized: &Value, + context: &HomeLaunchTokenContext, + capsule_id: &str, + grant_id: &str, +) -> Value { + let mut request = normalized.clone(); + if let Some(object) = request.as_object_mut() { + object.remove("runtime_binding"); + object.insert( + "remote_model".to_string(), + json!({ + "grant_id": grant_id, + "principal_id": context.principal_id, + "capsule_id": capsule_id, + }), + ); + } + request +} + +async fn call_grant( + registry: &ProviderRegistry, + grant: &ConsumerModelGrant, + op: &str, + request: Value, +) -> Result { + let response = registry + .invoke_provider(ProviderInvocation { + source: "model-consumer".to_string(), + target: "model".to_string(), + op: op.to_string(), + request, + transfer: ProviderTransfer::Json, + range: None, + progress: None, + transport: ProviderInvocationTransport::Carrier(ProviderCarrierRoute::ConnectTicket { + connect_ticket: grant.connect_ticket.clone(), + peer_did: Some(grant.peer_did.clone()), + timeout_ms: Some(CARRIER_TIMEOUT_MS), + }), + }) + .await + .map_err(|err| classify_carrier_error(&err.to_string()))?; + Ok(response) +} + +/// The Carrier client surfaces a destination denial as the bounded class text +/// and a transport failure as anything else. Only the first is a decision. +fn classify_carrier_error(message: &str) -> RemoteRouteError { + for code in [ + "denied", + "rate_limited", + "offer_unavailable", + "provider_failure", + "invalid_provider_invocation", + "unauthorized_provider_target", + ] { + if message.contains(code) { + return RemoteRouteError::Rejected { + code: code.to_string(), + }; + } + } + RemoteRouteError::Transport(message.to_string()) +} + +fn run_id_of(result: &Value) -> Option { + result + .pointer("/data/run/id") + .or_else(|| result.pointer("/data/run_id")) + .or_else(|| result.pointer("/run/id")) + .or_else(|| result.get("run_id")) + .and_then(Value::as_str) + .map(str::to_string) +} + +fn run_is_terminal(result: &Value) -> bool { + let status = result + .pointer("/data/run/status") + .or_else(|| result.pointer("/data/status")) + .or_else(|| result.pointer("/run/status")) + .and_then(Value::as_str) + .unwrap_or_default(); + matches!( + status, + "completed" | "failed" | "cancelled" | "settlement_unknown" + ) +} + +fn rewrite_offer_ids(value: &mut Value, grant_id: &str) { + match value { + Value::Object(map) => { + if let Some(Value::String(offer_id)) = map.get_mut("offer_id") { + if parse_remote_offer_id(offer_id).is_none() { + *offer_id = remote_offer_id(grant_id, offer_id); + } + } + for child in map.values_mut() { + rewrite_offer_ids(child, grant_id); + } + } + Value::Array(items) => items + .iter_mut() + .for_each(|item| rewrite_offer_ids(item, grant_id)), + _ => {} + } +} + +/// Append the offers each approved grant currently shares. An unreachable +/// grant contributes a status entry and no offers. +pub(crate) async fn append_remote_offers( + registry: &ProviderRegistry, + grants: &[ConsumerModelGrant], + context: &HomeLaunchTokenContext, + capsule_id: &str, + response: &mut Value, +) { + let mut services = Vec::new(); + for grant in grants { + let request = typed_request( + &json!({ "op": "offers_list" }), + context, + capsule_id, + &grant.grant_id, + ); + match call_grant(registry, grant, "offers_list", request).await { + Ok(result) => { + let offers = result + .pointer("/data/offers") + .or_else(|| result.get("offers")) + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + let count = offers.len(); + for mut offer in offers { + if let Some(Value::String(id)) = offer.get_mut("id") { + *id = remote_offer_id(&grant.grant_id, id); + } + offer["remote_service"] = json!({ + "grant_id": grant.grant_id, + "peer_did": grant.peer_did, + "expires_at": grant.expires_at, + }); + if let Some(Value::Array(target)) = response.pointer_mut("/data/offers") { + target.push(offer); + } else if let Some(Value::Array(target)) = response.get_mut("offers") { + target.push(offer); + } + } + services.push( + json!({ "grant_id": grant.grant_id, "status": "reachable", "offers": count }), + ); + } + Err(RemoteRouteError::Rejected { code }) => services + .push(json!({ "grant_id": grant.grant_id, "status": "rejected", "code": code })), + Err(RemoteRouteError::Transport(_)) => services + .push(json!({ "grant_id": grant.grant_id, "status": TRANSPORT_INTERRUPTED })), + Err(RemoteRouteError::Invalid(message)) => services.push( + json!({ "grant_id": grant.grant_id, "status": "invalid", "message": message }), + ), + } + } + if !services.is_empty() { + if response.get("data").is_some_and(Value::is_object) { + response["data"]["remote_services"] = Value::Array(services); + } else { + response["remote_services"] = Value::Array(services); + } + } +} + +/// Route one run operation through the grant that owns it. `Ok(None)` means +/// the operation is local. +pub(crate) async fn route_run_operation( + registry: Arc, + data_dir: &Path, + grants: &[ConsumerModelGrant], + context: &HomeLaunchTokenContext, + capsule_id: &str, + op: &str, + normalized: &Value, + now: u64, +) -> Result, RemoteRouteError> { + match op { + "runs_create" => { + let offer_id = normalized["offer_id"].as_str().unwrap_or_default(); + let Some((grant_id, inner_offer_id)) = parse_remote_offer_id(offer_id) else { + return Ok(None); + }; + let grant = grants + .iter() + .find(|grant| grant.grant_id == grant_id) + .ok_or_else(|| RemoteRouteError::Rejected { + code: "denied".to_string(), + })?; + let mut request = typed_request(normalized, context, capsule_id, grant_id); + request["offer_id"] = Value::String(inner_offer_id.to_string()); + let mut result = call_grant(®istry, grant, op, request).await?; + if let Some(run_id) = run_id_of(&result) { + let route = RemoteRunRoute { + principal_id: context.principal_id.clone(), + grant_id: grant.grant_id.clone(), + peer_did: grant.peer_did.clone(), + offer_id: offer_id.to_string(), + request_id: normalized + .pointer("/runtime_binding/request_id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(), + created_at: now, + terminal_at: run_is_terminal(&result).then_some(now), + }; + update_index(data_dir, now, |index| { + index.runs.entry(run_id).or_insert(route); + Ok(()) + }) + .map_err(|err| RemoteRouteError::Invalid(err.to_string()))?; + } + rewrite_offer_ids(&mut result, grant_id); + Ok(Some(result)) + } + "runs_get" | "runs_events" | "runs_cancel" => { + let run_id = normalized["run_id"].as_str().unwrap_or_default(); + let Some(route) = remote_run_route(data_dir, &context.principal_id, run_id) + .map_err(|err| RemoteRouteError::Invalid(err.to_string()))? + else { + return Ok(None); + }; + let grant = grants + .iter() + .find(|grant| grant.grant_id == route.grant_id) + .ok_or_else(|| RemoteRouteError::Rejected { + code: "denied".to_string(), + })?; + let request = typed_request(normalized, context, capsule_id, &route.grant_id); + let mut result = call_grant(®istry, grant, op, request).await?; + if route.terminal_at.is_none() && run_is_terminal(&result) { + let run_id = run_id.to_string(); + let _ = update_index(data_dir, now, |index| { + if let Some(entry) = index.runs.get_mut(&run_id) { + entry.terminal_at = Some(now); + } + Ok(()) + }); + } + rewrite_offer_ids(&mut result, &route.grant_id); + Ok(Some(result)) + } + _ => Ok(None), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn remote_offer_ids_round_trip_and_reject_malformed_values() { + let id = remote_offer_id("services-remote-model-grant-0011223344556677", "qwen3.5-9b"); + assert_eq!( + parse_remote_offer_id(&id), + Some(("services-remote-model-grant-0011223344556677", "qwen3.5-9b")) + ); + assert_eq!(parse_remote_offer_id("qwen3.5-9b"), None); + assert_eq!(parse_remote_offer_id("remote:"), None); + assert_eq!(parse_remote_offer_id("remote:grant-only"), None); + assert_eq!(parse_remote_offer_id("remote::offer"), None); + } + + #[test] + fn typed_request_replaces_the_local_binding_with_the_remote_fields() { + let context = HomeLaunchTokenContext { + principal_id: "seed-principal".into(), + proof_binding_id: None, + session_id: "s".into(), + grant_id: "launch-grant".into(), + }; + let normalized = json!({ + "op": "runs_create", "offer_id": "remote:g:qwen", "operation": "chat", "input": {"x": 1}, + "runtime_binding": { "principal_id": "seed-principal", "grant_id": "launch-grant" } + }); + let request = typed_request(&normalized, &context, "assistant", "g"); + assert!(request.get("runtime_binding").is_none()); + assert_eq!(request["remote_model"]["grant_id"], "g"); + assert_eq!(request["remote_model"]["principal_id"], "seed-principal"); + assert_eq!(request["remote_model"]["capsule_id"], "assistant"); + } + + #[test] + fn carrier_errors_split_into_decisions_and_transport_interruptions() { + assert!( + matches!(classify_carrier_error("model grant is not active for this requester (denied)"), RemoteRouteError::Rejected { code } if code == "denied") + ); + assert!( + matches!(classify_carrier_error("rate_limited"), RemoteRouteError::Rejected { code } if code == "rate_limited") + ); + assert!(matches!( + classify_carrier_error("connection timed out"), + RemoteRouteError::Transport(_) + )); + assert!(matches!( + classify_carrier_error( + "Carrier provider invocation requires registered Carrier invoker" + ), + RemoteRouteError::Transport(_) + )); + } + + #[test] + fn offer_ids_in_results_are_rewritten_once() { + let mut result = json!({ "data": { "run": { "id": "run:sha256:aa", "offer_id": "qwen" }, "offer_id": "remote:g:qwen" } }); + rewrite_offer_ids(&mut result, "g"); + assert_eq!(result["data"]["run"]["offer_id"], "remote:g:qwen"); + assert_eq!(result["data"]["offer_id"], "remote:g:qwen"); + } + + #[test] + fn remote_run_index_is_principal_scoped_and_pruned() { + let dir = tempfile::tempdir().unwrap(); + let route = RemoteRunRoute { + principal_id: "p1".into(), + grant_id: "g".into(), + peer_did: "did:key:z6Mkpeer".into(), + offer_id: "remote:g:qwen".into(), + request_id: "req".into(), + created_at: 10, + terminal_at: None, + }; + update_index(dir.path(), 10, |index| { + index.runs.insert("run:sha256:aa".into(), route.clone()); + index.runs.insert( + "run:sha256:bb".into(), + RemoteRunRoute { + terminal_at: Some(5), + ..route.clone() + }, + ); + Ok(()) + }) + .unwrap(); + assert_eq!( + remote_run_route(dir.path(), "p1", "run:sha256:aa").unwrap(), + Some(route) + ); + assert_eq!( + remote_run_route(dir.path(), "p2", "run:sha256:aa").unwrap(), + None + ); + update_index(dir.path(), 5 + REMOTE_RUN_RETENTION_SECS + 1, |_| Ok(())).unwrap(); + assert_eq!( + remote_run_route(dir.path(), "p1", "run:sha256:bb").unwrap(), + None + ); + } + + #[tokio::test] + async fn local_offers_and_unknown_runs_stay_local() { + let dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let context = HomeLaunchTokenContext { + principal_id: "p1".into(), + proof_binding_id: None, + session_id: "s".into(), + grant_id: "g".into(), + }; + let local_create = json!({ "op": "runs_create", "offer_id": "qwen", "operation": "chat", "input": {}, "runtime_binding": {} }); + assert!(route_run_operation( + registry.clone(), + dir.path(), + &[], + &context, + "assistant", + "runs_create", + &local_create, + 1 + ) + .await + .unwrap() + .is_none()); + let local_get = + json!({ "op": "runs_get", "run_id": "run:sha256:unknown", "runtime_binding": {} }); + assert!(route_run_operation( + registry.clone(), + dir.path(), + &[], + &context, + "assistant", + "runs_get", + &local_get, + 1 + ) + .await + .unwrap() + .is_none()); + let remote_without_grant = json!({ "op": "runs_create", "offer_id": "remote:g:qwen", "operation": "chat", "input": {}, "runtime_binding": {} }); + let err = route_run_operation( + registry, + dir.path(), + &[], + &context, + "assistant", + "runs_create", + &remote_without_grant, + 1, + ) + .await + .unwrap_err(); + assert!(matches!(err, RemoteRouteError::Rejected { code } if code == "denied")); + } +} diff --git a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs index d45288e8b..7a9654c46 100644 --- a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs +++ b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs @@ -1753,12 +1753,62 @@ pub(super) async fn gateway_provider_proxy( ) .into_response(); } + let mut remote_model_grants = Vec::new(); if scheme == "model" { request = match normalize_model_provider_request(&op, &request, &context, &launch_capsule_id) { Ok(value) => value, Err((status, message)) => return (status, message).into_response(), }; + // Approved remote model grants route runs to the granting Runtime. A + // grant lookup failure keeps the local path; it cannot widen authority. + remote_model_grants = super::gateway_model_remote::consumer_grants( + &state.data_dir, + &context, + state.collaboration_discovery_service.as_ref(), + ) + .unwrap_or_default(); + if !remote_model_grants.is_empty() && op != "offers_list" { + match super::gateway_model_remote::route_run_operation( + Arc::clone(®istry), + &state.data_dir, + &remote_model_grants, + &context, + &launch_capsule_id, + &op, + &request, + crate::auth::now_ts(), + ) + .await + { + Ok(Some(mut remote_response)) => { + if let Err(err) = project_model_provider_response(&op, &mut remote_response) { + return gateway_provider_error_response( + &scheme, + anyhow::anyhow!( + "remote model service returned an invalid typed response: {err}" + ), + ); + } + return (StatusCode::OK, Json(remote_response)).into_response(); + } + Ok(None) => {} + Err(super::gateway_model_remote::RemoteRouteError::Transport(message)) => { + return ( + StatusCode::SERVICE_UNAVAILABLE, + Json(serde_json::json!({ + "status": "error", + "code": super::gateway_model_remote::TRANSPORT_INTERRUPTED, + "message": format!("remote model service unreachable: {message}"), + })), + ) + .into_response(); + } + Err(err) => { + return gateway_provider_error_response(&scheme, anyhow::anyhow!("{err}")); + } + } + } } else { request["op"] = serde_json::Value::String(op.clone()); } @@ -1957,6 +2007,16 @@ pub(super) async fn gateway_provider_proxy( } }; if scheme == "model" { + if op == "offers_list" && !remote_model_grants.is_empty() { + super::gateway_model_remote::append_remote_offers( + ®istry, + &remote_model_grants, + &context, + &launch_capsule_id, + &mut response, + ) + .await; + } if let Err(err) = project_model_provider_response(&op, &mut response) { return gateway_provider_error_response( &scheme, From 3389d768b2f7f3df3ca6adbebdc48d1b9970825a Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 16:49:45 -0500 Subject: [PATCH 306/535] feat(services): show the AI model service beside Engine and Exit Services renders the `remote_model` kind on the Mine and People lists. Copy and ordering come from one table per kind instead of a branch per kind, so the Assistant is named as the consumer for model cards and Browser for the others. Empty-state copy names all three kinds. Verification: node --check capsules/services/browser/services.js node scripts/home-entropy-check.mjs cargo test -p elastos-server --lib -- services people_contact (28) --- capsules/services/browser/services.js | 105 +++++++++++++------------- 1 file changed, 54 insertions(+), 51 deletions(-) diff --git a/capsules/services/browser/services.js b/capsules/services/browser/services.js index c75964564..36ffc4bbf 100644 --- a/capsules/services/browser/services.js +++ b/capsules/services/browser/services.js @@ -11,8 +11,45 @@ const homeToken = new URLSearchParams(window.location.hash.replace(/^#/, "")).ge const homeOrigin = launchParams.get("home_origin") || ""; const EXIT_SERVICE_KIND = "remote_exit"; const BROWSER_ENGINE_SERVICE_KIND = "browser_engine"; +const MODEL_SERVICE_KIND = "remote_model"; const CONFIGURED_REMOTE_EXIT_SOURCE = "configured_remote_exit"; -const VISIBLE_SERVICE_KINDS = new Set([BROWSER_ENGINE_SERVICE_KIND, EXIT_SERVICE_KIND]); +const VISIBLE_SERVICE_KINDS = new Set([BROWSER_ENGINE_SERVICE_KIND, MODEL_SERVICE_KIND, EXIT_SERVICE_KIND]); +const SERVICE_KIND_COPY = { + [BROWSER_ENGINE_SERVICE_KIND]: { + rank: 0, + noun: "Browser Engine", + mineTitle: { shared: "My Browser Engine is shared", private: "Share my Browser Engine" }, + mineCopy: { + shared: "People you trust can use this Browser Engine after you approve access. Low-level connection details stay hidden.", + private: "Make this Browser Engine available to People you trust. Access stays under your approval.", + }, + consumer: "Browser", + otherFallback: "this person's Browser Engine", + }, + [MODEL_SERVICE_KIND]: { + rank: 1, + noun: "AI model", + mineTitle: { shared: "My AI model is shared", private: "Share my AI model" }, + mineCopy: { + shared: "People you trust can run your local AI model after you approve each request. The model and every run stay on this device; hosted models stay private.", + private: "Let People you trust run your local AI model. Access stays under your approval and you can revoke it.", + }, + consumer: "Assistant", + otherFallback: "this person's AI model", + }, + [EXIT_SERVICE_KIND]: { + rank: 2, + noun: "Browser Exit service", + mineTitle: { shared: "My Browser Exit service is shared", private: "Share my Browser Exit service" }, + mineCopy: { + shared: "People you trust can use this device's Browser Exit service after approval. Low-level network details stay hidden.", + private: "Let People you trust use this device's Browser Exit service. Access stays under your approval.", + }, + consumer: "Browser", + otherFallback: "this person's Browser Exit service", + }, +}; +const SERVICE_KIND_LIST_COPY = "Browser Engine, AI model or Browser Exit"; let currentServices = null; let pendingServiceAction = null; @@ -119,7 +156,7 @@ function renderServices(services) { selectedTitle: "Shared", availableTitle: "Available on this device", emptySelected: "No Services are shared.", - emptyAvailable: "No Browser Engine or Browser Exit service is installed on this device.", + emptyAvailable: `No ${SERVICE_KIND_LIST_COPY} service is installed on this device.`, }); otherServicesList.innerHTML = renderServiceSection({ selected: remoteOffers, @@ -128,7 +165,7 @@ function renderServices(services) { selectedTitle: "Subscribed", availableTitle: "Available from People", emptySelected: "No Services from others are subscribed.", - emptyAvailable: "No Browser Engine or Browser Exit services are available from People you are connected with.", + emptyAvailable: `No ${SERVICE_KIND_LIST_COPY} services are available from People you are connected with.`, }); } @@ -290,61 +327,34 @@ function visibleServiceOffers(offers) { : []; } +function serviceKindCopy(offer) { + return SERVICE_KIND_COPY[readText(offer?.service_kind)] || SERVICE_KIND_COPY[EXIT_SERVICE_KIND]; +} + function serviceTitle(offer, source, selected) { - const kind = readText(offer?.service_kind); - if (kind === BROWSER_ENGINE_SERVICE_KIND) { - if (source === "mine") { - return selected ? "My Browser Engine is shared" : "Share my Browser Engine"; - } - return readText(offer?.display_name) || "Browser Engine"; - } + const copy = serviceKindCopy(offer); if (source === "mine") { - return selected ? "My Browser Exit service is shared" : "Share my Browser Exit service"; + return selected ? copy.mineTitle.shared : copy.mineTitle.private; } - return readText(offer?.display_name) || "External Browser Exit service"; + return readText(offer?.display_name) || (copy === SERVICE_KIND_COPY[EXIT_SERVICE_KIND] ? "External Browser Exit service" : copy.noun); } function serviceCopy(offer, source, selected) { - const kind = readText(offer?.service_kind); - if (kind === BROWSER_ENGINE_SERVICE_KIND) { - if (source === "mine") { - return selected - ? "People you trust can use this Browser Engine after you approve access. Low-level connection details stay hidden." - : "Make this Browser Engine available to People you trust. Access stays under your approval."; - } - const name = readText(offer?.display_name) || "this person's Browser Engine"; - if (offer?.grant_required === true) { - const requestStatus = serviceRequestStatus(offer); - if (selected && requestStatus === "approved") { - return `${name} was approved. Browser can use it when access becomes active.`; - } - if (selected && requestStatus === "denied") { - return `${name} denied the request. Remove it and ask again if needed.`; - } - return selected - ? `${name} is waiting for approval.` - : `Ask to use ${name}. You need to be connected in People first.`; - } - return selected - ? `${name} is saved as a Browser Engine option. Browser can use it when the service connection is active.` - : `Subscribe to ${name}. You need to be connected in People first.`; - } - const name = readText(offer?.display_name) || "this person's Browser Exit service"; + const copy = serviceKindCopy(offer); if (source === "mine") { - return selected - ? "People you trust can use this device's Browser Exit service after approval. Low-level network details stay hidden." - : "Let People you trust use this device's Browser Exit service. Access stays under your approval."; + return selected ? copy.mineCopy.shared : copy.mineCopy.private; } + const name = readText(offer?.display_name) || copy.otherFallback; if (readText(offer?.source) === CONFIGURED_REMOTE_EXIT_SOURCE) { return `${name} is available as a Browser Exit option on this device.`; } if (readText(offer?.status) === "active" && offer?.enabled === true) { - return `${name} is active and ready for Browser.`; + return `${name} is active and ready for ${copy.consumer}.`; } if (offer?.grant_required === true) { const requestStatus = serviceRequestStatus(offer); if (selected && requestStatus === "approved") { - return `${name} was approved. Browser can use it when access becomes active.`; + return `${name} was approved. ${copy.consumer} can use it when access becomes active.`; } if (selected && requestStatus === "denied") { return `${name} denied the request. Remove it and ask again if needed.`; @@ -354,7 +364,7 @@ function serviceCopy(offer, source, selected) { : `Ask to use ${name}. You need to be connected in People first.`; } return selected - ? `${name} is saved as a Browser Exit option. Browser can use it when the service connection is active.` + ? `${name} is saved as a ${copy.noun} option. ${copy.consumer} can use it when the service connection is active.` : `Subscribe to ${name}. You need to be connected in People first.`; } @@ -409,14 +419,7 @@ function orderedServiceOffers(offers) { } function serviceKindRank(kind) { - switch (kind) { - case BROWSER_ENGINE_SERVICE_KIND: - return 0; - case EXIT_SERVICE_KIND: - return 1; - default: - return 10; - } + return SERVICE_KIND_COPY[kind]?.rank ?? 10; } function serviceStatusTone(offer, source) { From 09b3128ac40b0ac0eb50f02ab3c3434a72f4e45d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 16:51:48 -0500 Subject: [PATCH 307/535] feat(assistant): name the contact behind a remote model offer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The consumer grant keeps the service display name from the request record; offers routed through it carry `remote_service.display_name`, and the Assistant's model menu shows "Model offer · via " instead of "this Home" for those rows. Verification: cargo check --locked -p elastos-server cargo fmt --all -- --check node --check capsules/assistant/browser/model-contract.js --- capsules/assistant/browser/model-contract.js | 5 ++++- .../crates/elastos-server/src/api/gateway_home_system.rs | 1 + .../crates/elastos-server/src/api/gateway_model_remote.rs | 6 ++++++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/capsules/assistant/browser/model-contract.js b/capsules/assistant/browser/model-contract.js index 7af274af2..92a8fe2a6 100644 --- a/capsules/assistant/browser/model-contract.js +++ b/capsules/assistant/browser/model-contract.js @@ -39,7 +39,10 @@ export function textOfferRows(offers) { offerId: offer.id, operation: offer.operation, label: offer.title, - detail: "Model offer · this Home", + detail: + typeof offer.remote_service?.display_name === "string" && offer.remote_service.display_name.trim() !== "" + ? `Model offer · via ${offer.remote_service.display_name.trim()}` + : "Model offer · this Home", streamOutput: offer.stream_output === true, })); } diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index d25d8b157..a49b06e0c 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -3486,6 +3486,7 @@ fn home_services_remote_model_grant( "peer_did": record.target_peer_id, "connect_ticket": ticket, "principal_id": context.principal_id, + "service_display_name": record.service_display_name, "offer_scope": "local_engines", "operations": grant["operations"], "expires_at": expiry, diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 84f1bf1e1..ad26d5e48 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -139,6 +139,7 @@ pub(crate) struct ConsumerModelGrant { pub grant_id: String, pub peer_did: String, pub connect_ticket: String, + pub display_name: String, pub expires_at: u64, } @@ -148,6 +149,10 @@ impl ConsumerModelGrant { grant_id: grant["grant_id"].as_str()?.to_string(), peer_did: grant["peer_did"].as_str()?.to_string(), connect_ticket: grant["connect_ticket"].as_str()?.to_string(), + display_name: grant["service_display_name"] + .as_str() + .unwrap_or("a contact's AI model") + .to_string(), expires_at: grant["expires_at"].as_u64()?, }) } @@ -336,6 +341,7 @@ pub(crate) async fn append_remote_offers( offer["remote_service"] = json!({ "grant_id": grant.grant_id, "peer_did": grant.peer_did, + "display_name": grant.display_name, "expires_at": grant.expires_at, }); if let Some(Value::Array(target)) = response.pointer_mut("/data/offers") { From d53edf7bd4d49c7169ae40db6fe5d2c12c0dd3b0 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 17:04:18 -0500 Subject: [PATCH 308/535] test(model): prove the remote model service between two Runtimes over Carrier Two real Carrier nodes on one host: an owner with a fake `model` sub-provider, a passkey principal, a signed contact chain and an approved `remote_model` request record; a seed that speaks `provider_invoke` with its own device key. The tests prove that only local-engine offers cross Carrier, that a run is rebound to the deterministic remote principal with the grant as session and grant id and the seed request id preserved, that a hosted offer is `offer_unavailable`, that another seed principal is `denied` without reaching the provider, that reads stay allowed after the owner denies the request while `runs_create` is refused, and that a revoke sweep cancels the open run. The harness found three defects, fixed here: - the owning Runtime kept `op` in the rebuilt request, which the typed normalizer rejects for every operation; the same shape broke the revoke cancel sweep; - the consumer dropped `request_id` together with the local binding, so the destination's typed structs would have refused every call; - the Carrier invoker collapsed a destination decision to "invoke failed", so the consumer would have reported every denial as `transport_interrupted`. The invoker now passes a decision through as one fixed token from a bounded set; every other error stays redacted, as the existing test still asserts. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model_remote gateway_model_service test_carrier_provider_public_errors (17) cargo fmt --all -- --check node scripts/home-entropy-check.mjs git diff --check --- .../src/api/gateway_model_remote.rs | 15 +- .../src/api/gateway_model_service.rs | 5 +- .../src/api/gateway_tests/mod.rs | 1 + .../src/api/gateway_tests/remote_model.rs | 598 ++++++++++++++++++ elastos/crates/elastos-server/src/carrier.rs | 70 +- 5 files changed, 682 insertions(+), 7 deletions(-) create mode 100644 elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index ad26d5e48..7a4a24641 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -199,6 +199,10 @@ impl std::fmt::Display for RemoteRouteError { } } +/// The wire request for the granting Runtime: the typed fields the local +/// normalizer accepted, with the local binding replaced by the grant facts. +/// `request_id` lives inside the local binding after normalization, so it is +/// lifted back to the top level where the destination's typed structs expect it. fn typed_request( normalized: &Value, context: &HomeLaunchTokenContext, @@ -207,7 +211,13 @@ fn typed_request( ) -> Value { let mut request = normalized.clone(); if let Some(object) = request.as_object_mut() { - object.remove("runtime_binding"); + if let Some(request_id) = object + .remove("runtime_binding") + .and_then(|binding| binding.get("request_id").cloned()) + .filter(Value::is_string) + { + object.insert("request_id".to_string(), request_id); + } object.insert( "remote_model".to_string(), json!({ @@ -480,10 +490,11 @@ mod tests { }; let normalized = json!({ "op": "runs_create", "offer_id": "remote:g:qwen", "operation": "chat", "input": {"x": 1}, - "runtime_binding": { "principal_id": "seed-principal", "grant_id": "launch-grant" } + "runtime_binding": { "principal_id": "seed-principal", "grant_id": "launch-grant", "request_id": "req-7" } }); let request = typed_request(&normalized, &context, "assistant", "g"); assert!(request.get("runtime_binding").is_none()); + assert_eq!(request["request_id"], "req-7"); assert_eq!(request["remote_model"]["grant_id"], "g"); assert_eq!(request["remote_model"]["principal_id"], "seed-principal"); assert_eq!(request["remote_model"]["capsule_id"], "assistant"); diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index 297f8bb60..94a44c5d4 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -388,11 +388,13 @@ pub(crate) async fn invoke( }; let now = crate::auth::now_ts(); - // A fresh local request: only the fields the typed contract accepts. + // A fresh local request: only the fields the typed contract accepts. The + // normalizer takes the operation as a parameter and sets `op` itself. let mut local = request.clone(); let local_object = local.as_object_mut().expect("request is an object"); local_object.remove("remote_model"); local_object.remove("_runtime_invocation"); + local_object.remove("op"); let (context, indexed_run) = match operation { "offers_list" | "runs_create" => { @@ -576,7 +578,6 @@ pub(crate) async fn cancel_grant_runs( } let context = remote_context(&record.remote_principal_id, grant_id); let request = json!({ - "op": "runs_cancel", "run_id": run_id, "request_id": format!("revoke:{}", record.request_id), }); diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/mod.rs b/elastos/crates/elastos-server/src/api/gateway_tests/mod.rs index c051dc48d..0a7faf64f 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/mod.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/mod.rs @@ -860,6 +860,7 @@ mod model; mod recovery; #[cfg(unix)] mod remote_engine; +mod remote_model; mod room; mod site_publication; mod wallet; diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs new file mode 100644 index 000000000..1afcc54be --- /dev/null +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -0,0 +1,598 @@ +use super::home_system::{ + configured_discovery_network_profile_for_test, write_home_principal_object_json_for_authority, +}; +use super::*; +use crate::api::gateway::gateway_model_service::{ + cancel_grant_runs, model_grant_id, remote_principal_id, +}; +use crate::collaboration_contact_store::CollaborationContactStore; +use crate::collaboration_discovery::*; +use crate::collaboration_profile_authority::{ + signed_profile_document_for_test, VerifiedCollaborationProfileDocument, +}; +use elastos_common::collaboration_protocol::*; +use elastos_model_contract::{RuntimeAccessBinding, RuntimeCreateBinding}; +use elastos_runtime::signature::{generate_keypair, SigningKey}; +use tokio::io::{AsyncBufReadExt as _, BufReader}; + +const NETWORK: &str = "remote-model"; +const CARRIER_ALPN: &[u8] = b"elastos/carrier/1"; +const REQUEST_ID: &str = "request-model-1"; +const SEED_PRINCIPAL: &str = "seed-principal-a"; +const OTHER_SEED_PRINCIPAL: &str = "seed-principal-b"; + +fn keys(value: &Value) -> BTreeSet<&str> { + let object = value.as_object().unwrap(); + object.keys().map(String::as_str).collect() +} + +fn provider_error(err: impl std::fmt::Display) -> ProviderError { + ProviderError::Provider(err.to_string()) +} + +fn run_id_for(principal_id: &str, request_id: &str) -> String { + format!( + "run:sha256:{}", + hex::encode(Sha256::digest(format!("{principal_id}:{request_id}"))) + ) +} + +fn run_view(run_id: &str, status: &str) -> Value { + json!({ "status": "ok", "data": { + "schema": "elastos.model.run-view/v1", "run_id": run_id, "offer_id": "qwen-local", + "operation": "text.generate", "status": status, "sequence_cursor": 0 } }) +} + +fn events_page(run_id: &str, page: u32) -> Value { + let event = |sequence: u64, text: &str, terminal: bool| { + json!({ "schema": "elastos.model.run-event/v1", "sequence": sequence, "kind": "output", + "data": { "schema": "elastos.model.output.text/v1", "text": text }, "terminal": terminal }) + }; + let events = match page { + 1 => vec![event(1, "hel", false), event(2, "lo", false)], + _ => vec![event(3, "", true)], + }; + json!({ "status": "ok", "data": { + "schema": "elastos.model.run-events/v1", "run_id": run_id, + "status": if page == 1 { "running" } else { "completed" }, + "next_cursor": events.len(), "has_more": page == 1, "events": events } }) +} + +#[derive(Default)] +struct FakeModelProvider { + calls: TokioMutex>, + run_owners: std::sync::Mutex>, + event_pages: std::sync::Mutex>, +} + +#[async_trait::async_trait] +impl Provider for FakeModelProvider { + fn name(&self) -> &'static str { + "fake-model-provider" + } + fn schemes(&self) -> Vec<&'static str> { + vec!["model"] + } + async fn handle(&self, _: ResourceRequest) -> Result { + Err(ProviderError::Provider("raw only".into())) + } + async fn send_raw(&self, request: &Value) -> Result { + self.calls.lock().await.push(request.clone()); + let op = request["op"].as_str().unwrap_or_default(); + let typed_fields = match op { + "offers_list" => vec!["op"], + "runs_create" => vec!["op", "offer_id", "operation", "input", "runtime_binding"], + "runs_events" => vec!["op", "run_id", "after_sequence", "runtime_binding"], + _ => vec!["op", "run_id", "runtime_binding"], + }; + if keys(request) != typed_fields.into_iter().collect() { + return Ok(json!({ "status": "error", "message": "fields differ from the contract" })); + } + if op == "offers_list" { + return Ok(json!({ "status": "ok", "data": { "offers": [ + { "id": "qwen-local", "title": "Qwen", "operation": "text.generate", "hosted": null }, + { "id": "gpt-hosted", "title": "GPT", "operation": "text.generate", + "hosted": { "placement": "hosted", "backend_provider_label": "OpenAI" } }, + ] } })); + } + if op == "runs_create" { + let binding: RuntimeCreateBinding = + serde_json::from_value(request["runtime_binding"].clone()) + .map_err(provider_error)?; + let offer_id = request["offer_id"].as_str().unwrap_or_default(); + let operation = request["operation"].as_str().unwrap_or_default(); + binding + .validate(offer_id, operation, &request["input"]) + .map_err(provider_error)?; + let run_id = run_id_for(&binding.principal_id, &binding.request_id); + self.run_owners + .lock() + .unwrap() + .insert(run_id.clone(), binding.principal_id); + return Ok(run_view(&run_id, "running")); + } + let binding: RuntimeAccessBinding = + serde_json::from_value(request["runtime_binding"].clone()).map_err(provider_error)?; + let run_id = request["run_id"].as_str().unwrap_or_default(); + binding.validate(run_id).map_err(provider_error)?; + if self.run_owners.lock().unwrap().get(run_id) != Some(&binding.principal_id) { + return Ok(json!({ "status": "error", "message": "run owner mismatch" })); + } + Ok(match op { + "runs_get" => run_view(&binding.run_id, "running"), + "runs_cancel" => run_view(&binding.run_id, "cancelled"), + _ => { + let mut pages = self.event_pages.lock().unwrap(); + let page = pages.entry(binding.run_id.clone()).or_insert(0); + *page += 1; + events_page(&binding.run_id, *page) + } + }) + } +} + +fn signed_message( + key: &SigningKey, + sender_profile_did: &str, + conversation_id: &str, + recipient: CollaborationRecipient, + payload_type: &str, + payload: Value, + validity: std::ops::Range, +) -> Vec { + let message = CollaborationMessage { + schema: COLLABORATION_MESSAGE_SCHEMA_V1.to_string(), + network_id: NETWORK.to_string(), + conversation_id: conversation_id.to_string(), + message_id: crate::collaboration_core::random_hex_128().unwrap(), + nonce: crate::collaboration_core::random_hex_128().unwrap(), + created_at: validity.start, + expires_at: validity.end, + sender_profile_did: sender_profile_did.to_string(), + sender_service: COLLABORATION_DISCOVERY_SERVICE.to_string(), + recipient, + payload_type: payload_type.to_string(), + payload, + }; + let (signature, signer_did) = crate::crypto::domain_separated_sign( + key, + COLLABORATION_MESSAGE_SIGNATURE_DOMAIN_V1, + &canonical_collaboration_message_bytes(&message).unwrap(), + ); + canonical_signed_collaboration_message_bytes(&SignedCollaborationMessage { + payload: message, + signature, + signer_did, + }) + .unwrap() +} + +fn accept_seed_contact( + store: &CollaborationContactStore, + owner_key: &SigningKey, + owner_profile: &VerifiedCollaborationProfileDocument, + seed_key: &SigningKey, + seed_did: &str, +) { + let now = now_ts().saturating_sub(3); + let owner_profile_did = owner_profile.document().profile_did.clone(); + let (seed_profile_key, _) = generate_keypair(); + let seed_profile = signed_profile_document_for_test( + &seed_profile_key, + "Seed", + None, + 1, + None, + now, + vec![seed_did.to_string()], + ) + .unwrap(); + let advertisement = signed_message( + owner_key, + &owner_profile_did, + COLLABORATION_DISCOVERY_DIRECTORY_ID, + CollaborationRecipient { + kind: CollaborationRecipientKind::Conversation, + id: COLLABORATION_DISCOVERY_DIRECTORY_ID.to_string(), + }, + COLLABORATION_DISCOVERY_ADVERTISEMENT_PAYLOAD_TYPE, + serde_json::to_value(CollaborationDiscoveryAdvertisementPayload { + signed_profile: owner_profile.signed_envelope().clone(), + }) + .unwrap(), + now..now + COLLABORATION_DISCOVERY_ADVERTISEMENT_TTL_SECS, + ); + store + .store_local_advertisement(&advertisement, now) + .unwrap(); + let request = signed_message( + seed_key, + &seed_profile.document().profile_did, + COLLABORATION_DISCOVERY_CONTACT_ID, + CollaborationRecipient { + kind: CollaborationRecipientKind::Profile, + id: owner_profile_did.clone(), + }, + COLLABORATION_DISCOVERY_CONTACT_REQUEST_PAYLOAD_TYPE, + serde_json::to_value(CollaborationContactRequestPayload { + advertisement_envelope_sha256: collaboration_message_envelope_sha256(&advertisement), + signed_profile: seed_profile.signed_envelope().clone(), + }) + .unwrap(), + now + 1..now + 1 + COLLABORATION_DISCOVERY_CONTACT_REQUEST_TTL_SECS, + ); + store + .record_incoming_contact_request(&request, now + 1) + .unwrap(); + let envelope: SignedCollaborationMessage = serde_json::from_slice(&request).unwrap(); + let payload = CollaborationContactDecisionReceipt { + schema: COLLABORATION_CONTACT_DECISION_RECEIPT_SCHEMA_V1.to_string(), + network_id: NETWORK.to_string(), + request_envelope_sha256: collaboration_message_envelope_sha256(&request), + conversation_id: COLLABORATION_DISCOVERY_CONTACT_ID.to_string(), + requester_profile_did: seed_profile.document().profile_did.clone(), + requester_endpoint_did: envelope.signer_did, + request_message_id: envelope.payload.message_id, + request_message_nonce: envelope.payload.nonce, + recipient_profile_did: owner_profile_did, + recipient_endpoint_did: crate::crypto::encode_signing_key_did(owner_key), + decision: CollaborationContactDecision::Accepted, + decided_at: now + 2, + }; + let (signature, signer_did) = crate::crypto::domain_separated_sign( + owner_key, + COLLABORATION_CONTACT_DECISION_RECEIPT_SIGNATURE_DOMAIN_V1, + &serde_json::to_vec(&serde_json::to_value(&payload).unwrap()).unwrap(), + ); + let receipt = canonical_signed_collaboration_contact_decision_receipt_bytes( + &SignedCollaborationContactDecisionReceipt { + payload, + signature, + signer_did, + }, + ) + .unwrap(); + store + .record_contact_decision_receipt(&receipt, now + 2) + .unwrap(); + let contacts = store.snapshot().unwrap(); + let contact = &contacts.contacts()[0]; + assert_eq!(contact.remote_presence_device_did(), seed_did); +} + +async fn direct_addr(endpoint: &iroh::Endpoint) -> iroh::EndpointAddr { + use iroh::Watcher as _; + let mut watcher = endpoint.watch_addr(); + loop { + let addr = watcher.get(); + let is_ip = |a: &iroh::TransportAddr| matches!(a, iroh::TransportAddr::Ip(_)); + if addr.addrs.iter().any(is_ip) { + return addr; + } + watcher.updated().await.unwrap(); + } +} + +struct TwoRuntimes { + owner: tempfile::TempDir, + _seed: tempfile::TempDir, + authority: TestPasskeyAuthority, + provider: Arc, + registry: Arc, + owner_service: crate::carrier::CarrierRuntimeService, + owner_addr: iroh::EndpointAddr, + seed_node: crate::carrier::CarrierNode, + seed_did: String, + grant_id: String, +} + +impl TwoRuntimes { + async fn start() -> Self { + let owner = tempfile::tempdir().unwrap(); + let seed = tempfile::tempdir().unwrap(); + let authority = passkey_authority_with_profile(owner.path(), "Mac"); + let (owner_key, owner_did) = elastos_identity::load_or_create_did(owner.path()).unwrap(); + let (seed_key, seed_did) = elastos_identity::load_or_create_did(seed.path()).unwrap(); + let (trusted_key, _) = generate_keypair(); + let network = configured_discovery_network_profile_for_test(&trusted_key, NETWORK); + let owner_profile = load_profile_for_authority(owner.path(), &authority); + let store = CollaborationContactStore::new( + owner.path(), + &authority.principal_id, + &crate::auth::principal_localhost_root(&authority.principal_id), + network.clone(), + &owner_profile, + &owner_did, + ) + .unwrap(); + accept_seed_contact(&store, &owner_key, &owner_profile, &seed_key, &seed_did); + let provider = Arc::new(FakeModelProvider::default()); + let registry = Arc::new(ProviderRegistry::new()); + registry + .register_sub_provider("model", provider.clone()) + .await + .unwrap(); + let owner_node = crate::carrier::start_isolated_carrier_node_with_registry( + &owner_key, + &owner_did, + owner.path().to_path_buf(), + Some(Arc::downgrade(®istry)), + ) + .await + .unwrap(); + let owner_service = crate::carrier::CarrierRuntimeService::new(owner_node); + owner_service.configure_browser_exit_network(network).await; + let owner_addr = direct_addr(&owner_service.endpoint().unwrap()).await; + let seed_node = crate::carrier::start_isolated_carrier_node_with_registry( + &seed_key, + &seed_did, + seed.path().to_path_buf(), + None, + ) + .await + .unwrap(); + let fixture = Self { + owner, + _seed: seed, + authority, + provider, + registry, + owner_service, + owner_addr, + seed_node, + seed_did, + grant_id: model_grant_id(REQUEST_ID), + }; + write_home_principal_object_json_for_authority( + fixture.owner.path(), + &fixture.authority, + "services-state.json", + json!({ + "schema": "elastos.services.state/v1", + "principal_id": fixture.authority.principal_id, + "localhost_root": fixture.localhost_root(), + "updated_at": now_ts(), + "local_offer_ids": ["local:provider:model"], + }), + ); + fixture.write_request_record("approved"); + fixture + } + + fn localhost_root(&self) -> String { + crate::auth::principal_localhost_root(&self.authority.principal_id) + } + + fn write_request_record(&self, status: &str) { + let now = now_ts(); + write_home_principal_object_json_for_authority( + self.owner.path(), + &self.authority, + "services-requests.json", + json!({ + "schema": "elastos.services.requests/v1", + "principal_id": self.authority.principal_id, + "localhost_root": self.localhost_root(), + "updated_at": now, + "requests": { REQUEST_ID: { + "request_id": REQUEST_ID, + "offer_id": "local:provider:model", + "service_uri": "elastos://peer/model", + "service_kind": "remote_model", + "service_display_name": "Mac model", + "requester_peer_id": self.seed_node.endpoint.id().to_string(), + "requester_did": self.seed_did, + "requester_principal_id": SEED_PRINCIPAL, + "requester_display_name": "Seed", + "created_at": now, + "updated_at": now, + "status": status, + "authenticated_request": true, + "grant_expires_at": now + 3600, + } }, + }), + ); + } + + fn remote(&self, principal_id: &str, capsule_id: &str) -> Value { + json!({ "grant_id": self.grant_id, "principal_id": principal_id, "capsule_id": capsule_id }) + } + + fn remote_principal(&self) -> String { + remote_principal_id(&self.seed_did, SEED_PRINCIPAL) + } + + async fn call(&self, operation: &str, request: Value) -> Value { + let conn = self + .seed_node + .endpoint + .connect(self.owner_addr.clone(), CARRIER_ALPN) + .await + .unwrap(); + let (mut send, recv) = conn.open_bi().await.unwrap(); + let mut bytes = serde_json::to_vec(&json!({ + "op": "provider_invoke", "source": "model-consumer", "target": "model", + "operation": operation, "transfer": "json", "range": null, "progress": null, + "request": request, + })) + .unwrap(); + bytes.push(b'\n'); + send.write_all(&bytes).await.unwrap(); + send.finish().unwrap(); + let mut line = String::new(); + let mut reader = BufReader::new(recv); + tokio::time::timeout(Duration::from_secs(10), reader.read_line(&mut line)) + .await + .expect("owner Runtime answered within the Carrier deadline") + .unwrap(); + serde_json::from_str(line.trim()).unwrap() + } + + async fn create_run(&self, request_id: &str, offer_id: &str) -> Value { + self.call( + "runs_create", + json!({ + "op": "runs_create", "offer_id": offer_id, "operation": "text.generate", + "input": { "prompt": "hello" }, "request_id": request_id, + "remote_model": self.remote(SEED_PRINCIPAL, "assistant"), + "_runtime_invocation": { "schema": "elastos.provider.invocation/v1" }, + }), + ) + .await + } + + async fn run_operation(&self, op: &str, run_id: &str, principal_id: &str) -> Value { + self.call( + op, + json!({ + "op": op, "run_id": run_id, "request_id": format!("{op}:{principal_id}"), + "remote_model": self.remote(principal_id, "assistant"), + }), + ) + .await + } + + fn run_record(&self, run_id: &str) -> Value { + let path = self.owner.path().join("services-model-runs.json"); + let index: Value = serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap(); + assert_eq!(index["schema"], "elastos.services.model-runs/v1"); + index["runs"][run_id].clone() + } + + async fn provider_ops(&self) -> Vec { + let calls = self.provider.calls.lock().await; + let op = |call: &Value| call["op"].as_str().unwrap_or_default().to_string(); + calls.iter().map(op).collect() + } + + async fn last_provider_call(&self) -> Value { + self.provider.calls.lock().await.last().cloned().unwrap() + } + + async fn shutdown(mut self) { + self.owner_service.shutdown().await.unwrap(); + self.seed_node.shutdown().await; + } +} + +#[tokio::test] +async fn offers_list_over_carrier_shares_only_local_offers() { + let fx = TwoRuntimes::start().await; + let list = |capsule: &str| json!({ "op": "offers_list", "remote_model": fx.remote(SEED_PRINCIPAL, capsule) }); + let listed = fx.call("offers_list", list("assistant")).await; + assert_eq!(listed["ok"], true, "{listed}"); + let offers = &listed["result"]["data"]["offers"]; + assert_eq!(offers.as_array().unwrap().len(), 1, "{listed}"); + assert_eq!(offers[0]["id"], "qwen-local"); + assert!(!listed.to_string().contains("gpt-hosted"), "{listed}"); + let foreign = fx.call("offers_list", list("browser")).await; + assert_eq!(foreign["ok"], false); + assert_eq!(foreign["code"], "denied"); + assert_eq!(fx.provider_ops().await, vec!["offers_list"]); + fx.shutdown().await; +} + +#[tokio::test] +async fn runs_create_rebinds_the_request_to_the_remote_principal() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-1", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let expected_principal = fx.remote_principal(); + let run_id = run_id_for(&expected_principal, "seed-req-1"); + assert_eq!(created["result"]["data"]["run_id"], run_id); + let native = fx.last_provider_call().await; + assert_eq!( + keys(&native), + BTreeSet::from(["input", "offer_id", "op", "operation", "runtime_binding"]) + ); + assert_eq!( + native["runtime_binding"]["principal_id"], + expected_principal + ); + assert_eq!(native["runtime_binding"]["capsule_id"], "assistant"); + assert_eq!(native["runtime_binding"]["session_id"], fx.grant_id); + assert_eq!(native["runtime_binding"]["grant_id"], fx.grant_id); + assert_eq!(native["runtime_binding"]["request_id"], "seed-req-1"); + let record = fx.run_record(&run_id); + assert_eq!(record["source_endpoint_did"], fx.seed_did); + assert_eq!(record["requester_principal_id"], SEED_PRINCIPAL); + assert_eq!(record["remote_principal_id"], expected_principal); + assert_eq!(record["grant_id"], fx.grant_id); + assert_eq!(record["offer_id"], "qwen-local"); + assert!(record["terminal_at"].is_null(), "{record}"); + + let hosted = fx.create_run("seed-req-hosted", "gpt-hosted").await; + assert_eq!(hosted["ok"], false, "{hosted}"); + assert_eq!(hosted["code"], "offer_unavailable"); + assert_eq!( + fx.provider_ops().await, + vec!["offers_list", "runs_create", "offers_list"], + "a hosted offer never reaches runs_create" + ); + fx.shutdown().await; +} + +#[tokio::test] +async fn run_lifecycle_stays_with_the_creating_principal_through_denial_and_revoke() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-1", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-1"); + assert_eq!(created["result"]["data"]["run_id"], run_id); + let ops_before = fx.provider_ops().await; + + let other = fx + .run_operation("runs_get", &run_id, OTHER_SEED_PRINCIPAL) + .await; + assert_eq!(other["ok"], false, "{other}"); + assert_eq!(other["code"], "denied"); + assert_eq!(fx.provider_ops().await, ops_before); + + let first = fx + .run_operation("runs_events", &run_id, SEED_PRINCIPAL) + .await; + assert_eq!(first["ok"], true, "{first}"); + assert_eq!( + first["result"]["data"]["events"].as_array().unwrap().len(), + 2 + ); + assert_eq!(first["result"]["data"]["status"], "running"); + assert!(fx.run_record(&run_id)["terminal_at"].is_null()); + let second = fx + .run_operation("runs_events", &run_id, SEED_PRINCIPAL) + .await; + assert_eq!(second["ok"], true, "{second}"); + assert_eq!(second["result"]["data"]["status"], "completed"); + let record = fx.run_record(&run_id); + assert!(record["terminal_at"].is_u64(), "{record}"); + let binding = fx.last_provider_call().await["runtime_binding"].clone(); + assert_eq!(binding["principal_id"], fx.remote_principal()); + assert_eq!(binding["run_id"], run_id); + + let open = fx.create_run("seed-req-2", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let open_run_id = run_id_for(&fx.remote_principal(), "seed-req-2"); + fx.write_request_record("denied"); + let blocked = fx.create_run("seed-req-3", "qwen-local").await; + assert_eq!(blocked["ok"], false, "{blocked}"); + assert_eq!(blocked["code"], "denied"); + let read = fx.run_operation("runs_get", &run_id, SEED_PRINCIPAL).await; + assert_eq!(read["ok"], true, "{read}"); + assert_eq!(read["result"]["data"]["run_id"], run_id); + assert_eq!(fx.last_provider_call().await["op"], "runs_get"); + + let cancelled = cancel_grant_runs(fx.registry.clone(), fx.owner.path(), &fx.grant_id) + .await + .unwrap(); + assert_eq!(cancelled, vec![open_run_id.clone()]); + let cancel = fx.last_provider_call().await; + assert_eq!(cancel["op"], "runs_cancel"); + assert_eq!(cancel["run_id"], open_run_id); + assert_eq!( + cancel["runtime_binding"]["principal_id"], + fx.remote_principal() + ); + assert_eq!(cancel["runtime_binding"]["grant_id"], fx.grant_id); + assert_eq!(cancel["runtime_binding"]["request_id"], "revoke:seed-req-2"); + fx.shutdown().await; +} diff --git a/elastos/crates/elastos-server/src/carrier.rs b/elastos/crates/elastos-server/src/carrier.rs index 515d38487..06894a236 100644 --- a/elastos/crates/elastos-server/src/carrier.rs +++ b/elastos/crates/elastos-server/src/carrier.rs @@ -6694,7 +6694,10 @@ fn carrier_provider_public_invoke_error(index: usize, err: &anyhow::Error) -> St error = %format_args!("{err:#}"), "Carrier provider invocation failed" ); - format!("ticket[{index}] invoke failed") + match err.downcast_ref::() { + Some(decision) => format!("ticket[{index}] {}", decision.code), + None => format!("ticket[{index}] invoke failed"), + } } #[async_trait::async_trait] @@ -6912,6 +6915,42 @@ fn carrier_provider_invoke_message( }) } +/// A destination Runtime's decision, carried as one fixed token. The +/// destination chose the token from a bounded set, so passing it through +/// leaks no path, address or backend text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct CarrierProviderDecision { + pub code: &'static str, +} + +const CARRIER_PROVIDER_DECISION_CODES: [&str; 7] = [ + "denied", + "rate_limited", + "offer_unavailable", + "provider_failure", + "invalid_provider_invocation", + "unauthorized_provider_target", + "provider_registry_unavailable", +]; + +impl CarrierProviderDecision { + fn from_response(response: &serde_json::Value) -> Option { + let code = response.get("code").and_then(serde_json::Value::as_str)?; + CARRIER_PROVIDER_DECISION_CODES + .iter() + .find(|known| **known == code) + .map(|known| Self { code: known }) + } +} + +impl std::fmt::Display for CarrierProviderDecision { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.code) + } +} + +impl std::error::Error for CarrierProviderDecision {} + fn carrier_provider_invoke_result(response: serde_json::Value) -> Result { if response.get("ok").and_then(|value| value.as_bool()) == Some(true) { return Ok(response @@ -6922,8 +6961,12 @@ fn carrier_provider_invoke_result(response: serde_json::Value) -> Result Err(anyhow::Error::new(decision).context(message)), + None => anyhow::bail!(message), + } } impl CarrierClient { @@ -8338,6 +8381,27 @@ mod tests { assert_eq!(invoke, "ticket[3] invoke failed"); assert!(!invoke.contains("10.0.0.8")); assert!(!invoke.contains("root cause")); + + let decision = carrier_provider_invoke_result(serde_json::json!({ + "ok": false, + "code": "denied", + "error": "model grant is not active for /Users/owner/private", + })) + .unwrap_err(); + let public = carrier_provider_public_invoke_error(0, &decision); + assert_eq!(public, "ticket[0] denied"); + assert!(!public.contains("/Users/owner")); + + let unknown_code = carrier_provider_invoke_result(serde_json::json!({ + "ok": false, + "code": "backend_exploded_at_/tmp/x", + "error": "boom", + })) + .unwrap_err(); + assert_eq!( + carrier_provider_public_invoke_error(1, &unknown_code), + "ticket[1] invoke failed" + ); } #[test] From d9ceb3d95f70d5c13a6aef839798bf02a7d92854 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 17:10:42 -0500 Subject: [PATCH 309/535] fix(setup): drop the retention call the Browser merge left without a definition The donor moved every Browser VM image refresh function, including `browser_vm_backup_retention`, into `browser-vm-target-refresh.sh`, where the "must be 1" rule still runs before any backup is created. The merged `setup-source-home.sh` kept integration's validation call to the removed function and failed at startup on the first installed run. Verification: bash -n scripts/setup-source-home.sh bash scripts/setup-source-home-collaboration-config-smoke.sh --- scripts/setup-source-home.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/setup-source-home.sh b/scripts/setup-source-home.sh index 2ae15cc7e..2e0043f80 100755 --- a/scripts/setup-source-home.sh +++ b/scripts/setup-source-home.sh @@ -1197,7 +1197,6 @@ for tool in ffmpeg ffprobe; do exit 1 } done -browser_vm_backup_retention >/dev/null require_minimum_free_space "${ROOT}" require_minimum_free_space "${DATA_DIR}" From 54eae4d98a67f8f37d1aba69d5929b6c94f0210c Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 20:06:07 -0500 Subject: [PATCH 310/535] fix(services): route model decisions to the requester and name the request in Inbox The consumer's decision sync recognised a request only when its offer id matched the Exit or Engine suffix for the contact, so an approved model request stayed "Requested" on the seed while the owner's Inbox showed it approved. The kind-to-offer-id relation now lives in one function that the three Others card builders and the decision filter share, with a test that walks every requestable kind. Inbox copy for a service request now names the requested service (Browser Engine, AI model, Browser Exit Node) and what approval does, from one table. Found on the installed lane: Mac test Home d9ceb3d9 as owner, seed test root d9ceb3d9 as requester, request delivered and approved, decision never applied. Verification: cargo test -p elastos-server --lib -- services_kind_tests remote_model home_services (5) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_home_runtime.rs | 6 +- .../src/api/gateway_home_system.rs | 126 +++++++++++++++--- 2 files changed, 107 insertions(+), 25 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs b/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs index 6ac1fd59f..d9914303f 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_runtime.rs @@ -1114,7 +1114,7 @@ pub(super) fn home_service_offers_for_people_contact( } offers.push(HomeServiceOfferSummary { schema: "elastos.service.offer/v1".to_string(), - offer_id: format!("offer:{}:browser-exit", contact.contact_id), + offer_id: super::gateway_home_system::home_services_contact_offer_id(&contact.contact_id, "remote_exit"), service_uri: "elastos://peer/browser-exit".to_string(), service_kind: "remote_exit".to_string(), display_name: format!("{}'s Browser Exit", contact.display_name), @@ -1134,7 +1134,7 @@ pub(super) fn home_service_offers_for_people_contact( }); offers.push(HomeServiceOfferSummary { schema: "elastos.service.offer/v1".to_string(), - offer_id: format!("offer:{}:browser-engine", contact.contact_id), + offer_id: super::gateway_home_system::home_services_contact_offer_id(&contact.contact_id, crate::carrier::ENGINE_SERVICE_KIND), service_uri: crate::carrier::ENGINE_SERVICE_URI.to_string(), service_kind: crate::carrier::ENGINE_SERVICE_KIND.to_string(), display_name: format!("{}'s Browser Engine", contact.display_name), @@ -1149,7 +1149,7 @@ pub(super) fn home_service_offers_for_people_contact( }); offers.push(HomeServiceOfferSummary { schema: "elastos.service.offer/v1".to_string(), - offer_id: format!("offer:{}:model", contact.contact_id), + offer_id: super::gateway_home_system::home_services_contact_offer_id(&contact.contact_id, super::MODEL_SERVICE_KIND), service_uri: super::MODEL_SERVICE_URI.to_string(), service_kind: super::MODEL_SERVICE_KIND.to_string(), display_name: format!("{}'s AI model", contact.display_name), diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index a49b06e0c..ff969fbfd 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -2441,6 +2441,25 @@ fn home_services_supported_request(kind: &str, uri: &str) -> bool { || (kind == super::MODEL_SERVICE_KIND && uri == super::MODEL_SERVICE_URI) } +/// The one place that binds a service kind to the offer-id suffix of a +/// contact's Others card. Card builders and decision filters both use it. +pub(super) fn home_services_contact_offer_suffix(kind: &str) -> &'static str { + if kind == crate::carrier::ENGINE_SERVICE_KIND { + "browser-engine" + } else if kind == super::MODEL_SERVICE_KIND { + "model" + } else { + "browser-exit" + } +} + +pub(super) fn home_services_contact_offer_id(contact_id: &str, kind: &str) -> String { + format!( + "offer:{contact_id}:{}", + home_services_contact_offer_suffix(kind) + ) +} + fn home_services_local_model_shared( data_dir: &std::path::Path, context: &HomeLaunchTokenContext, @@ -2695,13 +2714,12 @@ fn home_services_sync_access_decisions( .values() .filter(|request| { contacts.contacts.values().any(|contact| { - let suffix = if request.service_kind == crate::carrier::ENGINE_SERVICE_KIND { - "browser-engine" - } else { - "browser-exit" - }; home_services_supported_request(&request.service_kind, &request.service_uri) - && request.offer_id == format!("offer:{}:{suffix}", contact.contact_id) + && request.offer_id + == home_services_contact_offer_id( + &contact.contact_id, + &request.service_kind, + ) && request.target_peer_id == contact.peer_id }) }) @@ -3908,6 +3926,27 @@ fn home_services_merge_access_request( changed } +/// Inbox copy per service kind: what the owner is asked for, and what an +/// approval does. +fn home_services_request_notification_copy(kind: &str, uri: &str) -> (&'static str, &'static str) { + if kind == crate::carrier::ENGINE_SERVICE_KIND && uri == crate::carrier::ENGINE_SERVICE_URI { + ( + "Browser Engine", + "Approval allows the requested Browser Engine operations through your Runtime.", + ) + } else if kind == super::MODEL_SERVICE_KIND && uri == super::MODEL_SERVICE_URI { + ( + "AI model", + "Approval lets their Assistant run your shared local model through your Runtime. Your Runtime decides every request and keeps hosted models private.", + ) + } else { + ( + "Browser Exit Node", + "Approval records your intent; Browser access still requires an installed remote Exit grant.", + ) + } +} + pub(super) fn append_home_service_access_notifications( data_dir: &std::path::Path, context: &HomeLaunchTokenContext, @@ -3933,8 +3972,8 @@ pub(super) fn append_home_service_access_notifications( if existing_ids.contains(&id) { continue; } - let engine_request = request.service_kind == crate::carrier::ENGINE_SERVICE_KIND - && request.service_uri == crate::carrier::ENGINE_SERVICE_URI; + let (service_noun, approval_effect) = + home_services_request_notification_copy(&request.service_kind, &request.service_uri); notifications.unread_count += 1; notifications.attention_count += 1; notifications.entries.push(HomeNotificationEntrySummary { @@ -3942,21 +3981,13 @@ pub(super) fn append_home_service_access_notifications( source_app: SERVICES_CAPSULE_ID.to_string(), kind: "service_access_request".to_string(), title: format!( - "{} requests your {}", - request.requester_display_name, - if engine_request { "Browser Engine" } else { "Browser Exit Node" } + "{} requests your {service_noun}", + request.requester_display_name + ), + body: format!( + "{} wants to use {}. {approval_effect}", + request.requester_display_name, request.service_display_name ), - body: if engine_request { - format!( - "{} wants to use {}. Approval allows the requested Browser Engine operations through your Runtime.", - request.requester_display_name, request.service_display_name - ) - } else { - format!( - "{} wants to use {}. Approval records your intent; Browser access still requires an installed remote Exit grant.", - request.requester_display_name, request.service_display_name - ) - }, action_ref: Some(HomeNotificationActionSummary { app: SERVICES_CAPSULE_ID.to_string(), action_id: format!("service-approve-request:{}", request.request_id), @@ -7128,3 +7159,54 @@ mod home_realtime_tests { ); } } + +#[cfg(test)] +mod services_kind_tests { + use super::*; + + // The Others card id and the decision filter must agree for every kind; + // a model decision was silently ignored when the filter knew two kinds. + #[test] + fn contact_offer_id_covers_every_requestable_service_kind() { + for (kind, uri, suffix) in [ + ( + HOME_REMOTE_EXIT_SERVICE_KIND, + HOME_BROWSER_EXIT_PEER_SERVICE_URI, + "browser-exit", + ), + ( + crate::carrier::ENGINE_SERVICE_KIND, + crate::carrier::ENGINE_SERVICE_URI, + "browser-engine", + ), + ( + super::super::MODEL_SERVICE_KIND, + super::super::MODEL_SERVICE_URI, + "model", + ), + ] { + assert!(home_services_supported_request(kind, uri), "{kind}"); + assert_eq!( + home_services_contact_offer_id("c1", kind), + format!("offer:c1:{suffix}") + ); + } + } + + #[test] + fn request_notification_copy_names_the_requested_service() { + let (noun, effect) = home_services_request_notification_copy( + super::super::MODEL_SERVICE_KIND, + super::super::MODEL_SERVICE_URI, + ); + assert_eq!(noun, "AI model"); + assert!(effect.contains("shared local model")); + let (noun, _) = home_services_request_notification_copy( + crate::carrier::ENGINE_SERVICE_KIND, + crate::carrier::ENGINE_SERVICE_URI, + ); + assert_eq!(noun, "Browser Engine"); + let (noun, _) = home_services_request_notification_copy("remote_exit", "elastos://x"); + assert_eq!(noun, "Browser Exit Node"); + } +} From 36f4d7f6daf118725cc5e18c34c948fb14d49e20 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 20:23:04 -0500 Subject: [PATCH 311/535] fix(model): list granted remote offers on a Home without a local model provider The consumer proxy appended remote offers to the local `offers_list` reply. A Home that runs no model provider (the Linux seed without a GPU) answered `provider_error` there, so the Assistant showed "No model offer on this Home yet" although a grant was approved. When remote grants exist and the local reply is not `ok`, the proxy now starts from the empty typed offers list, with `local_provider: "unavailable"`, and appends the granted offers to it. Found on the installed lane: seed test root 54eae4d9 with an approved grant. Verification: cargo test -p elastos-server --lib -- gateway_model_remote remote_model cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_remote.rs | 32 +++++++++++++++++++ .../src/api/gateway_provider_proxy.rs | 5 +++ 2 files changed, 37 insertions(+) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 7a4a24641..cf5134cb1 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -320,6 +320,22 @@ fn rewrite_offer_ids(value: &mut Value, grant_id: &str) { /// Append the offers each approved grant currently shares. An unreachable /// grant contributes a status entry and no offers. +/// The typed `offers_list` reply of a Home that runs no local model provider. +/// Remote offers append to it, so a Linux Home without a model still lists +/// every granted service. `local_provider` names why the local list is empty. +pub(crate) fn offers_list_without_local_provider() -> Value { + json!({ + "status": "ok", + "data": { + "schema": "elastos.model.offers-list/v1", + "provider": "model-provider", + "protocol_version": "elastos.model-provider/v1", + "offers": [], + "local_provider": "unavailable", + } + }) +} + pub(crate) async fn append_remote_offers( registry: &ProviderRegistry, grants: &[ConsumerModelGrant], @@ -467,6 +483,22 @@ pub(crate) async fn route_run_operation( mod tests { use super::*; + #[test] + fn a_home_without_a_local_model_provider_still_lists_remote_offers() { + let mut response = offers_list_without_local_provider(); + assert_eq!(response["status"], "ok"); + assert_eq!(response["data"]["schema"], "elastos.model.offers-list/v1"); + assert_eq!(response["data"]["offers"], json!([])); + assert_eq!(response["data"]["local_provider"], "unavailable"); + // The same target array `append_remote_offers` pushes into. + response + .pointer_mut("/data/offers") + .and_then(Value::as_array_mut) + .expect("offers array") + .push(json!({ "id": "remote:g:qwen" })); + assert_eq!(response["data"]["offers"].as_array().unwrap().len(), 1); + } + #[test] fn remote_offer_ids_round_trip_and_reject_malformed_values() { let id = remote_offer_id("services-remote-model-grant-0011223344556677", "qwen3.5-9b"); diff --git a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs index 7a9654c46..e2aa30370 100644 --- a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs +++ b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs @@ -2008,6 +2008,11 @@ pub(super) async fn gateway_provider_proxy( }; if scheme == "model" { if op == "offers_list" && !remote_model_grants.is_empty() { + // A consumer Home may run no local model provider at all; granted + // remote offers still form its list. + if response.get("status").and_then(serde_json::Value::as_str) != Some("ok") { + response = super::gateway_model_remote::offers_list_without_local_provider(); + } super::gateway_model_remote::append_remote_offers( ®istry, &remote_model_grants, From 9bef09adef9bb910ca34350e0d1ea6743ffd0c3a Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Fri, 11 Sep 2026 20:44:23 -0500 Subject: [PATCH 312/535] fix(model): route consumer calls by the granting Runtime's did:key The services layer stores the granting Runtime's Carrier peer id under `peer_did` (the wire shape shared with Engine grants). The Carrier route pins ticket endpoints to a `did:key`, so the raw peer id matched no endpoint and every consumer call ended as `transport_interrupted`. The consumer grant now converts the peer id once, the way the Engine consumer does, and a malformed id yields no grant. Found on the installed lane: seed test root 36f4d7f6 with an approved grant, `offers_list` reporting `remote_services[0].status = transport_interrupted`. Verification: cargo test -p elastos-server --lib -- gateway_model_remote remote_model (11) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_remote.rs | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index cf5134cb1..080e77ad5 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -144,10 +144,17 @@ pub(crate) struct ConsumerModelGrant { } impl ConsumerModelGrant { + /// The services layer stores the granting Runtime's Carrier peer id under + /// `peer_did` (shared wire shape with Engine grants). The Carrier route + /// pins the ticket to a `did:key`, so the id is converted once here. fn from_record(grant: &Value) -> Option { + let peer_id = grant["peer_did"] + .as_str()? + .parse::() + .ok()?; Some(Self { grant_id: grant["grant_id"].as_str()?.to_string(), - peer_did: grant["peer_did"].as_str()?.to_string(), + peer_did: crate::carrier::public_key_to_did(&peer_id).ok()?, connect_ticket: grant["connect_ticket"].as_str()?.to_string(), display_name: grant["service_display_name"] .as_str() @@ -483,6 +490,30 @@ pub(crate) async fn route_run_operation( mod tests { use super::*; + // The Carrier route filters ticket endpoints by `did:key`; a raw peer id + // there matched nothing and every call reported `transport_interrupted`. + #[test] + fn consumer_grant_routes_by_the_granting_runtime_did() { + let key = iroh::SecretKey::from_bytes(&[7; 32]).public(); + let grant = ConsumerModelGrant::from_record(&json!({ + "grant_id": "services-remote-model-grant-0011223344556677", + "peer_did": key.to_string(), + "connect_ticket": "ticket", + "service_display_name": "Mac", + "expires_at": 1, + })) + .expect("grant"); + assert_eq!( + grant.peer_did, + crate::carrier::public_key_to_did(&key).unwrap() + ); + assert!(grant.peer_did.starts_with("did:key:z6Mk")); + assert!(ConsumerModelGrant::from_record(&json!({ + "grant_id": "g", "peer_did": "not-a-peer-id", "connect_ticket": "t", "expires_at": 1, + })) + .is_none()); + } + #[test] fn a_home_without_a_local_model_provider_still_lists_remote_offers() { let mut response = offers_list_without_local_provider(); From a9026bf6278bebe085a974dfc6700d53b0b5135b Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 09:32:47 -0500 Subject: [PATCH 313/535] fix(model): settle remote run records from the real provider contract and reserve capacity first Both run indexes read a `status` field to decide that a run had settled. A `runs_events` page carries no status; its terminal event carries `terminal: true`. The Assistant settles from that stream, so every completed run stayed recorded as open on the owner and on the consumer, and the fake provider in the loopback test added the field the real one lacks. One shared predicate now reads the run view (`status`, `terminal`) and the events page (`events[*].terminal`), and the test provider answers with the contract shapes. Open records evaded pruning, so the owner index could fill; a `runs_create` then dispatched model work and failed to record it. Both sides now reserve a record slot before any dispatch and refuse with `rate_limited` when full; if recording fails after a dispatch, the dispatched run is cancelled. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model (19) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_remote.rs | 51 ++++++++---- .../src/api/gateway_model_service.rs | 82 +++++++++++++++---- .../src/api/gateway_tests/remote_model.rs | 65 +++++++++++++-- 3 files changed, 160 insertions(+), 38 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 080e77ad5..57ebe15d7 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -293,17 +293,18 @@ fn run_id_of(result: &Value) -> Option { .map(str::to_string) } -fn run_is_terminal(result: &Value) -> bool { - let status = result - .pointer("/data/run/status") - .or_else(|| result.pointer("/data/status")) - .or_else(|| result.pointer("/run/status")) - .and_then(Value::as_str) - .unwrap_or_default(); - matches!( - status, - "completed" | "failed" | "cancelled" | "settlement_unknown" - ) +use super::gateway_model_service::run_result_is_terminal as run_is_terminal; + +/// Prune settled routes past retention and confirm one more run fits. +fn reserve_run_route(data_dir: &Path, now: u64) -> Result<(), RemoteRouteError> { + update_index(data_dir, now, |index| { + anyhow::ensure!( + index.runs.len() < REMOTE_RUN_INDEX_MAX, + "remote run index is full" + ); + Ok(()) + }) + .map_err(|err| RemoteRouteError::Invalid(err.to_string())) } fn rewrite_offer_ids(value: &mut Value, grant_id: &str) { @@ -431,6 +432,9 @@ pub(crate) async fn route_run_operation( })?; let mut request = typed_request(normalized, context, capsule_id, grant_id); request["offer_id"] = Value::String(inner_offer_id.to_string()); + // Route capacity is reserved before the granting Runtime starts + // work, so a full index refuses here instead of orphaning a run. + reserve_run_route(data_dir, now)?; let mut result = call_grant(®istry, grant, op, request).await?; if let Some(run_id) = run_id_of(&result) { let route = RemoteRunRoute { @@ -446,11 +450,28 @@ pub(crate) async fn route_run_operation( created_at: now, terminal_at: run_is_terminal(&result).then_some(now), }; - update_index(data_dir, now, |index| { - index.runs.entry(run_id).or_insert(route); + if let Err(err) = update_index(data_dir, now, |index| { + index.runs.entry(run_id.clone()).or_insert(route); Ok(()) - }) - .map_err(|err| RemoteRouteError::Invalid(err.to_string()))?; + }) { + // The granting Runtime runs work this Home cannot address; + // ask it to settle that run before reporting the failure. + let cancel = typed_request( + &json!({ "op": "runs_cancel", "run_id": run_id, "runtime_binding": { + "request_id": format!("unrouted:{run_id}") } }), + context, + capsule_id, + grant_id, + ); + if let Err(cancel_err) = + call_grant(®istry, grant, "runs_cancel", cancel).await + { + tracing::warn!( + "unrouted remote model run {run_id}: cancel failed: {cancel_err}" + ); + } + return Err(RemoteRouteError::Invalid(err.to_string())); + } } rewrite_offer_ids(&mut result, grant_id); Ok(Some(result)) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index 94a44c5d4..689a21ab7 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -179,6 +179,43 @@ fn update_run_index( Ok(result) } +/// Prune settled records past retention and confirm one more run fits. +fn reserve_run_record(data_dir: &Path, now: u64) -> anyhow::Result<()> { + update_run_index(data_dir, now, |index| { + anyhow::ensure!( + index.runs.len() < MODEL_RUN_INDEX_MAX, + "model run index is full" + ); + Ok(()) + }) +} + +/// Best-effort cancel of a run this Runtime dispatched but could not record. +async fn cancel_unrecorded_run( + registry: &ProviderRegistry, + context: &HomeLaunchTokenContext, + capsule_id: &str, + run_id: &str, +) { + let request = json!({ "run_id": run_id, "request_id": format!("unrecorded:{run_id}") }); + let Ok(normalized) = + normalize_model_provider_request("runs_cancel", &request, context, capsule_id) + else { + tracing::warn!("unrecorded model run {run_id} could not form a cancel request"); + return; + }; + match registry.send_raw("model", &normalized).await { + Ok(result) if provider_status_error(&result).is_none() => { + tracing::info!("unrecorded model run {run_id}: cancel requested") + } + Ok(result) => tracing::warn!( + "unrecorded model run {run_id}: cancel returned {}", + provider_status_error(&result).unwrap_or_default() + ), + Err(err) => tracing::warn!("unrecorded model run {run_id}: cancel failed: {err}"), + } +} + pub(crate) fn remote_model_run( data_dir: &Path, run_id: &str, @@ -308,18 +345,23 @@ async fn read_authority( .context("model authority deadline")?? } -fn run_terminal(result: &Value) -> bool { - let status = result - .pointer("/data/run/status") - .or_else(|| result.pointer("/data/status")) - .or_else(|| result.pointer("/run/status")) - .or_else(|| result.get("status")) - .and_then(Value::as_str) - .unwrap_or_default(); - matches!( - status, - "completed" | "failed" | "cancelled" | "settlement_unknown" - ) +/// Whether a typed model reply shows the run has settled. `runs_create`, +/// `runs_get` and `runs_cancel` answer with a run view whose `status` is one of +/// the terminal statuses (a `terminal` outcome accompanies it). `runs_events` +/// answers with an events page and no status; its terminal event carries +/// `terminal: true`. Both indexes settle their records from this one reading. +pub(super) fn run_result_is_terminal(result: &Value) -> bool { + let data = &result["data"]; + let view_terminal = data["status"].as_str().is_some_and(|status| { + matches!( + status, + "completed" | "failed" | "cancelled" | "settlement_unknown" + ) + }) || data["terminal"].is_object(); + let event_terminal = data["events"] + .as_array() + .is_some_and(|events| events.iter().any(|event| event["terminal"] == true)); + view_terminal || event_terminal } /// Serve one model operation for a remote consumer. Every path returns a @@ -489,6 +531,15 @@ pub(crate) async fn invoke( "model offer is not shared through this grant", ); } + // Recording capacity is reserved before any model work starts, so a + // full index refuses the request instead of orphaning a dispatched run. + if let Err(err) = reserve_run_record(data_dir, now) { + tracing::info!("remote model run refused before dispatch: {err}"); + return denied( + "rate_limited", + "model run capacity on this Runtime is exhausted", + ); + } } let result = match registry.send_raw("model", &normalized).await { @@ -532,13 +583,16 @@ pub(crate) async fn invoke( .unwrap_or_default() .to_string(), created_at: now, - terminal_at: run_terminal(&result).then_some(now), + terminal_at: run_result_is_terminal(&result).then_some(now), }; if let Err(err) = update_run_index(data_dir, now, |index| { index.runs.entry(run_id.clone()).or_insert(record); Ok(()) }) { + // The run exists on this Runtime but has no owner record; + // settle it now rather than leave it running unaccounted. tracing::warn!("remote model run index write failed: {err}"); + cancel_unrecorded_run(®istry, &context, capsule_id, &run_id).await; return denied( "provider_failure", "model run could not be recorded on this Runtime", @@ -549,7 +603,7 @@ pub(crate) async fn invoke( } _ => { if let Some((run_id, record)) = indexed_run { - if record.terminal_at.is_none() && run_terminal(&result) { + if record.terminal_at.is_none() && run_result_is_terminal(&result) { let _ = update_run_index(data_dir, now, |index| { if let Some(entry) = index.runs.get_mut(&run_id) { entry.terminal_at = Some(now); diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 1afcc54be..9857d316a 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -37,24 +37,36 @@ fn run_id_for(principal_id: &str, request_id: &str) -> String { ) } +// Replies follow the provider contract (`capsules/model-provider/src/contract.rs`): +// a `RunView` carries `status` and, once settled, a `terminal` outcome; a +// `RunEventsPage` carries no status at all, only events with a `terminal` flag. fn run_view(run_id: &str, status: &str) -> Value { - json!({ "status": "ok", "data": { + let mut view = json!({ "schema": "elastos.model.run-view/v1", "run_id": run_id, "offer_id": "qwen-local", - "operation": "text.generate", "status": status, "sequence_cursor": 0 } }) + "operation": "text.generate", "status": status, "sequence_cursor": 0 }); + if matches!( + status, + "completed" | "failed" | "cancelled" | "settlement_unknown" + ) { + view["terminal"] = json!({ "status": status }); + } + json!({ "status": "ok", "data": view }) } fn events_page(run_id: &str, page: u32) -> Value { - let event = |sequence: u64, text: &str, terminal: bool| { - json!({ "schema": "elastos.model.run-event/v1", "sequence": sequence, "kind": "output", + let event = |sequence: u64, kind: &str, text: &str, terminal: bool| { + json!({ "schema": "elastos.model.run-event/v1", "sequence": sequence, "kind": kind, "data": { "schema": "elastos.model.output.text/v1", "text": text }, "terminal": terminal }) }; let events = match page { - 1 => vec![event(1, "hel", false), event(2, "lo", false)], - _ => vec![event(3, "", true)], + 1 => vec![ + event(1, "delta", "hel", false), + event(2, "delta", "lo", false), + ], + _ => vec![event(3, "output", "hello", true)], }; json!({ "status": "ok", "data": { "schema": "elastos.model.run-events/v1", "run_id": run_id, - "status": if page == 1 { "running" } else { "completed" }, "next_cursor": events.len(), "has_more": page == 1, "events": events } }) } @@ -556,13 +568,14 @@ async fn run_lifecycle_stays_with_the_creating_principal_through_denial_and_revo first["result"]["data"]["events"].as_array().unwrap().len(), 2 ); - assert_eq!(first["result"]["data"]["status"], "running"); + // An events page has no status; only its terminal event settles the record. + assert!(first["result"]["data"].get("status").is_none(), "{first}"); assert!(fx.run_record(&run_id)["terminal_at"].is_null()); let second = fx .run_operation("runs_events", &run_id, SEED_PRINCIPAL) .await; assert_eq!(second["ok"], true, "{second}"); - assert_eq!(second["result"]["data"]["status"], "completed"); + assert_eq!(second["result"]["data"]["events"][0]["terminal"], true); let record = fx.run_record(&run_id); assert!(record["terminal_at"].is_u64(), "{record}"); let binding = fx.last_provider_call().await["runtime_binding"].clone(); @@ -596,3 +609,37 @@ async fn run_lifecycle_stays_with_the_creating_principal_through_denial_and_revo assert_eq!(cancel["runtime_binding"]["request_id"], "revoke:seed-req-2"); fx.shutdown().await; } + +#[tokio::test] +async fn a_full_run_index_refuses_before_any_model_work_starts() { + let fx = TwoRuntimes::start().await; + // 512 open records: the destination's recording capacity is exhausted. + let mut runs = serde_json::Map::new(); + for n in 0..512 { + runs.insert( + format!("run:sha256:{n:064x}"), + json!({ + "grant_id": fx.grant_id, "source_endpoint_did": fx.seed_did, + "requester_principal_id": SEED_PRINCIPAL, + "remote_principal_id": fx.remote_principal(), "capsule_id": "assistant", + "offer_id": "qwen-local", "request_id": format!("open-{n}"), "created_at": now_ts(), + }), + ); + } + std::fs::write( + fx.owner.path().join("services-model-runs.json"), + serde_json::to_vec(&json!({ "schema": "elastos.services.model-runs/v1", "runs": runs })) + .unwrap(), + ) + .unwrap(); + + let refused = fx.create_run("seed-req-full", "qwen-local").await; + assert_eq!(refused["ok"], false, "{refused}"); + assert_eq!(refused["code"], "rate_limited"); + // Only the offer check reached the provider; no run was dispatched. + assert_eq!(fx.provider_ops().await, vec!["offers_list"]); + assert!(fx + .run_record(&run_id_for(&fx.remote_principal(), "seed-req-full")) + .is_null()); + fx.shutdown().await; +} From 3f26235a3a2f613d6b22de43e72fbece369e89a6 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 09:40:25 -0500 Subject: [PATCH 314/535] fix(model): settle routed runs after the grant expires or is denied The consumer required a currently approved grant even to read or cancel a run it had already routed, and the proxy skipped remote routing entirely when no active grant remained. That contradicted the owning Runtime, which keeps `runs_get`, `runs_events` and `runs_cancel` open for runs created under a grant. The stored route now carries the granting Runtime's ticket and display name; run operations always consult the route, settle through it when the grant is gone, and only `runs_create` needs an active grant. The loopback fixture gains the seed's own registry with a Carrier invoker, so a test drives `route_run_operation` the way the seed gateway does: create under an active grant, deny on the owner, then settle both event pages with no active grant while a new create is refused with `denied`. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model consumer_path (20) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_remote.rs | 56 ++++++- .../src/api/gateway_provider_proxy.rs | 9 +- .../src/api/gateway_tests/remote_model.rs | 147 +++++++++++++++++- 3 files changed, 199 insertions(+), 13 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 57ebe15d7..ea38e8847 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -45,6 +45,12 @@ pub(crate) struct RemoteRunRoute { pub principal_id: String, pub grant_id: String, pub peer_did: String, + /// The granting Runtime's Carrier ticket at creation time. Settlement of + /// this run (`runs_get`, `runs_events`, `runs_cancel`) keeps working after + /// the grant expires or is denied; the granting Runtime still decides. + #[serde(default)] + pub connect_ticket: String, + pub display_name: String, pub offer_id: String, pub request_id: String, pub created_at: u64, @@ -52,6 +58,20 @@ pub(crate) struct RemoteRunRoute { pub terminal_at: Option, } +impl RemoteRunRoute { + /// The grant facts needed to settle this run, independent of the grant's + /// current status on this Home. Authority to start new runs is not implied. + fn settlement_grant(&self) -> Option { + (!self.connect_ticket.is_empty()).then(|| ConsumerModelGrant { + grant_id: self.grant_id.clone(), + peer_did: self.peer_did.clone(), + connect_ticket: self.connect_ticket.clone(), + display_name: self.display_name.clone(), + expires_at: 0, + }) + } +} + #[derive(Debug, Clone, Serialize, Deserialize, Default)] struct RemoteRunIndex { schema: String, @@ -147,7 +167,7 @@ impl ConsumerModelGrant { /// The services layer stores the granting Runtime's Carrier peer id under /// `peer_did` (shared wire shape with Engine grants). The Carrier route /// pins the ticket to a `did:key`, so the id is converted once here. - fn from_record(grant: &Value) -> Option { + pub(crate) fn from_record(grant: &Value) -> Option { let peer_id = grant["peer_did"] .as_str()? .parse::() @@ -441,6 +461,8 @@ pub(crate) async fn route_run_operation( principal_id: context.principal_id.clone(), grant_id: grant.grant_id.clone(), peer_did: grant.peer_did.clone(), + connect_ticket: grant.connect_ticket.clone(), + display_name: grant.display_name.clone(), offer_id: offer_id.to_string(), request_id: normalized .pointer("/runtime_binding/request_id") @@ -483,14 +505,19 @@ pub(crate) async fn route_run_operation( else { return Ok(None); }; - let grant = grants - .iter() - .find(|grant| grant.grant_id == route.grant_id) - .ok_or_else(|| RemoteRouteError::Rejected { - code: "denied".to_string(), - })?; + // An active grant is preferred; a run created under a grant that + // has since expired or been denied still settles through the + // route it was created with. The granting Runtime owns that check. + let grant = match grants.iter().find(|grant| grant.grant_id == route.grant_id) { + Some(grant) => grant.clone(), + None => route + .settlement_grant() + .ok_or_else(|| RemoteRouteError::Rejected { + code: "denied".to_string(), + })?, + }; let request = typed_request(normalized, context, capsule_id, &route.grant_id); - let mut result = call_grant(®istry, grant, op, request).await?; + let mut result = call_grant(®istry, &grant, op, request).await?; if route.terminal_at.is_none() && run_is_terminal(&result) { let run_id = run_id.to_string(); let _ = update_index(data_dir, now, |index| { @@ -619,11 +646,24 @@ mod tests { principal_id: "p1".into(), grant_id: "g".into(), peer_did: "did:key:z6Mkpeer".into(), + connect_ticket: "ticket".into(), + display_name: "Mac".into(), offer_id: "remote:g:qwen".into(), request_id: "req".into(), created_at: 10, terminal_at: None, }; + // A route written before tickets were stored cannot settle by itself. + assert!(RemoteRunRoute { + connect_ticket: String::new(), + ..route.clone() + } + .settlement_grant() + .is_none()); + assert_eq!( + route.settlement_grant().map(|grant| grant.grant_id), + Some("g".to_string()) + ); update_index(dir.path(), 10, |index| { index.runs.insert("run:sha256:aa".into(), route.clone()); index.runs.insert( diff --git a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs index e2aa30370..8b0cf39f7 100644 --- a/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs +++ b/elastos/crates/elastos-server/src/api/gateway_provider_proxy.rs @@ -1760,15 +1760,18 @@ pub(super) async fn gateway_provider_proxy( Ok(value) => value, Err((status, message)) => return (status, message).into_response(), }; - // Approved remote model grants route runs to the granting Runtime. A - // grant lookup failure keeps the local path; it cannot widen authority. + // Approved remote model grants route new runs to the granting Runtime. + // A grant lookup failure keeps the local path; it cannot widen + // authority. Runs already routed remotely settle through their stored + // route even after the grant expired or was denied, so every run + // operation consults the route; a local run returns to the local path. remote_model_grants = super::gateway_model_remote::consumer_grants( &state.data_dir, &context, state.collaboration_discovery_service.as_ref(), ) .unwrap_or_default(); - if !remote_model_grants.is_empty() && op != "offers_list" { + if op != "offers_list" { match super::gateway_model_remote::route_run_operation( Arc::clone(®istry), &state.data_dir, diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 9857d316a..03097415c 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -287,10 +287,12 @@ async fn direct_addr(endpoint: &iroh::Endpoint) -> iroh::EndpointAddr { struct TwoRuntimes { owner: tempfile::TempDir, - _seed: tempfile::TempDir, + seed: tempfile::TempDir, authority: TestPasskeyAuthority, provider: Arc, registry: Arc, + /// The seed's own registry with a Carrier invoker: the real consumer path. + seed_registry: Arc, owner_service: crate::carrier::CarrierRuntimeService, owner_addr: iroh::EndpointAddr, seed_node: crate::carrier::CarrierNode, @@ -343,12 +345,22 @@ impl TwoRuntimes { ) .await .unwrap(); + let seed_registry = Arc::new(ProviderRegistry::new()); + seed_registry + .set_carrier_invoker(Arc::new( + crate::carrier::CarrierProviderInvoker::with_carrier_endpoint_and_registry( + seed_node.endpoint.clone(), + Arc::downgrade(&seed_registry), + ), + )) + .await; let fixture = Self { owner, - _seed: seed, + seed, authority, provider, registry, + seed_registry, owner_service, owner_addr, seed_node, @@ -643,3 +655,134 @@ async fn a_full_run_index_refuses_before_any_model_work_starts() { .is_null()); fx.shutdown().await; } + +// The seed gateway's own routing over Carrier: the path the Assistant uses. +mod consumer_path { + use super::*; + use crate::api::gateway::gateway_model_remote::{ + remote_offer_id, remote_run_route, route_run_operation, ConsumerModelGrant, + RemoteRouteError, + }; + + fn ticket_for(endpoint: &iroh::EndpointAddr) -> String { + let bytes = serde_json::to_vec(&json!({ "topic": null, "endpoints": [endpoint] })).unwrap(); + let mut encoded = data_encoding::BASE32_NOPAD.encode(&bytes); + encoded.make_ascii_lowercase(); + encoded + } + + fn seed_context() -> HomeLaunchTokenContext { + HomeLaunchTokenContext { + principal_id: SEED_PRINCIPAL.to_string(), + session_id: "auth:seed".to_string(), + proof_binding_id: None, + grant_id: "grant:seed-launch".to_string(), + } + } + + impl TwoRuntimes { + fn consumer_grant(&self) -> ConsumerModelGrant { + ConsumerModelGrant::from_record(&json!({ + "grant_id": self.grant_id, + "peer_did": self.owner_service.endpoint().unwrap().id().to_string(), + "connect_ticket": ticket_for(&self.owner_addr), + "service_display_name": "Mac model", + "expires_at": now_ts() + 3600, + })) + .expect("grant record") + } + + async fn route( + &self, + grants: &[ConsumerModelGrant], + op: &str, + normalized: Value, + ) -> Result, RemoteRouteError> { + route_run_operation( + self.seed_registry.clone(), + self.seed.path(), + grants, + &seed_context(), + "assistant", + op, + &normalized, + now_ts(), + ) + .await + } + } + + fn access(op: &str, run_id: &str, request_id: &str) -> Value { + json!({ "op": op, "run_id": run_id, "after_sequence": 0, + "runtime_binding": { "principal_id": SEED_PRINCIPAL, "request_id": request_id } }) + } + + #[tokio::test] + async fn existing_runs_settle_after_the_grant_is_denied_while_new_runs_are_refused() { + let fx = TwoRuntimes::start().await; + let grant = fx.consumer_grant(); + let create = json!({ "op": "runs_create", "offer_id": remote_offer_id(&fx.grant_id, "qwen-local"), + "operation": "text.generate", "input": { "prompt": "hello" }, + "runtime_binding": { "principal_id": SEED_PRINCIPAL, "request_id": "seed-req-c" } }); + let created = fx + .route(std::slice::from_ref(&grant), "runs_create", create.clone()) + .await + .unwrap() + .expect("remote route"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-c"); + assert_eq!(created["data"]["run_id"], run_id); + let route = remote_run_route(fx.seed.path(), SEED_PRINCIPAL, &run_id) + .unwrap() + .expect("stored route"); + assert_eq!(route.grant_id, fx.grant_id); + assert_eq!(route.connect_ticket, grant.connect_ticket); + assert!(route.terminal_at.is_none()); + + // The owner denies the request: no active grant remains on the seed. + fx.write_request_record("denied"); + let refused = fx.route(&[], "runs_create", create).await; + assert!( + matches!(refused, Err(RemoteRouteError::Rejected { ref code }) if code == "denied"), + "{refused:?}" + ); + + let page = fx + .route( + &[], + "runs_events", + access("runs_events", &run_id, "seed-ev-1"), + ) + .await + .unwrap() + .expect("settlement route"); + assert_eq!(page["data"]["events"].as_array().unwrap().len(), 2); + assert_eq!(fx.last_provider_call().await["op"], "runs_events"); + let last = fx + .route( + &[], + "runs_events", + access("runs_events", &run_id, "seed-ev-2"), + ) + .await + .unwrap() + .expect("settlement route"); + assert_eq!(last["data"]["events"][0]["terminal"], true); + let settled = remote_run_route(fx.seed.path(), SEED_PRINCIPAL, &run_id) + .unwrap() + .expect("stored route"); + assert!(settled.terminal_at.is_some(), "{settled:?}"); + assert!(fx.run_record(&run_id)["terminal_at"].is_u64()); + + // A run this Home never routed remotely stays on the local path. + let local = fx + .route( + &[], + "runs_get", + access("runs_get", "run:sha256:00", "seed-local"), + ) + .await + .unwrap(); + assert!(local.is_none()); + fx.shutdown().await; + } +} From 5f31a934decfa4262a0a505f16bda1731a90600d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 10:20:11 -0500 Subject: [PATCH 315/535] fix(services): cancel a denied grant's runs whether or not the denial was delivered The Inbox deny handler returned the gossip delivery error before its cancellation sweep, so a denial the requester could not receive left that requester's runs working on the owner's Runtime. Local revocation effects now run first: the sweep always follows the recorded denial, and the delivery result is returned after it. A settled cancel also closes the run record, so a later sweep does not cancel the same run again. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model (21) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../crates/elastos-server/src/api/gateway.rs | 7 +-- .../elastos-server/src/api/gateway_inbox.rs | 23 ++++------ .../src/api/gateway_model_service.rs | 40 ++++++++++++++++- .../src/api/gateway_tests/remote_model.rs | 43 ++++++++++++++++++- 4 files changed, 93 insertions(+), 20 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway.rs b/elastos/crates/elastos-server/src/api/gateway.rs index 6411260f2..681eccd80 100644 --- a/elastos/crates/elastos-server/src/api/gateway.rs +++ b/elastos/crates/elastos-server/src/api/gateway.rs @@ -81,9 +81,10 @@ mod gateway_model_remote; #[path = "gateway_model_service.rs"] mod gateway_model_service; pub(crate) use gateway_model_service::{ - cancel_grant_runs as cancel_remote_model_grant_runs, invoke as invoke_remote_model, - model_grant_id, ModelServiceGrant, MODEL_GRANT_SCHEMA, MODEL_GRANT_SCOPE, MODEL_GRANT_TTL_SECS, - MODEL_LOCAL_OFFER, MODEL_OPERATIONS, MODEL_SERVICE_KIND, MODEL_SERVICE_URI, + invoke as invoke_remote_model, model_grant_id, + settle_denied_grant as settle_denied_model_grant, ModelServiceGrant, MODEL_GRANT_SCHEMA, + MODEL_GRANT_SCOPE, MODEL_GRANT_TTL_SECS, MODEL_LOCAL_OFFER, MODEL_OPERATIONS, + MODEL_SERVICE_KIND, MODEL_SERVICE_URI, }; #[path = "gateway_origin.rs"] mod gateway_origin; diff --git a/elastos/crates/elastos-server/src/api/gateway_inbox.rs b/elastos/crates/elastos-server/src/api/gateway_inbox.rs index 5f9169356..ed11fb3aa 100644 --- a/elastos/crates/elastos-server/src/api/gateway_inbox.rs +++ b/elastos/crates/elastos-server/src/api/gateway_inbox.rs @@ -396,21 +396,14 @@ async fn dispatch_inbox_action( } }) .await - .map_err(|err| anyhow::anyhow!(err))??; - // Revocation closes future authority and settles the runs it had opened. - if let Some(registry) = state.provider_registry.clone() { - match super::cancel_remote_model_grant_runs(registry, &data_dir, &grant_id).await { - Ok(cancelled) if !cancelled.is_empty() => tracing::info!( - "revoked model grant {grant_id}: cancel requested for {} run(s)", - cancelled.len() - ), - Ok(_) => {} - Err(err) => { - tracing::warn!("revoked model grant {grant_id}: cancel sweep failed: {err}") - } - } - } - return Ok(denied); + .map_err(|err| anyhow::anyhow!(err))?; + return super::settle_denied_model_grant( + state.provider_registry.clone(), + &data_dir, + &grant_id, + denied, + ) + .await; } if let Some(request_id) = action_id.strip_prefix("inspect-approve-request:") { let Some(step_up_token) = action.step_up_token.as_deref() else { diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index 689a21ab7..5b25385a6 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -618,6 +618,31 @@ pub(crate) async fn invoke( json!({ "ok": true, "result": result }) } +/// The owner's denial has two effects: the recorded decision, which the +/// requester learns about through gossip, and the settlement of runs the +/// grant had opened on this Runtime. The second effect never waits for the +/// first. The denial result is returned unchanged after the sweep. +pub(crate) async fn settle_denied_grant( + registry: Option>, + data_dir: &Path, + grant_id: &str, + denied: anyhow::Result, +) -> anyhow::Result { + if let Some(registry) = registry { + match cancel_grant_runs(registry, data_dir, grant_id).await { + Ok(cancelled) if !cancelled.is_empty() => tracing::info!( + "revoked model grant {grant_id}: cancel requested for {} run(s)", + cancelled.len() + ), + Ok(_) => {} + Err(err) => { + tracing::warn!("revoked model grant {grant_id}: cancel sweep failed: {err}") + } + } + } + denied +} + /// Cancel every open run created under a grant, after revoke or denial. /// Returns the run ids that received a cancel request. pub(crate) async fn cancel_grant_runs( @@ -641,7 +666,20 @@ pub(crate) async fn cancel_grant_runs( continue; }; match registry.send_raw("model", &normalized).await { - Ok(result) if provider_status_error(&result).is_none() => cancelled.push(run_id), + Ok(result) if provider_status_error(&result).is_none() => { + // A settled cancel closes the record; a later sweep skips it. + if run_result_is_terminal(&result) { + let now = crate::auth::now_ts(); + let settled = run_id.clone(); + let _ = update_run_index(data_dir, now, |index| { + if let Some(entry) = index.runs.get_mut(&settled) { + entry.terminal_at = Some(now); + } + Ok(()) + }); + } + cancelled.push(run_id) + } Ok(result) => tracing::info!( "revoke cancel for {run_id} returned {}", provider_status_error(&result).unwrap_or_default() diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 03097415c..7a535bd9c 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -3,7 +3,7 @@ use super::home_system::{ }; use super::*; use crate::api::gateway::gateway_model_service::{ - cancel_grant_runs, model_grant_id, remote_principal_id, + cancel_grant_runs, model_grant_id, remote_principal_id, settle_denied_grant, }; use crate::collaboration_contact_store::CollaborationContactStore; use crate::collaboration_discovery::*; @@ -786,3 +786,44 @@ mod consumer_path { fx.shutdown().await; } } + +#[tokio::test] +async fn a_denial_that_failed_to_reach_the_requester_still_cancels_open_runs() { + let fx = TwoRuntimes::start().await; + let open = fx.create_run("seed-req-open", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let open_run_id = run_id_for(&fx.remote_principal(), "seed-req-open"); + fx.write_request_record("denied"); + + // The Inbox handler passes the denial result through the sweep unchanged, + // so a gossip delivery error cannot skip the cancellation. + let delivery_failed: anyhow::Result = + Err(anyhow::anyhow!("service access request delivery failed")); + let result = settle_denied_grant( + Some(fx.registry.clone()), + fx.owner.path(), + &fx.grant_id, + delivery_failed, + ) + .await; + assert!(result.is_err(), "{result:?}"); + let cancel = fx.last_provider_call().await; + assert_eq!(cancel["op"], "runs_cancel"); + assert_eq!(cancel["run_id"], open_run_id); + assert_eq!(cancel["runtime_binding"]["grant_id"], fx.grant_id); + + // A delivered denial returns its message after the same sweep; nothing + // is left to cancel the second time. + let ops_before = fx.provider_ops().await; + let delivered = settle_denied_grant( + Some(fx.registry.clone()), + fx.owner.path(), + &fx.grant_id, + Ok("Denied service request from Seed.".to_string()), + ) + .await + .unwrap(); + assert_eq!(delivered, "Denied service request from Seed."); + assert_eq!(fx.provider_ops().await, ops_before); + fx.shutdown().await; +} From ac66c4042e8926fded3b5fc148afe51b74d8cd9c Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 11:31:05 -0500 Subject: [PATCH 316/535] fix(model): read consumer run routes written before tickets were stored The route's new `display_name` field had no default, so an installed consumer index from the previous build failed to parse and every `runs_create` on that Home returned "remote run index parse". Both new fields now default to empty, and a test parses a legacy index. Found on the installed lane: seed test root 5f31a934 with three routes from 9bef09ad. Verification: cargo test -p elastos-server --lib -- gateway_model_remote remote_model cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../elastos-server/src/api/gateway_model_remote.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index ea38e8847..9f6cd08ef 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -50,6 +50,7 @@ pub(crate) struct RemoteRunRoute { /// the grant expires or is denied; the granting Runtime still decides. #[serde(default)] pub connect_ticket: String, + #[serde(default)] pub display_name: String, pub offer_id: String, pub request_id: String, @@ -653,6 +654,17 @@ mod tests { created_at: 10, terminal_at: None, }; + // Routes written before tickets and names were stored still parse; an + // installed index must never block new runs. + let legacy: RemoteRunIndex = serde_json::from_value(json!({ + "schema": REMOTE_RUN_INDEX_SCHEMA, + "runs": { "run:sha256:old": { + "principal_id": "p1", "grant_id": "g", "peer_did": "did:key:z6Mkpeer", + "offer_id": "remote:g:qwen", "request_id": "req", "created_at": 1 } } + })) + .expect("legacy index parses"); + assert_eq!(legacy.runs["run:sha256:old"].connect_ticket, ""); + assert_eq!(legacy.runs["run:sha256:old"].display_name, ""); // A route written before tickets were stored cannot settle by itself. assert!(RemoteRunRoute { connect_ticket: String::new(), From 91619ebd5e66c49157cbbfd55bd0c8310578b118 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 11:40:14 -0500 Subject: [PATCH 317/535] fix(model): bound open run records by age on both indexes An open record whose settlement a Runtime never observed stayed forever. No offer runs longer than its provider-bounded runtime, so an open record now leaves after the same retention window measured from its creation. Observed on the installed lane: three records from earlier builds stayed open after their runs had long completed. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model (21) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_remote.rs | 7 ++++--- .../src/api/gateway_model_service.rs | 15 ++++++++++++--- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 9f6cd08ef..31b8eef4a 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -117,10 +117,11 @@ fn update_index( .lock() .map_err(|_| anyhow::anyhow!("remote run index unavailable"))?; let mut index = read_index(data_dir)?; + // Settled routes leave after the retention window; open routes leave + // after the same window from creation, since no run lasts that long. index.runs.retain(|_, route| { - route - .terminal_at - .is_none_or(|terminal_at| now.saturating_sub(terminal_at) <= REMOTE_RUN_RETENTION_SECS) + let anchor = route.terminal_at.unwrap_or(route.created_at); + now.saturating_sub(anchor) <= REMOTE_RUN_RETENTION_SECS }); let result = update(&mut index)?; anyhow::ensure!( diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index 5b25385a6..6de5a4de5 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -165,10 +165,13 @@ fn update_run_index( .lock() .map_err(|_| anyhow::anyhow!("model run index unavailable"))?; let mut index = read_run_index(data_dir)?; + // A settled record leaves after the retention window. An open record + // leaves after the same window from creation: no offer runs that long + // (the provider bounds runtime in seconds), so such a record is a run + // whose settlement this Runtime never observed. index.runs.retain(|_, record| { - record - .terminal_at - .is_none_or(|terminal_at| now.saturating_sub(terminal_at) <= MODEL_RUN_RETENTION_SECS) + let anchor = record.terminal_at.unwrap_or(record.created_at); + now.saturating_sub(anchor) <= MODEL_RUN_RETENTION_SECS }); let result = update(&mut index)?; anyhow::ensure!( @@ -816,6 +819,12 @@ mod tests { assert!(remote_model_run(dir.path(), "run:sha256:bbbb") .unwrap() .is_none()); + // The open record (created at 100) leaves once the same window has + // passed since its creation; it cannot still be running. + update_run_index(dir.path(), 100 + MODEL_RUN_RETENTION_SECS + 1, |_| Ok(())).unwrap(); + assert!(remote_model_runs_for_grant(dir.path(), "g") + .unwrap() + .is_empty()); } #[tokio::test] From 66a2367c0f0c8a35a53d4aae38976c5707f1187d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 12:08:14 -0500 Subject: [PATCH 318/535] fix(services): sweep a denied grant's runs only after the owner's denial is saved The previous repair ran the cancellation sweep on every denial result, including one that failed before anything was saved. The decision lookup is principal-scoped while the sweep selects runs from the shared index by grant id, so another Inbox principal who knew a request id could cancel the owner's runs, and a failed save could cancel work under a grant that stayed approved. The recorded decision is now a typed outcome: `Err` means nothing was saved, `Ok` carries the message and a separate delivery result. The Inbox deny action saves the principal-scoped denial, then sweeps, then reports the delivery result. Gateway-level tests cover another principal (no cancel, grant stays approved), a denial that could not be saved (no cancel), and a saved denial whose delivery failed (the open run is cancelled and a new create is refused). Verification: cargo test -p elastos-server --lib -- remote_model gateway_model (23) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../crates/elastos-server/src/api/gateway.rs | 4 +- .../src/api/gateway_home_system.rs | 57 ++++++-- .../elastos-server/src/api/gateway_inbox.rs | 27 +--- .../src/api/gateway_model_service.rs | 39 +++-- .../src/api/gateway_tests/remote_model.rs | 135 +++++++++++++----- 5 files changed, 181 insertions(+), 81 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway.rs b/elastos/crates/elastos-server/src/api/gateway.rs index 681eccd80..9c445ae50 100644 --- a/elastos/crates/elastos-server/src/api/gateway.rs +++ b/elastos/crates/elastos-server/src/api/gateway.rs @@ -80,9 +80,9 @@ mod gateway_marketplace; mod gateway_model_remote; #[path = "gateway_model_service.rs"] mod gateway_model_service; +pub(in crate::api) use gateway_model_service::deny_grant_and_settle as deny_model_grant_and_settle; pub(crate) use gateway_model_service::{ - invoke as invoke_remote_model, model_grant_id, - settle_denied_grant as settle_denied_model_grant, ModelServiceGrant, MODEL_GRANT_SCHEMA, + invoke as invoke_remote_model, model_grant_id, ModelServiceGrant, MODEL_GRANT_SCHEMA, MODEL_GRANT_SCOPE, MODEL_GRANT_TTL_SECS, MODEL_LOCAL_OFFER, MODEL_OPERATIONS, MODEL_SERVICE_KIND, MODEL_SERVICE_URI, }; diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index ff969fbfd..02970a4c6 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -4010,6 +4010,8 @@ pub(super) fn approve_home_service_access_request( home_services_mark_access_request(data_dir, context, discovery_service, request_id, "approved") } +/// Deny as this principal. `Err` means no denial was saved (unknown request +/// for this principal, invalid id, or a save failure); nothing may follow it. pub(super) fn deny_home_service_access_request( data_dir: &std::path::Path, context: &HomeLaunchTokenContext, @@ -4017,8 +4019,24 @@ pub(super) fn deny_home_service_access_request( &crate::collaboration_discovery_runtime::CollaborationDiscoveryService, >, request_id: &str, -) -> anyhow::Result { - home_services_mark_access_request(data_dir, context, discovery_service, request_id, "denied") +) -> anyhow::Result { + home_services_record_access_decision(data_dir, context, discovery_service, request_id, "denied") +} + +/// A service access decision this principal made and this Runtime saved. The +/// decision holds whether or not the requester could be notified; `delivery` +/// reports that second, separate step. +pub(super) struct HomeServiceAccessDecisionRecorded { + pub message: String, + pub delivery: anyhow::Result<()>, +} + +impl HomeServiceAccessDecisionRecorded { + /// The one-line result for a UI action: a delivery failure surfaces as the + /// error the caller always showed, after the decision itself is safe. + pub fn into_message(self) -> anyhow::Result { + self.delivery.map(|()| self.message) + } } fn home_services_mark_access_request( @@ -4030,6 +4048,22 @@ fn home_services_mark_access_request( request_id: &str, status: &str, ) -> anyhow::Result { + home_services_record_access_decision(data_dir, context, discovery_service, request_id, status)? + .into_message() +} + +/// Authorize (principal-scoped lookup), apply and save the decision, then try +/// to deliver it. `Err` means nothing was saved; `Ok` means the decision is +/// this Runtime's truth even if `delivery` failed. +fn home_services_record_access_decision( + data_dir: &std::path::Path, + context: &HomeLaunchTokenContext, + discovery_service: Option< + &crate::collaboration_discovery_runtime::CollaborationDiscoveryService, + >, + request_id: &str, + status: &str, +) -> anyhow::Result { let request_id = request_id.trim(); if !home_services_request_id_is_valid(request_id) { anyhow::bail!("service request id is invalid"); @@ -4083,7 +4117,7 @@ fn home_services_mark_access_request( state.updated_at = revision; // The provider owns revocation even when the requester cannot receive gossip. home_save_services_requests_state(data_dir, context, &state)?; - home_services_send_access_decision( + let delivery = home_services_send_access_decision( data_dir, context, discovery_service, @@ -4097,14 +4131,17 @@ fn home_services_mark_access_request( } else { err.context("service access request delivery failed") } - })?; + }); let requester = request.requester_display_name.clone(); - Ok(match status { - "approved" => format!( - "Approved service request from {requester}. A private scoped grant was sent to the requester." - ), - "denied" => format!("Denied service request from {requester}."), - _ => "Updated service request.".to_string(), + Ok(HomeServiceAccessDecisionRecorded { + message: match status { + "approved" => format!( + "Approved service request from {requester}. A private scoped grant was sent to the requester." + ), + "denied" => format!("Denied service request from {requester}."), + _ => "Updated service request.".to_string(), + }, + delivery, }) } diff --git a/elastos/crates/elastos-server/src/api/gateway_inbox.rs b/elastos/crates/elastos-server/src/api/gateway_inbox.rs index ed11fb3aa..eb0da4df4 100644 --- a/elastos/crates/elastos-server/src/api/gateway_inbox.rs +++ b/elastos/crates/elastos-server/src/api/gateway_inbox.rs @@ -379,29 +379,12 @@ async fn dispatch_inbox_action( .map_err(|err| anyhow::anyhow!(err))?; } if let Some(request_id) = action_id.strip_prefix("service-deny-request:") { - let discovery_service = state.collaboration_discovery_service.clone(); - let data_dir = data_dir.clone(); - let context = context.clone(); - let request_id = request_id.to_string(); - let grant_id = super::model_grant_id(&request_id); - let denied = tokio::task::spawn_blocking({ - let data_dir = data_dir.clone(); - move || { - deny_home_service_access_request( - &data_dir, - &context, - discovery_service.as_ref(), - &request_id, - ) - } - }) - .await - .map_err(|err| anyhow::anyhow!(err))?; - return super::settle_denied_model_grant( + return super::deny_model_grant_and_settle( state.provider_registry.clone(), - &data_dir, - &grant_id, - denied, + data_dir, + context, + state.collaboration_discovery_service.as_ref(), + request_id, ) .await; } diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index 6de5a4de5..e0fc7acb3 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -621,18 +621,39 @@ pub(crate) async fn invoke( json!({ "ok": true, "result": result }) } -/// The owner's denial has two effects: the recorded decision, which the -/// requester learns about through gossip, and the settlement of runs the -/// grant had opened on this Runtime. The second effect never waits for the -/// first. The denial result is returned unchanged after the sweep. -pub(crate) async fn settle_denied_grant( +/// Deny a service request as this principal and settle the runs its grant had +/// opened on this Runtime. Three steps, in this order: the principal-scoped +/// denial is saved (or nothing happens), the sweep cancels that grant's open +/// runs, and only then does the delivery result decide the message. A denial +/// by another principal or a failed save never reaches the sweep. +pub(in crate::api) async fn deny_grant_and_settle( registry: Option>, data_dir: &Path, - grant_id: &str, - denied: anyhow::Result, + context: &HomeLaunchTokenContext, + discovery_service: Option< + &crate::collaboration_discovery_runtime::CollaborationDiscoveryService, + >, + request_id: &str, ) -> anyhow::Result { + let recorded = { + let data_dir = data_dir.to_path_buf(); + let context = context.clone(); + let discovery_service = discovery_service.cloned(); + let request_id = request_id.to_string(); + tokio::task::spawn_blocking(move || { + super::gateway_home_system::deny_home_service_access_request( + &data_dir, + &context, + discovery_service.as_ref(), + &request_id, + ) + }) + .await + .map_err(|err| anyhow::anyhow!(err))?? + }; + let grant_id = model_grant_id(request_id); if let Some(registry) = registry { - match cancel_grant_runs(registry, data_dir, grant_id).await { + match cancel_grant_runs(registry, data_dir, &grant_id).await { Ok(cancelled) if !cancelled.is_empty() => tracing::info!( "revoked model grant {grant_id}: cancel requested for {} run(s)", cancelled.len() @@ -643,7 +664,7 @@ pub(crate) async fn settle_denied_grant( } } } - denied + recorded.into_message() } /// Cancel every open run created under a grant, after revoke or denial. diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 7a535bd9c..0d0d6c702 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -3,7 +3,7 @@ use super::home_system::{ }; use super::*; use crate::api::gateway::gateway_model_service::{ - cancel_grant_runs, model_grant_id, remote_principal_id, settle_denied_grant, + cancel_grant_runs, model_grant_id, remote_principal_id, }; use crate::collaboration_contact_store::CollaborationContactStore; use crate::collaboration_discovery::*; @@ -787,43 +787,102 @@ mod consumer_path { } } -#[tokio::test] -async fn a_denial_that_failed_to_reach_the_requester_still_cancels_open_runs() { - let fx = TwoRuntimes::start().await; - let open = fx.create_run("seed-req-open", "qwen-local").await; - assert_eq!(open["ok"], true, "{open}"); - let open_run_id = run_id_for(&fx.remote_principal(), "seed-req-open"); - fx.write_request_record("denied"); +// Denial authority: the sweep follows only a denial this principal made and +// this Runtime saved. Delivery to the requester is a separate step. +mod denial_authority { + use super::*; + use crate::api::gateway::gateway_model_service::deny_grant_and_settle; - // The Inbox handler passes the denial result through the sweep unchanged, - // so a gossip delivery error cannot skip the cancellation. - let delivery_failed: anyhow::Result = - Err(anyhow::anyhow!("service access request delivery failed")); - let result = settle_denied_grant( - Some(fx.registry.clone()), - fx.owner.path(), - &fx.grant_id, - delivery_failed, - ) - .await; - assert!(result.is_err(), "{result:?}"); - let cancel = fx.last_provider_call().await; - assert_eq!(cancel["op"], "runs_cancel"); - assert_eq!(cancel["run_id"], open_run_id); - assert_eq!(cancel["runtime_binding"]["grant_id"], fx.grant_id); + impl TwoRuntimes { + fn owner_context(&self) -> HomeLaunchTokenContext { + HomeLaunchTokenContext { + principal_id: self.authority.principal_id.clone(), + session_id: self.authority.session_id.clone(), + proof_binding_id: Some(self.authority.proof_binding_id.clone()), + grant_id: self.authority.grant_id.clone(), + } + } - // A delivered denial returns its message after the same sweep; nothing - // is left to cancel the second time. - let ops_before = fx.provider_ops().await; - let delivered = settle_denied_grant( - Some(fx.registry.clone()), - fx.owner.path(), - &fx.grant_id, - Ok("Denied service request from Seed.".to_string()), - ) - .await - .unwrap(); - assert_eq!(delivered, "Denied service request from Seed."); - assert_eq!(fx.provider_ops().await, ops_before); - fx.shutdown().await; + async fn deny_as(&self, context: &HomeLaunchTokenContext) -> anyhow::Result { + deny_grant_and_settle( + Some(self.registry.clone()), + self.owner.path(), + context, + None, + REQUEST_ID, + ) + .await + } + + fn requests_state_dir(&self) -> std::path::PathBuf { + let root = crate::auth::principal_localhost_root(&self.authority.principal_id); + elastos_common::localhost::rooted_localhost_fs_path( + self.owner.path(), + &format!("{root}/.AppData/ElastOS/Home/services-requests.json"), + ) + .unwrap() + .parent() + .unwrap() + .to_path_buf() + } + } + + #[tokio::test] + async fn another_principal_cannot_deny_or_cancel_the_owners_runs() { + let fx = TwoRuntimes::start().await; + let open = fx.create_run("seed-req-open", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let ops_before = fx.provider_ops().await; + let intruder = HomeLaunchTokenContext { + principal_id: "person:local:intruder".to_string(), + session_id: "auth:intruder".to_string(), + proof_binding_id: Some("proof:passkey:intruder".to_string()), + grant_id: "grant:intruder".to_string(), + }; + let result = fx.deny_as(&intruder).await; + assert!(result.is_err(), "{result:?}"); + assert_eq!(fx.provider_ops().await, ops_before, "no cancel may follow"); + assert_eq!( + fx.create_run("seed-req-still-open", "qwen-local").await["ok"], + true, + "the grant stays approved" + ); + fx.shutdown().await; + } + + #[cfg(unix)] + #[tokio::test] + async fn a_denial_that_could_not_be_saved_cancels_nothing() { + use std::os::unix::fs::PermissionsExt; + let fx = TwoRuntimes::start().await; + let open = fx.create_run("seed-req-open", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let ops_before = fx.provider_ops().await; + let state_dir = fx.requests_state_dir(); + let mode = std::fs::metadata(&state_dir).unwrap().permissions(); + std::fs::set_permissions(&state_dir, std::fs::Permissions::from_mode(0o500)).unwrap(); + let result = fx.deny_as(&fx.owner_context()).await; + std::fs::set_permissions(&state_dir, mode).unwrap(); + assert!(result.is_err(), "{result:?}"); + assert_eq!(fx.provider_ops().await, ops_before, "no cancel may follow"); + fx.shutdown().await; + } + + #[tokio::test] + async fn a_saved_denial_cancels_the_open_run_even_when_delivery_fails() { + let fx = TwoRuntimes::start().await; + let open = fx.create_run("seed-req-open", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let open_run_id = run_id_for(&fx.remote_principal(), "seed-req-open"); + // No discovery runtime: the decision cannot be delivered, yet it is saved. + let result = fx.deny_as(&fx.owner_context()).await; + assert!(result.is_err(), "delivery failure surfaces: {result:?}"); + let cancel = fx.last_provider_call().await; + assert_eq!(cancel["op"], "runs_cancel"); + assert_eq!(cancel["run_id"], open_run_id); + assert_eq!(cancel["runtime_binding"]["grant_id"], fx.grant_id); + let refused = fx.create_run("seed-req-after", "qwen-local").await; + assert_eq!(refused["code"], "denied", "{refused}"); + fx.shutdown().await; + } } From 08fd81f2d5fa52b53f42d6d9aaf0ef13ddc333dd Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 12:13:46 -0500 Subject: [PATCH 319/535] fix(model): hold a run slot atomically and recognise retried requests The previous capacity check counted records and released the index lock before dispatch, so two creates could both pass the last free slot; it also refused a retry of an already recorded request when the index was full. Both indexes now keep pending reservations beside their records. A create first recognises an existing record for the same grant, principal and request id (a retry after a lost reply consumes no slot; the provider's create is idempotent), otherwise it holds a slot under one lock and one index write. The slot is committed with the run's record or released when nothing was dispatched; a slot older than ten minutes returns to capacity. Tests: two concurrent creates at the last slot dispatch exactly one run and the other is refused with `rate_limited`; a recorded request retries through a full index without a new slot while an unrelated request is refused; the consumer's retry reuses its route. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model (25) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_remote.rs | 121 ++++++++++++--- .../src/api/gateway_model_service.rs | 143 +++++++++++++++--- .../src/api/gateway_tests/remote_model.rs | 124 ++++++++++++--- 3 files changed, 329 insertions(+), 59 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 31b8eef4a..55a21d56d 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -78,6 +78,21 @@ struct RemoteRunIndex { schema: String, #[serde(default)] runs: BTreeMap, + /// Slots held between a create's reservation and its route (key to + /// reservation time); they count toward capacity. + #[serde(default)] + pending: BTreeMap, +} + +const REMOTE_RUN_PENDING_TTL_SECS: u64 = 600; + +fn route_reservation_key(principal_id: &str, grant_id: &str, request_id: &str) -> String { + format!("{principal_id}:{grant_id}:{request_id}") +} + +enum RouteSlot { + Existing, + Reserved, } fn index_path(data_dir: &Path) -> PathBuf { @@ -103,6 +118,7 @@ fn read_index(data_dir: &Path) -> anyhow::Result { Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(RemoteRunIndex { schema: REMOTE_RUN_INDEX_SCHEMA.to_string(), runs: BTreeMap::new(), + pending: BTreeMap::new(), }), Err(err) => Err(err).context("remote run index read"), } @@ -123,9 +139,12 @@ fn update_index( let anchor = route.terminal_at.unwrap_or(route.created_at); now.saturating_sub(anchor) <= REMOTE_RUN_RETENTION_SECS }); + index + .pending + .retain(|_, reserved_at| now.saturating_sub(*reserved_at) <= REMOTE_RUN_PENDING_TTL_SECS); let result = update(&mut index)?; anyhow::ensure!( - index.runs.len() <= REMOTE_RUN_INDEX_MAX, + index.runs.len() + index.pending.len() <= REMOTE_RUN_INDEX_MAX, "remote run index is full" ); let path = index_path(data_dir); @@ -317,18 +336,62 @@ fn run_id_of(result: &Value) -> Option { use super::gateway_model_service::run_result_is_terminal as run_is_terminal; -/// Prune settled routes past retention and confirm one more run fits. -fn reserve_run_route(data_dir: &Path, now: u64) -> Result<(), RemoteRouteError> { +/// Hold one route slot for a create, or recognise a request that already has +/// its route (a retry after a lost reply). Check and hold share one lock and +/// one index write, so two creates cannot both take the last slot. +fn reserve_route_slot( + data_dir: &Path, + now: u64, + key: &str, + principal_id: &str, + grant_id: &str, + request_id: &str, +) -> Result { update_index(data_dir, now, |index| { + let existing = index.runs.values().any(|route| { + route.principal_id == principal_id + && route.grant_id == grant_id + && route.request_id == request_id + }); + if existing { + return Ok(RouteSlot::Existing); + } + if index.pending.contains_key(key) { + anyhow::bail!("remote run reservation is already held"); + } anyhow::ensure!( - index.runs.len() < REMOTE_RUN_INDEX_MAX, + index.runs.len() + index.pending.len() < REMOTE_RUN_INDEX_MAX, "remote run index is full" ); - Ok(()) + index.pending.insert(key.to_string(), now); + Ok(RouteSlot::Reserved) }) .map_err(|err| RemoteRouteError::Invalid(err.to_string())) } +fn commit_route_slot( + data_dir: &Path, + now: u64, + key: &str, + run_id: &str, + route: RemoteRunRoute, +) -> anyhow::Result<()> { + update_index(data_dir, now, |index| { + index.pending.remove(key); + index.runs.entry(run_id.to_string()).or_insert(route); + Ok(()) + }) +} + +fn release_route_slot(data_dir: &Path, now: u64, key: &str) { + if let Err(err) = update_index(data_dir, now, |index| { + index.pending.remove(key); + Ok(()) + }) { + tracing::warn!("remote run reservation {key} could not be released: {err}"); + } +} + fn rewrite_offer_ids(value: &mut Value, grant_id: &str) { match value { Value::Object(map) => { @@ -454,10 +517,34 @@ pub(crate) async fn route_run_operation( })?; let mut request = typed_request(normalized, context, capsule_id, grant_id); request["offer_id"] = Value::String(inner_offer_id.to_string()); - // Route capacity is reserved before the granting Runtime starts - // work, so a full index refuses here instead of orphaning a run. - reserve_run_route(data_dir, now)?; - let mut result = call_grant(®istry, grant, op, request).await?; + // A route slot is held before the granting Runtime starts work; a + // retry of a routed request needs no slot. A full index refuses here. + let request_id = normalized + .pointer("/runtime_binding/request_id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(); + let key = route_reservation_key(&context.principal_id, grant_id, &request_id); + let held_slot = matches!( + reserve_route_slot( + data_dir, + now, + &key, + &context.principal_id, + grant_id, + &request_id + )?, + RouteSlot::Reserved + ); + let mut result = match call_grant(®istry, grant, op, request).await { + Ok(result) => result, + Err(err) => { + if held_slot { + release_route_slot(data_dir, now, &key); + } + return Err(err); + } + }; if let Some(run_id) = run_id_of(&result) { let route = RemoteRunRoute { principal_id: context.principal_id.clone(), @@ -466,18 +553,14 @@ pub(crate) async fn route_run_operation( connect_ticket: grant.connect_ticket.clone(), display_name: grant.display_name.clone(), offer_id: offer_id.to_string(), - request_id: normalized - .pointer("/runtime_binding/request_id") - .and_then(Value::as_str) - .unwrap_or_default() - .to_string(), + request_id: request_id.clone(), created_at: now, terminal_at: run_is_terminal(&result).then_some(now), }; - if let Err(err) = update_index(data_dir, now, |index| { - index.runs.entry(run_id.clone()).or_insert(route); - Ok(()) - }) { + if let Err(err) = commit_route_slot(data_dir, now, &key, &run_id, route) { + if held_slot { + release_route_slot(data_dir, now, &key); + } // The granting Runtime runs work this Home cannot address; // ask it to settle that run before reporting the failure. let cancel = typed_request( @@ -496,6 +579,8 @@ pub(crate) async fn route_run_operation( } return Err(RemoteRouteError::Invalid(err.to_string())); } + } else if held_slot { + release_route_slot(data_dir, now, &key); } rewrite_offer_ids(&mut result, grant_id); Ok(Some(result)) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index e0fc7acb3..e72f03372 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -110,6 +110,28 @@ struct RemoteModelRunIndex { schema: String, #[serde(default)] runs: BTreeMap, + /// Slots held for runs between their reservation and their record: + /// reservation key to reservation time. They count toward capacity. + #[serde(default)] + pending: BTreeMap, +} + +/// A slot held longer than this belongs to a create that never committed or +/// released (a crash between dispatch and record); it returns to capacity. +const MODEL_RUN_PENDING_TTL_SECS: u64 = 600; + +/// One create attempt is identified by who asked for what: the grant, the +/// destination-owned principal and the request id. A retry of the same +/// request reuses its record and never consumes a second slot. +fn run_reservation_key(grant_id: &str, remote_principal_id: &str, request_id: &str) -> String { + format!("{grant_id}:{remote_principal_id}:{request_id}") +} + +enum RunSlot { + /// This request already has a run record; the provider's create is idempotent. + Existing, + /// A slot is held under the reservation key until commit or release. + Reserved, } fn run_index_path(data_dir: &Path) -> PathBuf { @@ -129,6 +151,7 @@ fn read_run_index(data_dir: &Path) -> anyhow::Result { return Ok(RemoteModelRunIndex { schema: MODEL_RUN_INDEX_SCHEMA.to_string(), runs: BTreeMap::new(), + pending: BTreeMap::new(), }); } Err(err) => return Err(err).context("model run index read"), @@ -173,26 +196,75 @@ fn update_run_index( let anchor = record.terminal_at.unwrap_or(record.created_at); now.saturating_sub(anchor) <= MODEL_RUN_RETENTION_SECS }); + index + .pending + .retain(|_, reserved_at| now.saturating_sub(*reserved_at) <= MODEL_RUN_PENDING_TTL_SECS); let result = update(&mut index)?; anyhow::ensure!( - index.runs.len() <= MODEL_RUN_INDEX_MAX, + index.runs.len() + index.pending.len() <= MODEL_RUN_INDEX_MAX, "model run index is full" ); write_run_index(data_dir, &index)?; Ok(result) } -/// Prune settled records past retention and confirm one more run fits. -fn reserve_run_record(data_dir: &Path, now: u64) -> anyhow::Result<()> { +/// Hold one slot for a create, or recognise that the request already has a +/// record. The check and the hold happen under the same lock and land in the +/// same index write, so two creates cannot both take the last slot. +fn reserve_run_slot( + data_dir: &Path, + now: u64, + key: &str, + grant_id: &str, + remote_principal_id: &str, + request_id: &str, +) -> anyhow::Result { update_run_index(data_dir, now, |index| { + let existing = index.runs.values().any(|record| { + record.grant_id == grant_id + && record.remote_principal_id == remote_principal_id + && record.request_id == request_id + }); + if existing { + return Ok(RunSlot::Existing); + } + if index.pending.contains_key(key) { + anyhow::bail!("model run reservation is already held"); + } anyhow::ensure!( - index.runs.len() < MODEL_RUN_INDEX_MAX, + index.runs.len() + index.pending.len() < MODEL_RUN_INDEX_MAX, "model run index is full" ); + index.pending.insert(key.to_string(), now); + Ok(RunSlot::Reserved) + }) +} + +/// Turn a held slot into the run's record (or keep an existing record). +fn commit_run_slot( + data_dir: &Path, + now: u64, + key: &str, + run_id: &str, + record: RemoteModelRunRecord, +) -> anyhow::Result<()> { + update_run_index(data_dir, now, |index| { + index.pending.remove(key); + index.runs.entry(run_id.to_string()).or_insert(record); Ok(()) }) } +/// Return a held slot to capacity after a create that dispatched nothing. +fn release_run_slot(data_dir: &Path, now: u64, key: &str) { + if let Err(err) = update_run_index(data_dir, now, |index| { + index.pending.remove(key); + Ok(()) + }) { + tracing::warn!("model run reservation {key} could not be released: {err}"); + } +} + /// Best-effort cancel of a run this Runtime dispatched but could not record. async fn cancel_unrecorded_run( registry: &ProviderRegistry, @@ -534,26 +606,54 @@ pub(crate) async fn invoke( "model offer is not shared through this grant", ); } - // Recording capacity is reserved before any model work starts, so a - // full index refuses the request instead of orphaning a dispatched run. - if let Err(err) = reserve_run_record(data_dir, now) { - tracing::info!("remote model run refused before dispatch: {err}"); - return denied( - "rate_limited", - "model run capacity on this Runtime is exhausted", - ); - } } + // A create holds its record slot before any model work starts; a retry of + // an already recorded request needs no slot. A full index refuses here. + let request_id = normalized + .pointer("/runtime_binding/request_id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(); + let reservation_key = run_reservation_key(grant_id, &context.principal_id, &request_id); + let held_slot = if operation == "runs_create" { + match reserve_run_slot( + data_dir, + now, + &reservation_key, + grant_id, + &context.principal_id, + &request_id, + ) { + Ok(RunSlot::Reserved) => true, + Ok(RunSlot::Existing) => false, + Err(err) => { + tracing::info!("remote model run refused before dispatch: {err}"); + return denied( + "rate_limited", + "model run capacity on this Runtime is exhausted", + ); + } + } + } else { + false + }; + let result = match registry.send_raw("model", &normalized).await { Ok(result) => result, Err(err) => { + if held_slot { + release_run_slot(data_dir, now, &reservation_key); + } let class = redact_provider_error(&err.to_string()); tracing::info!("remote model provider error ({class}): {err}"); return denied(class, "model provider rejected the operation"); } }; if let Some(message) = provider_status_error(&result) { + if held_slot { + release_run_slot(data_dir, now, &reservation_key); + } let class = redact_provider_error(&message); tracing::info!("remote model provider status error ({class}): {message}"); return denied(class, "model provider rejected the operation"); @@ -580,27 +680,26 @@ pub(crate) async fn invoke( .as_str() .unwrap_or_default() .to_string(), - request_id: normalized - .pointer("/runtime_binding/request_id") - .and_then(Value::as_str) - .unwrap_or_default() - .to_string(), + request_id: request_id.clone(), created_at: now, terminal_at: run_result_is_terminal(&result).then_some(now), }; - if let Err(err) = update_run_index(data_dir, now, |index| { - index.runs.entry(run_id.clone()).or_insert(record); - Ok(()) - }) { + if let Err(err) = commit_run_slot(data_dir, now, &reservation_key, &run_id, record) + { // The run exists on this Runtime but has no owner record; // settle it now rather than leave it running unaccounted. tracing::warn!("remote model run index write failed: {err}"); cancel_unrecorded_run(®istry, &context, capsule_id, &run_id).await; + if held_slot { + release_run_slot(data_dir, now, &reservation_key); + } return denied( "provider_failure", "model run could not be recorded on this Runtime", ); } + } else if held_slot { + release_run_slot(data_dir, now, &reservation_key); } result } diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 0d0d6c702..2c8db500c 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -622,29 +622,50 @@ async fn run_lifecycle_stays_with_the_creating_principal_through_denial_and_revo fx.shutdown().await; } -#[tokio::test] -async fn a_full_run_index_refuses_before_any_model_work_starts() { - let fx = TwoRuntimes::start().await; - // 512 open records: the destination's recording capacity is exhausted. - let mut runs = serde_json::Map::new(); - for n in 0..512 { - runs.insert( - format!("run:sha256:{n:064x}"), +impl TwoRuntimes { + /// Fill the owner's run index with `count` open records, optionally + /// including the record a specific earlier request would have left. + fn fill_run_index(&self, count: usize, recorded_request: Option<&str>) { + let mut runs = serde_json::Map::new(); + let record = |request_id: String| { json!({ - "grant_id": fx.grant_id, "source_endpoint_did": fx.seed_did, + "grant_id": self.grant_id, "source_endpoint_did": self.seed_did, "requester_principal_id": SEED_PRINCIPAL, - "remote_principal_id": fx.remote_principal(), "capsule_id": "assistant", - "offer_id": "qwen-local", "request_id": format!("open-{n}"), "created_at": now_ts(), - }), - ); - } - std::fs::write( - fx.owner.path().join("services-model-runs.json"), - serde_json::to_vec(&json!({ "schema": "elastos.services.model-runs/v1", "runs": runs })) + "remote_principal_id": self.remote_principal(), "capsule_id": "assistant", + "offer_id": "qwen-local", "request_id": request_id, "created_at": now_ts(), + }) + }; + for n in 0..count { + runs.insert(format!("run:sha256:{n:064x}"), record(format!("open-{n}"))); + } + if let Some(request_id) = recorded_request { + runs.insert( + run_id_for(&self.remote_principal(), request_id), + record(request_id.to_string()), + ); + } + std::fs::write( + self.owner.path().join("services-model-runs.json"), + serde_json::to_vec( + &json!({ "schema": "elastos.services.model-runs/v1", "runs": runs }), + ) .unwrap(), - ) - .unwrap(); + ) + .unwrap(); + } + + fn run_index_len(&self) -> usize { + let path = self.owner.path().join("services-model-runs.json"); + let index: Value = serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap(); + index["runs"].as_object().unwrap().len() + + index["pending"].as_object().map_or(0, |p| p.len()) + } +} +#[tokio::test] +async fn a_full_run_index_refuses_before_any_model_work_starts() { + let fx = TwoRuntimes::start().await; + fx.fill_run_index(512, None); let refused = fx.create_run("seed-req-full", "qwen-local").await; assert_eq!(refused["ok"], false, "{refused}"); assert_eq!(refused["code"], "rate_limited"); @@ -656,6 +677,54 @@ async fn a_full_run_index_refuses_before_any_model_work_starts() { fx.shutdown().await; } +#[tokio::test] +async fn two_creates_at_the_last_slot_dispatch_exactly_one_run() { + let fx = TwoRuntimes::start().await; + fx.fill_run_index(511, None); + let (a, b) = tokio::join!( + fx.create_run("seed-req-race-a", "qwen-local"), + fx.create_run("seed-req-race-b", "qwen-local"), + ); + let outcomes = [a["ok"] == true, b["ok"] == true]; + assert_eq!( + outcomes.iter().filter(|ok| **ok).count(), + 1, + "exactly one create wins the slot: {a} {b}" + ); + let refused = if a["ok"] == true { &b } else { &a }; + assert_eq!(refused["code"], "rate_limited", "{refused}"); + let dispatched = fx + .provider_ops() + .await + .iter() + .filter(|op| *op == "runs_create") + .count(); + assert_eq!(dispatched, 1, "the refused create dispatched nothing"); + assert_eq!(fx.run_index_len(), 512, "no slot stays held after commit"); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_recorded_request_retries_through_a_full_index_without_a_new_slot() { + let fx = TwoRuntimes::start().await; + // The reply to "seed-req-lost" was lost after its run was recorded, and + // the index has since filled up. + fx.fill_run_index(511, Some("seed-req-lost")); + assert_eq!(fx.run_index_len(), 512); + let retried = fx.create_run("seed-req-lost", "qwen-local").await; + assert_eq!(retried["ok"], true, "{retried}"); + assert_eq!( + retried["result"]["data"]["run_id"], + run_id_for(&fx.remote_principal(), "seed-req-lost") + ); + assert_eq!(fx.last_provider_call().await["op"], "runs_create"); + assert_eq!(fx.run_index_len(), 512, "the retry consumed no capacity"); + // An unrelated new request is still refused. + let refused = fx.create_run("seed-req-new", "qwen-local").await; + assert_eq!(refused["code"], "rate_limited", "{refused}"); + fx.shutdown().await; +} + // The seed gateway's own routing over Carrier: the path the Assistant uses. mod consumer_path { use super::*; @@ -738,6 +807,23 @@ mod consumer_path { assert_eq!(route.connect_ticket, grant.connect_ticket); assert!(route.terminal_at.is_none()); + // A retry of the same request (its reply was lost) reuses the route + // and the owner's record; both sides answer with the same run. + let retried = fx + .route(std::slice::from_ref(&grant), "runs_create", create.clone()) + .await + .unwrap() + .expect("remote route"); + assert_eq!(retried["data"]["run_id"], run_id); + let seed_index: Value = serde_json::from_slice( + &std::fs::read(fx.seed.path().join("services-model-remote-runs.json")).unwrap(), + ) + .unwrap(); + assert_eq!(seed_index["runs"].as_object().unwrap().len(), 1); + assert!(seed_index["pending"] + .as_object() + .is_none_or(|p| p.is_empty())); + // The owner denies the request: no active grant remains on the seed. fx.write_request_record("denied"); let refused = fx.route(&[], "runs_create", create).await; From 98b40a2b9582adfb77df82c8ca4e7a390b97b99c Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 12:18:19 -0500 Subject: [PATCH 320/535] feat(model): answer settlement reads from the owner's record after the journal prunes a run The provider journal keeps a run for its offer's retention; a consumer that reconnects later found only a provider error and showed "Run acceptance is unknown" for a run that had completed. The owning Runtime's record now keeps the run's operation and the terminal status it observed. When the provider no longer knows a run, `runs_get` answers with that status (or an honest `settlement_unknown` when no settlement was observed), marked `settlement_source: runtime_index` and `output_retained: false`, and `runs_events` answers an empty final page. Ownership checks still apply. Verification: cargo test -p elastos-server --lib -- remote_model gateway_model (26) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- .../src/api/gateway_model_service.rs | 126 ++++++++++++++++-- .../src/api/gateway_tests/remote_model.rs | 56 ++++++++ 2 files changed, 170 insertions(+), 12 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index e72f03372..028afde55 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -99,10 +99,51 @@ pub(crate) struct RemoteModelRunRecord { pub remote_principal_id: String, pub capsule_id: String, pub offer_id: String, + /// The typed operation of the run, kept so this Runtime can still answer + /// a settlement read after the provider journal has pruned the run. + #[serde(default)] + pub operation: String, pub request_id: String, pub created_at: u64, #[serde(default, skip_serializing_if = "Option::is_none")] pub terminal_at: Option, + /// The settled status this Runtime observed, durable beyond the journal. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub terminal_status: Option, +} + +impl RemoteModelRunRecord { + /// The durable settlement reply for a run the provider no longer knows: + /// the observed terminal status, or `settlement_unknown` when this + /// Runtime never saw the run settle. Output is not retained here. + fn settlement_view(&self, run_id: &str) -> Value { + let status = self + .terminal_status + .clone() + .unwrap_or_else(|| "settlement_unknown".to_string()); + json!({ "status": "ok", "data": { + "schema": "elastos.model.run-view/v1", + "run_id": run_id, + "offer_id": self.offer_id, + "operation": self.operation, + "status": status, + "sequence_cursor": 0, + "terminal": { "status": status }, + "settlement_source": "runtime_index", + "output_retained": false, + } }) + } + + fn settlement_events_page(&self, run_id: &str, after_sequence: u64) -> Value { + json!({ "status": "ok", "data": { + "schema": "elastos.model.run-events/v1", + "run_id": run_id, + "next_cursor": after_sequence, + "has_more": false, + "events": [], + "settlement_source": "runtime_index", + } }) + } } #[derive(Debug, Clone, Serialize, Deserialize, Default)] @@ -426,17 +467,33 @@ async fn read_authority( /// answers with an events page and no status; its terminal event carries /// `terminal: true`. Both indexes settle their records from this one reading. pub(super) fn run_result_is_terminal(result: &Value) -> bool { + run_result_terminal_status(result).is_some() +} + +const RUN_TERMINAL_STATUSES: [&str; 4] = ["completed", "failed", "cancelled", "settlement_unknown"]; + +/// The settled status a typed model reply reports, if any: the run view's +/// `status` or `terminal.status`, or the kind of the events page's terminal +/// event (`output` settles as `completed`). +pub(super) fn run_result_terminal_status(result: &Value) -> Option { let data = &result["data"]; - let view_terminal = data["status"].as_str().is_some_and(|status| { - matches!( - status, - "completed" | "failed" | "cancelled" | "settlement_unknown" - ) - }) || data["terminal"].is_object(); - let event_terminal = data["events"] - .as_array() - .is_some_and(|events| events.iter().any(|event| event["terminal"] == true)); - view_terminal || event_terminal + let terminal = |value: &Value| { + value + .as_str() + .filter(|status| RUN_TERMINAL_STATUSES.contains(status)) + .map(str::to_string) + }; + terminal(&data["status"]) + .or_else(|| terminal(&data["terminal"]["status"])) + .or_else(|| { + data["events"].as_array()?.iter().find_map(|event| { + (event["terminal"] == true).then(|| match event["kind"].as_str() { + Some("output") | None => "completed".to_string(), + Some(kind) => terminal(&Value::String(kind.to_string())) + .unwrap_or_else(|| "completed".to_string()), + }) + }) + }) } /// Serve one model operation for a remote consumer. Every path returns a @@ -654,6 +711,27 @@ pub(crate) async fn invoke( if held_slot { release_run_slot(data_dir, now, &reservation_key); } + // The provider journal keeps a run for a bounded time; this Runtime's + // record outlives it. A settlement read of a run the provider no + // longer knows is answered from the record, so the consumer either + // learns the observed terminal status or an honest `settlement_unknown`. + if let Some((run_id, record)) = indexed_run + .as_ref() + .filter(|_| matches!(operation, "runs_get" | "runs_events")) + { + tracing::info!( + "remote model run {run_id} settled from the Runtime record after provider error: {message}" + ); + let reply = if operation == "runs_get" { + record.settlement_view(run_id) + } else { + record.settlement_events_page( + run_id, + normalized["after_sequence"].as_u64().unwrap_or(0), + ) + }; + return json!({ "ok": true, "result": reply }); + } let class = redact_provider_error(&message); tracing::info!("remote model provider status error ({class}): {message}"); return denied(class, "model provider rejected the operation"); @@ -680,8 +758,13 @@ pub(crate) async fn invoke( .as_str() .unwrap_or_default() .to_string(), + operation: normalized["operation"] + .as_str() + .unwrap_or_default() + .to_string(), request_id: request_id.clone(), created_at: now, + terminal_status: run_result_terminal_status(&result), terminal_at: run_result_is_terminal(&result).then_some(now), }; if let Err(err) = commit_run_slot(data_dir, now, &reservation_key, &run_id, record) @@ -705,10 +788,13 @@ pub(crate) async fn invoke( } _ => { if let Some((run_id, record)) = indexed_run { - if record.terminal_at.is_none() && run_result_is_terminal(&result) { + if let Some(status) = + run_result_terminal_status(&result).filter(|_| record.terminal_at.is_none()) + { let _ = update_run_index(data_dir, now, |index| { if let Some(entry) = index.runs.get_mut(&run_id) { entry.terminal_at = Some(now); + entry.terminal_status = Some(status); } Ok(()) }); @@ -791,12 +877,13 @@ pub(crate) async fn cancel_grant_runs( match registry.send_raw("model", &normalized).await { Ok(result) if provider_status_error(&result).is_none() => { // A settled cancel closes the record; a later sweep skips it. - if run_result_is_terminal(&result) { + if let Some(status) = run_result_terminal_status(&result) { let now = crate::auth::now_ts(); let settled = run_id.clone(); let _ = update_run_index(data_dir, now, |index| { if let Some(entry) = index.runs.get_mut(&settled) { entry.terminal_at = Some(now); + entry.terminal_status = Some(status); } Ok(()) }); @@ -906,10 +993,25 @@ mod tests { remote_principal_id: remote_principal_id("did:key:z6Mksource", "p"), capsule_id: "assistant".into(), offer_id: "qwen".into(), + operation: "text.generate".into(), request_id: "req-1".into(), created_at: 100, terminal_at: None, + terminal_status: None, }; + // A record written before `operation` and `terminal_status` existed + // still parses, and a pruned run then reports `settlement_unknown`. + let legacy: RemoteModelRunRecord = serde_json::from_value(json!({ + "grant_id": "g", "source_endpoint_did": "did:key:z6Mksource", + "requester_principal_id": "p", "remote_principal_id": "remote:abc", + "capsule_id": "assistant", "offer_id": "qwen", "request_id": "req-0", + "created_at": 1, + })) + .expect("legacy record parses"); + assert_eq!( + legacy.settlement_view("run:sha256:old")["data"]["status"], + "settlement_unknown" + ); update_run_index(dir.path(), 100, |index| { index.runs.insert("run:sha256:aaaa".into(), record.clone()); index.runs.insert( diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 2c8db500c..50f5197ee 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -972,3 +972,59 @@ mod denial_authority { fx.shutdown().await; } } + +// The provider journal prunes a run after its retention; the owning Runtime's +// record outlives it and answers the consumer's settlement read. +#[tokio::test] +async fn a_pruned_run_settles_from_the_owners_record() { + let fx = TwoRuntimes::start().await; + // Run A settles through its events page; run B is never observed settling. + let a = fx.create_run("seed-req-a", "qwen-local").await; + assert_eq!(a["ok"], true, "{a}"); + let run_a = run_id_for(&fx.remote_principal(), "seed-req-a"); + for _ in 0..2 { + let page = fx + .run_operation("runs_events", &run_a, SEED_PRINCIPAL) + .await; + assert_eq!(page["ok"], true, "{page}"); + } + assert_eq!(fx.run_record(&run_a)["terminal_status"], "completed"); + let b = fx.create_run("seed-req-b", "qwen-local").await; + assert_eq!(b["ok"], true, "{b}"); + let run_b = run_id_for(&fx.remote_principal(), "seed-req-b"); + assert!(fx.run_record(&run_b)["terminal_status"].is_null()); + + // The provider forgets both runs (journal pruned). + fx.provider.run_owners.lock().unwrap().clear(); + + let settled = fx.run_operation("runs_get", &run_a, SEED_PRINCIPAL).await; + assert_eq!(settled["ok"], true, "{settled}"); + assert_eq!(settled["result"]["data"]["status"], "completed"); + assert_eq!(settled["result"]["data"]["terminal"]["status"], "completed"); + assert_eq!(settled["result"]["data"]["output_retained"], false); + assert_eq!( + settled["result"]["data"]["settlement_source"], + "runtime_index" + ); + let page = fx + .run_operation("runs_events", &run_a, SEED_PRINCIPAL) + .await; + assert_eq!(page["ok"], true, "{page}"); + assert_eq!(page["result"]["data"]["has_more"], false); + assert_eq!(page["result"]["data"]["events"], json!([])); + + let unknown = fx.run_operation("runs_get", &run_b, SEED_PRINCIPAL).await; + assert_eq!(unknown["ok"], true, "{unknown}"); + assert_eq!(unknown["result"]["data"]["status"], "settlement_unknown"); + assert_eq!( + unknown["result"]["data"]["terminal"]["status"], + "settlement_unknown" + ); + + // Another principal still learns nothing about either run. + let foreign = fx + .run_operation("runs_get", &run_a, OTHER_SEED_PRINCIPAL) + .await; + assert_eq!(foreign["code"], "denied", "{foreign}"); + fx.shutdown().await; +} From 6dd1790f4bb8454258a156bda14e95340496d6da Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 14:13:55 -0500 Subject: [PATCH 321/535] fix(services): name an expired grant on the Others card and let the user ask again The seed card kept reading Approved after the six-hour model grant expired, and selecting it again did nothing because the offer was already subscribed. The card now shows Expired and Ask to use sends a fresh request. The unused access-grants summary field is not part of this change. Verification: cargo test -p elastos-server --lib -- an_approved_model_grant_past_its_expiry remote_model gateway_model (27) cargo fmt --all -- --check; node scripts/home-entropy-check.mjs; git diff --check --- capsules/services/browser/services.js | 19 ++++++- .../src/api/gateway_home_system.rs | 50 ++++++++++++++++++- scripts/home-entropy-check.mjs | 1 + 3 files changed, 67 insertions(+), 3 deletions(-) diff --git a/capsules/services/browser/services.js b/capsules/services/browser/services.js index 36ffc4bbf..a099ea74a 100644 --- a/capsules/services/browser/services.js +++ b/capsules/services/browser/services.js @@ -196,6 +196,10 @@ function renderServiceCard(offer, source, selected) { const statusTone = serviceStatusTone(offer, source); const primaryAction = serviceActionLabel(source, selected, offer); const pending = pendingServiceAction?.offerId === offerId && pendingServiceAction?.section === source; + const expired = grantRequired && serviceRequestStatus(offer) === "expired"; + // An expired grant stays in the subscribed list. Asking again posts + // selected=true so the Runtime sends a fresh request. + const nextSelected = expired ? "true" : selected ? "false" : "true"; return `
@@ -208,7 +212,7 @@ function renderServiceCard(offer, source, selected) {

${escapeHtml(copy)}

- ${offerId && !readOnly ? `` : ""} + ${offerId && !readOnly ? `` : ""} ${readOnly ? 'Managed by config' : ""}
@@ -356,6 +360,9 @@ function serviceCopy(offer, source, selected) { if (selected && requestStatus === "approved") { return `${name} was approved. ${copy.consumer} can use it when access becomes active.`; } + if (selected && requestStatus === "expired") { + return `${name} approval expired. Ask again to renew access.`; + } if (selected && requestStatus === "denied") { return `${name} denied the request. Remove it and ask again if needed.`; } @@ -384,6 +391,9 @@ function serviceStatus(offer, source, selected) { if (requestStatus === "approved") { return "Approved"; } + if (requestStatus === "expired") { + return "Expired"; + } if (requestStatus === "denied") { return "Denied"; } @@ -396,6 +406,9 @@ function serviceStatus(offer, source, selected) { } function serviceActionLabel(source, selected, offer = null) { + if (selected && serviceRequestStatus(offer) === "expired") { + return "Ask to use"; + } if (selected) { return source === "others" ? "Remove" : "Stop sharing"; } @@ -407,7 +420,9 @@ function serviceActionLabel(source, selected, offer = null) { function serviceRequestStatus(offer) { const status = readText(offer?.status); - return status === "approved" || status === "denied" ? status : "requested"; + return status === "approved" || status === "denied" || status === "expired" + ? status + : "requested"; } function orderedServiceOffers(offers) { diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index 02970a4c6..06c3a27b4 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -2104,7 +2104,14 @@ pub(super) async fn services_offer_update( ); } } else if req.selected { - if offer.grant_required && !services_state.remote_offer_ids.contains(offer_id) { + // A first request, or a fresh one after the earlier grant expired. + let expired_grant = services_state + .remote_offer_requests + .get(offer_id) + .is_some_and(|request| remote_offer_request_expired(request, now_ts())); + if offer.grant_required + && (!services_state.remote_offer_ids.contains(offer_id) || expired_grant) + { let sent = home_services_send_access_request( &data_dir, &context, @@ -4158,12 +4165,14 @@ fn apply_home_services_selection( partition_service_offers(local_offers, &state.local_offer_ids); let (mut remote_offers, available_remote_offers) = partition_service_offers(remote_offers, &state.remote_offer_ids); + let now = now_ts(); for offer in &mut remote_offers { if let Some(request) = state.remote_offer_requests.get(&offer.offer_id) { let installed = request.status == "approved" && request.installed_remote_exit_id.is_some(); offer.status = match request.status.as_str() { "approved" if installed => "active", + "approved" if remote_offer_request_expired(request, now) => "expired", "approved" => "approved", "denied" => "denied", _ => "requested", @@ -4189,6 +4198,21 @@ fn apply_home_services_selection( Ok(()) } +/// An approved request whose signed grant has passed its expiry. The card +/// says so, and asking again sends a fresh request instead of pretending the +/// old approval still works. +fn remote_offer_request_expired(request: &HomeServicesRemoteOfferRequestRecord, now: u64) -> bool { + if request.status != "approved" { + return false; + } + [&request.remote_model_grant, &request.remote_engine_grant] + .into_iter() + .flatten() + .filter_map(|grant| grant["expires_at"].as_u64()) + .next() + .is_some_and(|expires_at| expires_at <= now) +} + fn partition_service_offers( offers: Vec, selected_ids: &BTreeSet, @@ -7246,4 +7270,28 @@ mod services_kind_tests { let (noun, _) = home_services_request_notification_copy("remote_exit", "elastos://x"); assert_eq!(noun, "Browser Exit Node"); } + + #[test] + fn an_approved_model_grant_past_its_expiry_is_expired() { + let request = HomeServicesRemoteOfferRequestRecord { + status: "approved".into(), + remote_model_grant: Some(serde_json::json!({ "expires_at": 100 })), + ..Default::default() + }; + assert!(remote_offer_request_expired(&request, 100)); + assert!(remote_offer_request_expired(&request, 101)); + assert!(!remote_offer_request_expired(&request, 99)); + let live = HomeServicesRemoteOfferRequestRecord { + status: "approved".into(), + remote_model_grant: Some(serde_json::json!({ "expires_at": 200 })), + ..Default::default() + }; + assert!(!remote_offer_request_expired(&live, 100)); + let denied = HomeServicesRemoteOfferRequestRecord { + status: "denied".into(), + remote_model_grant: Some(serde_json::json!({ "expires_at": 50 })), + ..Default::default() + }; + assert!(!remote_offer_request_expired(&denied, 100)); + } } diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index f71a6d63f..0a1c3fa61 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -2763,6 +2763,7 @@ assert( servicesScript.includes("isReadOnlyServiceOffer") && servicesScript.includes("Managed by config") && servicesScript.includes("Approved") && + servicesScript.includes("Expired") && servicesScript.includes("Denied") && servicesScript.includes("serviceRequestStatus") && servicesScript.includes("Share with People") && From 5d75d866c41cd3d8da88d4c154b656d4dd31e2f8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 15:43:52 -0500 Subject: [PATCH 322/535] fix(services): settle retained model runs without a second dispatch Replay after journal cleanup answers from the owner record and a live journal read. Only a missing-run reply uses the status-only fallback. Assistant settles a terminal create or a retained events page at the client cursor. --- capsules/assistant/browser/agent-context.js | 1 + capsules/assistant/browser/agent-live.js | 71 ++- capsules/assistant/browser/agent-stream.js | 45 ++ capsules/assistant/browser/assistant.js | 19 + capsules/assistant/browser/model-contract.js | 12 + .../src/api/gateway_model_remote.rs | 54 +- .../src/api/gateway_model_service.rs | 333 ++++++++++- .../src/api/gateway_tests/remote_model.rs | 534 +++++++++++++++++- .../assistant-studio-preservation.test.mjs | 59 ++ scripts/home-agent-cancellation.test.mjs | 410 +++++++++++++- scripts/home-agent-shell-smoke.mjs | 13 + 11 files changed, 1471 insertions(+), 80 deletions(-) diff --git a/capsules/assistant/browser/agent-context.js b/capsules/assistant/browser/agent-context.js index b52ba4612..b4ec4875f 100644 --- a/capsules/assistant/browser/agent-context.js +++ b/capsules/assistant/browser/agent-context.js @@ -946,6 +946,7 @@ export function cheapTurnSnapshot(turn) { startedAt: Number(turn.startedAt) || undefined, completedAt: Number(turn.completedAt) || undefined, error: turn.error ? String(turn.error).slice(0, 120) : undefined, + ...(turn.outputRetained === false ? { outputRetained: false } : {}), }; } diff --git a/capsules/assistant/browser/agent-live.js b/capsules/assistant/browser/agent-live.js index c9764178a..c659e8b79 100644 --- a/capsules/assistant/browser/agent-live.js +++ b/capsules/assistant/browser/agent-live.js @@ -457,9 +457,6 @@ export async function streamChatViaContract( providerRunId: runId, state: TurnState.SUBMITTED, }); - patch({ state: activeRun === run ? TurnState.STREAMING : TurnState.SETTLEMENT_UNKNOWN, completedAt: null }); - if (activeRun === run) onAccepted?.({ run_id: runId, turnId: turn.turnId }); - if (run.cancelRequested && activeRun === run) void abortLiveChatStream(); let seq = 0; let streamedChars = 0; @@ -469,6 +466,7 @@ export async function streamChatViaContract( const next = extra.detached ? turn : patch({ state: extra.settlementUnknown ? TurnState.SETTLEMENT_UNKNOWN : extra.aborted ? TurnState.STOPPED : TurnState.COMPLETED, completedAt: Date.now(), + ...(extra.outputUnretained ? { outputRetained: false } : {}), }); return { usage: null, @@ -482,8 +480,48 @@ export async function streamChatViaContract( turnManifest: next, }; }; + const settleFromTerminal = (terminal) => { + if (!terminal || typeof terminal !== "object") { + return null; + } + if (terminal.status === "completed") { + if (terminal.outputRetained === false) { + return finish({ outputUnretained: true }); + } + const finalText = terminalOutputText(terminal.output); + if (finalText && streamedChars === 0) { + seq += 1; + emit("", finalText, false); + } + return finish(); + } + if (terminal.status === "cancelled") { + return finish({ aborted: true }); + } + if (terminal.status === "settlement_unknown") { + return finish({ settlementUnknown: true }); + } + patch({ state: TurnState.FAILED, completedAt: Date.now() }); + const detail = terminal.error && typeof terminal.error === "object" ? terminal.error : {}; + throw contractError( + String(detail.code || terminal.status || "run_failed"), + String(detail.message || `run ${terminal.status}`), + ); + }; + const createdTerminal = created.terminal && typeof created.terminal === "object" ? created.terminal : null; + if (createdTerminal && created.output_retained === false) { + createdTerminal.outputRetained = false; + } + if (!createdTerminal) { + patch({ state: activeRun === run ? TurnState.STREAMING : TurnState.SETTLEMENT_UNKNOWN, completedAt: null }); + } + if (activeRun === run) onAccepted?.({ run_id: runId, turnId: turn.turnId }); + if (!createdTerminal && run.cancelRequested && activeRun === run) void abortLiveChatStream(); try { + if (createdTerminal) { + return settleFromTerminal(createdTerminal); + } for (;;) { if (epoch !== liveStreamEpoch || activeRun !== run) { return finish({ detached: true }); @@ -517,29 +555,10 @@ export async function streamChatViaContract( if (epoch !== liveStreamEpoch || activeRun !== run) { return finish({ detached: true }); } - const terminal = applied.terminal || (!applied.hasMore && resuming ? created.terminal : null); - if (terminal) { - if (terminal.status === "completed") { - /* A provider that did not stream settles with the whole text once. */ - const finalText = terminalOutputText(terminal.output); - if (finalText && streamedChars === 0) { - seq += 1; - emit("", finalText, false); - } - return finish(); - } - if (terminal.status === "cancelled") { - return finish({ aborted: true }); - } - if (terminal.status === "settlement_unknown") { - return finish({ settlementUnknown: true }); - } - patch({ state: TurnState.FAILED, completedAt: Date.now() }); - const detail = terminal.error && typeof terminal.error === "object" ? terminal.error : {}; - throw contractError( - String(detail.code || terminal.status || "run_failed"), - String(detail.message || `run ${terminal.status}`), - ); + const terminal = applied.terminal || (!applied.hasMore ? created.terminal : null); + const settled = settleFromTerminal(terminal); + if (settled) { + return settled; } if (applied.hasMore) { continue; diff --git a/capsules/assistant/browser/agent-stream.js b/capsules/assistant/browser/agent-stream.js index b74290718..d3bdee53a 100644 --- a/capsules/assistant/browser/agent-stream.js +++ b/capsules/assistant/browser/agent-stream.js @@ -540,6 +540,9 @@ export function formatStreamError(err) { : "Model run failed"; } +export const OUTPUT_UNRETAINED_STATUS = + "The run completed. This Runtime did not keep the output."; + export function setStreamStatus(label, { tone = "idle" } = {}) { const el = document.querySelector("[data-agent-stream-status]"); if (!el) { @@ -586,6 +589,13 @@ function showRunSettlement(turn, detail = turn?.error === "run_not_found" el.append(" ", button); } +function showOutputUnretained(turn) { + if (turn?.outputRetained !== false) { + return; + } + setStreamStatus(OUTPUT_UNRETAINED_STATUS); +} + export function ensureJumpToLatest() { let btn = document.querySelector("[data-agent-jump-latest]"); if (btn) { @@ -891,6 +901,8 @@ export function renderActiveSession() { void startLiveTurnForPrompt("", { resumeTurn: session.lastTurn }); } else if (session.lastTurn?.state === TurnState.SETTLEMENT_UNKNOWN) { showRunSettlement(session.lastTurn); + } else if (session.lastTurn?.outputRetained === false) { + showOutputUnretained(session.lastTurn); } } @@ -2442,6 +2454,39 @@ async function startLiveTurnForPrompt(userText, { resumeTurn = null } = {}) { showRunSettlement(result.turnManifest); return; } + if (result.outputUnretained) { + const partial = materializeCanonical(); + syncCanonical(); + state.renderQueued = false; + renderAbort?.abort(); + renderAbort = null; + if (renderTimer) { + window.clearTimeout(renderTimer); + renderTimer = 0; + } + setPhase("finalizing"); + qos.ingestEvent({ type: "completed" }); + flushStreamingUI(true); + const answer = String(partial.answer || "").trim(); + const reasoning = String(partial.reasoning || "").trim(); + commitAgentReply(session, answer, reasoning, { turn: result.turnManifest }); + answerRow?.classList.remove("is-streaming"); + if (!answer) { + progressEl?.remove(); + answerRow?.remove(); + } + if (session && result.turnManifest) { + liveTurn = result.turnManifest; + session.lastTurn = cheapTurnSnapshot(liveTurn); + try { + host.persistAgentWorkspaceSoon?.(); + } catch { + /* optional */ + } + } + showOutputUnretained(result.turnManifest); + return; + } /* Unlock before joining the canonical string. */ ctx.turnBusy = false; setAgentComposerProcessing(false); diff --git a/capsules/assistant/browser/assistant.js b/capsules/assistant/browser/assistant.js index 30909f490..a46c05016 100644 --- a/capsules/assistant/browser/assistant.js +++ b/capsules/assistant/browser/assistant.js @@ -981,6 +981,17 @@ export function createAssistantApp({ state.studioProgress = null; } if (terminal.status === "completed") { + if (terminal.outputRetained === false) { + if (run.mode === MODE_STUDIO) { + state.studioResult = null; + } + state.statusMessage = "The run completed. This Runtime did not keep the output."; + if (run.mode !== MODE_STUDIO) { + markWorkspaceDirty(); + } + notify(); + return true; + } if (run.mode === MODE_STUDIO) { const output = parseStudioOutput(terminal.output); if (!output) { @@ -1026,6 +1037,7 @@ export function createAssistantApp({ status: runView.status, output: runView.terminal?.output ?? null, error: runView.terminal?.error ?? null, + outputRetained: runView.output_retained !== false, }); } @@ -1058,6 +1070,13 @@ export function createAssistantApp({ output: event.data ?? null, error: null, }; + } else if (event.kind === "completed") { + terminal = { + status: "completed", + output: event.data?.output_retained === false ? null : event.data ?? null, + error: null, + outputRetained: event.data?.output_retained !== false, + }; } else if (event.kind === "failed") { terminal = { status: "failed", diff --git a/capsules/assistant/browser/model-contract.js b/capsules/assistant/browser/model-contract.js index 92a8fe2a6..4bdcd0a3c 100644 --- a/capsules/assistant/browser/model-contract.js +++ b/capsules/assistant/browser/model-contract.js @@ -126,6 +126,14 @@ export function applyRunEventsPage(page, afterSequence) { } } else if (kind === "output") { terminal = { status: "completed", output: event.data ?? null, error: null }; + } else if (kind === "completed") { + const retained = event.data?.output_retained !== false; + terminal = { + status: "completed", + output: retained ? event.data ?? null : null, + error: null, + ...(retained ? {} : { outputRetained: false }), + }; } else if (kind === "failed" || kind === "cancelled" || kind === "settlement_unknown") { terminal = { status: kind, output: null, error: event.data ?? null }; } @@ -133,6 +141,10 @@ export function applyRunEventsPage(page, afterSequence) { terminal = { status: "completed", output: null, error: null }; } } + if (terminal && page.output_retained === false) { + terminal.outputRetained = false; + terminal.output = null; + } if (nextCursor < lastSequence) { throw contractError("bad_cursor", "run events cursor behind last sequence"); } diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 55a21d56d..40e2d5751 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -24,7 +24,7 @@ use super::gateway_home_token::HomeLaunchTokenContext; const REMOTE_OFFER_PREFIX: &str = "remote:"; const REMOTE_RUN_INDEX_SCHEMA: &str = "elastos.services.model-remote-runs/v1"; -const REMOTE_RUN_INDEX_MAX: usize = 1024; +pub(crate) const REMOTE_RUN_INDEX_MAX: usize = 1024; const REMOTE_RUN_RETENTION_SECS: u64 = 7 * 24 * 3600; const CARRIER_TIMEOUT_MS: u64 = 5_000; pub(crate) const TRANSPORT_INTERRUPTED: &str = "transport_interrupted"; @@ -54,6 +54,10 @@ pub(crate) struct RemoteRunRoute { pub display_name: String, pub offer_id: String, pub request_id: String, + /// Empty on routes written before capsule identity was stored; those + /// routes belong to Assistant. + #[serde(default)] + pub capsule_id: String, pub created_at: u64, #[serde(default, skip_serializing_if = "Option::is_none")] pub terminal_at: Option, @@ -86,8 +90,24 @@ struct RemoteRunIndex { const REMOTE_RUN_PENDING_TTL_SECS: u64 = 600; -fn route_reservation_key(principal_id: &str, grant_id: &str, request_id: &str) -> String { - format!("{principal_id}:{grant_id}:{request_id}") +fn route_capsule_id(capsule_id: &str) -> &str { + if capsule_id.is_empty() { + "assistant" + } else { + capsule_id + } +} + +fn route_reservation_key( + principal_id: &str, + grant_id: &str, + capsule_id: &str, + request_id: &str, +) -> String { + format!( + "{principal_id}:{grant_id}:{}:{request_id}", + route_capsule_id(capsule_id) + ) } enum RouteSlot { @@ -345,12 +365,15 @@ fn reserve_route_slot( key: &str, principal_id: &str, grant_id: &str, + capsule_id: &str, request_id: &str, ) -> Result { - update_index(data_dir, now, |index| { + let capsule_id = route_capsule_id(capsule_id); + match update_index(data_dir, now, |index| { let existing = index.runs.values().any(|route| { route.principal_id == principal_id && route.grant_id == grant_id + && route_capsule_id(&route.capsule_id) == capsule_id && route.request_id == request_id }); if existing { @@ -365,8 +388,19 @@ fn reserve_route_slot( ); index.pending.insert(key.to_string(), now); Ok(RouteSlot::Reserved) - }) - .map_err(|err| RemoteRouteError::Invalid(err.to_string())) + }) { + Ok(slot) => Ok(slot), + Err(err) => { + let message = err.to_string(); + if message.contains("remote run index is full") { + Err(RemoteRouteError::Rejected { + code: "rate_limited".into(), + }) + } else { + Err(RemoteRouteError::Invalid(message)) + } + } + } } fn commit_route_slot( @@ -524,7 +558,8 @@ pub(crate) async fn route_run_operation( .and_then(Value::as_str) .unwrap_or_default() .to_string(); - let key = route_reservation_key(&context.principal_id, grant_id, &request_id); + let key = + route_reservation_key(&context.principal_id, grant_id, capsule_id, &request_id); let held_slot = matches!( reserve_route_slot( data_dir, @@ -532,7 +567,8 @@ pub(crate) async fn route_run_operation( &key, &context.principal_id, grant_id, - &request_id + capsule_id, + &request_id, )?, RouteSlot::Reserved ); @@ -554,6 +590,7 @@ pub(crate) async fn route_run_operation( display_name: grant.display_name.clone(), offer_id: offer_id.to_string(), request_id: request_id.clone(), + capsule_id: capsule_id.to_string(), created_at: now, terminal_at: run_is_terminal(&result).then_some(now), }; @@ -737,6 +774,7 @@ mod tests { display_name: "Mac".into(), offer_id: "remote:g:qwen".into(), request_id: "req".into(), + capsule_id: "assistant".into(), created_at: 10, terminal_at: None, }; diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index 028afde55..d4b690fa8 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -104,6 +104,10 @@ pub(crate) struct RemoteModelRunRecord { #[serde(default)] pub operation: String, pub request_id: String, + /// Canonical hash of the create input. Empty on records written before + /// this field existed; those records still parse and still retry. + #[serde(default)] + pub input_hash: String, pub created_at: u64, #[serde(default, skip_serializing_if = "Option::is_none")] pub terminal_at: Option, @@ -135,13 +139,41 @@ impl RemoteModelRunRecord { } fn settlement_events_page(&self, run_id: &str, after_sequence: u64) -> Value { + let status = self + .terminal_status + .clone() + .unwrap_or_else(|| "settlement_unknown".to_string()); + let kind = match status.as_str() { + "completed" => "completed", + "failed" => "failed", + "cancelled" => "cancelled", + _ => "settlement_unknown", + }; + // An attached client has already consumed prepared/dispatched events. + // Emit the terminal at the next compatible sequence so a cursor of 2 + // or 7 still settles, and keep next_cursor equal to that sequence. + // Completed index settlement keeps the status and states that output + // is no longer retained; it does not send an empty output payload. + let sequence = after_sequence.saturating_add(1); + let event_data = if kind == "completed" { + json!({ "output_retained": false }) + } else { + json!({}) + }; json!({ "status": "ok", "data": { "schema": "elastos.model.run-events/v1", "run_id": run_id, - "next_cursor": after_sequence, + "next_cursor": sequence, "has_more": false, - "events": [], + "events": [json!({ + "schema": "elastos.model.run-event/v1", + "sequence": sequence, + "kind": kind, + "data": event_data, + "terminal": true, + })], "settlement_source": "runtime_index", + "output_retained": false, } }) } } @@ -162,15 +194,33 @@ struct RemoteModelRunIndex { const MODEL_RUN_PENDING_TTL_SECS: u64 = 600; /// One create attempt is identified by who asked for what: the grant, the -/// destination-owned principal and the request id. A retry of the same -/// request reuses its record and never consumes a second slot. -fn run_reservation_key(grant_id: &str, remote_principal_id: &str, request_id: &str) -> String { - format!("{grant_id}:{remote_principal_id}:{request_id}") +/// destination-owned principal, the consumer capsule and the request id. +/// A retry of that same request reuses its record and never consumes a +/// second slot. Another capsule with the same request id is a different run. +fn run_reservation_key( + grant_id: &str, + remote_principal_id: &str, + capsule_id: &str, + request_id: &str, +) -> String { + format!("{grant_id}:{remote_principal_id}:{capsule_id}:{request_id}") +} + +fn record_capsule_id(record: &RemoteModelRunRecord) -> &str { + if record.capsule_id.is_empty() { + "assistant" + } else { + &record.capsule_id + } } enum RunSlot { - /// This request already has a run record; the provider's create is idempotent. - Existing, + /// This request already has a run record. Answer from that record + /// and, while the run is still open, a live journal read. + Existing { + run_id: String, + record: RemoteModelRunRecord, + }, /// A slot is held under the reservation key until commit or release. Reserved, } @@ -258,16 +308,20 @@ fn reserve_run_slot( key: &str, grant_id: &str, remote_principal_id: &str, + capsule_id: &str, request_id: &str, ) -> anyhow::Result { update_run_index(data_dir, now, |index| { - let existing = index.runs.values().any(|record| { + if let Some((run_id, record)) = index.runs.iter().find(|(_, record)| { record.grant_id == grant_id && record.remote_principal_id == remote_principal_id + && record_capsule_id(record) == capsule_id && record.request_id == request_id - }); - if existing { - return Ok(RunSlot::Existing); + }) { + return Ok(RunSlot::Existing { + run_id: run_id.clone(), + record: record.clone(), + }); } if index.pending.contains_key(key) { anyhow::bail!("model run reservation is already held"); @@ -387,6 +441,166 @@ fn redact_provider_error(err: &str) -> &'static str { } } +fn provider_error_code(result: &Value) -> &str { + result.get("code").and_then(Value::as_str).unwrap_or("") +} + +fn mark_run_settled(data_dir: &Path, now: u64, run_id: &str, status: &str) { + let _ = update_run_index(data_dir, now, |index| { + if let Some(entry) = index.runs.get_mut(run_id) { + if entry.terminal_at.is_none() { + entry.terminal_at = Some(now); + entry.terminal_status = Some(status.to_string()); + } + } + Ok(()) + }); +} + +fn record_for_index_settlement( + data_dir: &Path, + now: u64, + run_id: &str, + record: &RemoteModelRunRecord, +) -> RemoteModelRunRecord { + if record.terminal_status.is_some() { + return record.clone(); + } + mark_run_settled(data_dir, now, run_id, "settlement_unknown"); + RemoteModelRunRecord { + terminal_at: Some(now), + terminal_status: Some("settlement_unknown".into()), + ..record.clone() + } +} + +fn index_settlement_reply( + data_dir: &Path, + now: u64, + operation: &str, + run_id: &str, + record: &RemoteModelRunRecord, + after_sequence: u64, +) -> Value { + let record = record_for_index_settlement(data_dir, now, run_id, record); + let reply = if operation == "runs_events" { + record.settlement_events_page(run_id, after_sequence) + } else { + record.settlement_view(run_id) + }; + json!({ "ok": true, "result": reply }) +} + +fn recover_missing_run_or_deny( + data_dir: &Path, + now: u64, + operation: &str, + run_id: &str, + record: &RemoteModelRunRecord, + result: &Value, + message: &str, + after_sequence: u64, +) -> Value { + if provider_error_code(result) == "run_not_found" { + tracing::info!( + "remote model run {run_id} settled from the Runtime record after the provider forgot it" + ); + return index_settlement_reply(data_dir, now, operation, run_id, record, after_sequence); + } + let class = redact_provider_error(message); + tracing::info!("remote model provider status error ({class}): {message}"); + denied(class, "model provider rejected the operation") +} + +fn request_conflicts_with_record(record: &RemoteModelRunRecord, normalized: &Value) -> bool { + let offer_id = normalized["offer_id"].as_str().unwrap_or(""); + let operation = normalized["operation"].as_str().unwrap_or(""); + let input_hash = normalized + .pointer("/runtime_binding/input_hash") + .and_then(Value::as_str) + .unwrap_or(""); + let capsule_id = normalized + .pointer("/runtime_binding/capsule_id") + .and_then(Value::as_str) + .unwrap_or(""); + (!record.capsule_id.is_empty() && record.capsule_id != capsule_id) + || (!record.offer_id.is_empty() && record.offer_id != offer_id) + || (!record.operation.is_empty() && record.operation != operation) + || (!record.input_hash.is_empty() && record.input_hash != input_hash) +} + +fn refuse_unbound_or_conflicting_create( + record: &RemoteModelRunRecord, + normalized: &Value, +) -> Option { + if request_conflicts_with_record(record, normalized) { + return Some(denied( + "denied", + "request_id conflicts with an existing model run", + )); + } + // A pre-repair record has no input hash. RunView does not carry the + // original fingerprint, so this Runtime cannot prove a replay. Refuse + // create here; owned get, events and cancel still use the record. + if record.input_hash.is_empty() { + return Some(denied( + "request_unbound", + "this request cannot be replayed because its original input was not retained", + )); + } + None +} + +async fn answer_retained_create( + registry: &ProviderRegistry, + data_dir: &Path, + now: u64, + context: &HomeLaunchTokenContext, + capsule_id: &str, + normalized: &Value, + request_id: &str, + run_id: String, + record: RemoteModelRunRecord, +) -> Value { + if let Some(refusal) = refuse_unbound_or_conflicting_create(&record, normalized) { + return refusal; + } + let get_request = json!({ + "run_id": run_id, + "request_id": format!("retry:{request_id}"), + }); + let get_normalized = + match normalize_model_provider_request("runs_get", &get_request, context, capsule_id) { + Ok(normalized) => normalized, + Err((_, message)) => { + tracing::info!("remote model retained get rejected: {message}"); + return denied( + "invalid_provider_invocation", + "model request did not match the typed contract", + ); + } + }; + let result = match registry.send_raw("model", &get_normalized).await { + Ok(result) => result, + Err(err) => { + let class = redact_provider_error(&err.to_string()); + tracing::info!("remote model provider error ({class}): {err}"); + return denied(class, "model provider rejected the operation"); + } + }; + if let Some(message) = provider_status_error(&result) { + return recover_missing_run_or_deny( + data_dir, now, "runs_get", &run_id, &record, &result, &message, 0, + ); + } + if let Some(status) = + run_result_terminal_status(&result).filter(|_| record.terminal_at.is_none()) + { + mark_run_settled(data_dir, now, &run_id, &status); + } + json!({ "ok": true, "result": result }) +} + fn provider_status_error(result: &Value) -> Option { (result.get("status").and_then(Value::as_str) == Some("error")).then(|| { result @@ -665,14 +879,16 @@ pub(crate) async fn invoke( } } - // A create holds its record slot before any model work starts; a retry of - // an already recorded request needs no slot. A full index refuses here. + // A create holds its record slot before any model work starts. A retry of + // an already recorded request is answered from that record before another + // create. A full index refuses here. let request_id = normalized .pointer("/runtime_binding/request_id") .and_then(Value::as_str) .unwrap_or_default() .to_string(); - let reservation_key = run_reservation_key(grant_id, &context.principal_id, &request_id); + let reservation_key = + run_reservation_key(grant_id, &context.principal_id, capsule_id, &request_id); let held_slot = if operation == "runs_create" { match reserve_run_slot( data_dir, @@ -680,10 +896,24 @@ pub(crate) async fn invoke( &reservation_key, grant_id, &context.principal_id, + capsule_id, &request_id, ) { Ok(RunSlot::Reserved) => true, - Ok(RunSlot::Existing) => false, + Ok(RunSlot::Existing { run_id, record }) => { + return answer_retained_create( + ®istry, + data_dir, + now, + &context, + capsule_id, + &normalized, + &request_id, + run_id, + record, + ) + .await; + } Err(err) => { tracing::info!("remote model run refused before dispatch: {err}"); return denied( @@ -712,25 +942,22 @@ pub(crate) async fn invoke( release_run_slot(data_dir, now, &reservation_key); } // The provider journal keeps a run for a bounded time; this Runtime's - // record outlives it. A settlement read of a run the provider no - // longer knows is answered from the record, so the consumer either - // learns the observed terminal status or an honest `settlement_unknown`. + // record outlives it. Only a missing-run reply uses that record. Other + // provider failures stay recoverable. if let Some((run_id, record)) = indexed_run .as_ref() .filter(|_| matches!(operation, "runs_get" | "runs_events")) { - tracing::info!( - "remote model run {run_id} settled from the Runtime record after provider error: {message}" + return recover_missing_run_or_deny( + data_dir, + now, + operation, + run_id, + record, + &result, + &message, + normalized["after_sequence"].as_u64().unwrap_or(0), ); - let reply = if operation == "runs_get" { - record.settlement_view(run_id) - } else { - record.settlement_events_page( - run_id, - normalized["after_sequence"].as_u64().unwrap_or(0), - ) - }; - return json!({ "ok": true, "result": reply }); } let class = redact_provider_error(&message); tracing::info!("remote model provider status error ({class}): {message}"); @@ -763,6 +990,11 @@ pub(crate) async fn invoke( .unwrap_or_default() .to_string(), request_id: request_id.clone(), + input_hash: normalized + .pointer("/runtime_binding/input_hash") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(), created_at: now, terminal_status: run_result_terminal_status(&result), terminal_at: run_result_is_terminal(&result).then_some(now), @@ -966,6 +1198,46 @@ mod tests { assert_eq!(filtered["data"]["offers"].as_array().unwrap().len(), 2); } + #[test] + fn settlement_events_page_emits_one_terminal_event() { + let record = RemoteModelRunRecord { + grant_id: "g".into(), + source_endpoint_did: "did:key:z6Mksource".into(), + requester_principal_id: "p".into(), + remote_principal_id: remote_principal_id("did:key:z6Mksource", "p"), + capsule_id: "assistant".into(), + offer_id: "qwen".into(), + operation: "text.generate".into(), + request_id: "req-1".into(), + input_hash: "sha256:aa".into(), + created_at: 100, + terminal_at: Some(150), + terminal_status: Some("completed".into()), + }; + let page = record.settlement_events_page("run:1", 0); + assert_eq!(page["data"]["events"][0]["kind"], "completed"); + assert_eq!(page["data"]["events"][0]["data"]["output_retained"], false); + assert_eq!(page["data"]["events"][0]["sequence"], 1); + assert_eq!(page["data"]["events"][0]["terminal"], true); + assert_eq!(page["data"]["next_cursor"], 1); + assert_eq!(page["data"]["output_retained"], false); + let mid = record.settlement_events_page("run:1", 2); + assert_eq!(mid["data"]["events"][0]["sequence"], 3); + assert_eq!(mid["data"]["next_cursor"], 3); + let high = record.settlement_events_page("run:1", 7); + assert_eq!(high["data"]["events"][0]["sequence"], 8); + assert_eq!(high["data"]["next_cursor"], 8); + let unknown = RemoteModelRunRecord { + terminal_at: None, + terminal_status: None, + ..record + }; + assert_eq!( + unknown.settlement_events_page("run:2", 0)["data"]["events"][0]["kind"], + "settlement_unknown" + ); + } + #[test] fn provider_errors_cross_carrier_only_as_bounded_classes() { assert_eq!( @@ -995,6 +1267,7 @@ mod tests { offer_id: "qwen".into(), operation: "text.generate".into(), request_id: "req-1".into(), + input_hash: String::new(), created_at: 100, terminal_at: None, terminal_status: None, diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 50f5197ee..ce3de9e42 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -11,7 +11,7 @@ use crate::collaboration_profile_authority::{ signed_profile_document_for_test, VerifiedCollaborationProfileDocument, }; use elastos_common::collaboration_protocol::*; -use elastos_model_contract::{RuntimeAccessBinding, RuntimeCreateBinding}; +use elastos_model_contract::{model_input_hash, RuntimeAccessBinding, RuntimeCreateBinding}; use elastos_runtime::signature::{generate_keypair, SigningKey}; use tokio::io::{AsyncBufReadExt as _, BufReader}; @@ -31,9 +31,15 @@ fn provider_error(err: impl std::fmt::Display) -> ProviderError { } fn run_id_for(principal_id: &str, request_id: &str) -> String { + run_id_for_capsule(principal_id, request_id, "assistant") +} + +fn run_id_for_capsule(principal_id: &str, request_id: &str, capsule_id: &str) -> String { format!( "run:sha256:{}", - hex::encode(Sha256::digest(format!("{principal_id}:{request_id}"))) + hex::encode(Sha256::digest(format!( + "{principal_id}:{capsule_id}:{request_id}" + ))) ) } @@ -48,7 +54,14 @@ fn run_view(run_id: &str, status: &str) -> Value { status, "completed" | "failed" | "cancelled" | "settlement_unknown" ) { - view["terminal"] = json!({ "status": status }); + let mut terminal = json!({ "status": status }); + if status == "completed" { + terminal["output"] = json!({ + "schema": "elastos.model.output.text/v1", + "text": "hello", + }); + } + view["terminal"] = terminal; } json!({ "status": "ok", "data": view }) } @@ -75,6 +88,9 @@ struct FakeModelProvider { calls: TokioMutex>, run_owners: std::sync::Mutex>, event_pages: std::sync::Mutex>, + run_status: std::sync::Mutex>, + /// Injected once onto the next `runs_get` or `runs_events` reply. + next_error: std::sync::Mutex>, } #[async_trait::async_trait] @@ -116,28 +132,73 @@ impl Provider for FakeModelProvider { binding .validate(offer_id, operation, &request["input"]) .map_err(provider_error)?; - let run_id = run_id_for(&binding.principal_id, &binding.request_id); + let run_id = run_id_for_capsule( + &binding.principal_id, + &binding.request_id, + &binding.capsule_id, + ); self.run_owners .lock() .unwrap() .insert(run_id.clone(), binding.principal_id); + self.run_status + .lock() + .unwrap() + .insert(run_id.clone(), "running".to_string()); return Ok(run_view(&run_id, "running")); } let binding: RuntimeAccessBinding = serde_json::from_value(request["runtime_binding"].clone()).map_err(provider_error)?; let run_id = request["run_id"].as_str().unwrap_or_default(); binding.validate(run_id).map_err(provider_error)?; + if matches!(op, "runs_get" | "runs_events") { + if let Some(code) = self.next_error.lock().unwrap().take() { + return Ok(json!({ "status": "error", "code": code, "message": code })); + } + } + if !self.run_owners.lock().unwrap().contains_key(run_id) { + return Ok(json!({ + "status": "error", + "code": "run_not_found", + "message": "model run is not available for the current caller", + })); + } if self.run_owners.lock().unwrap().get(run_id) != Some(&binding.principal_id) { - return Ok(json!({ "status": "error", "message": "run owner mismatch" })); + return Ok( + json!({ "status": "error", "code": "denied", "message": "run owner mismatch" }), + ); } Ok(match op { - "runs_get" => run_view(&binding.run_id, "running"), - "runs_cancel" => run_view(&binding.run_id, "cancelled"), + "runs_get" => { + let status = self + .run_status + .lock() + .unwrap() + .get(&binding.run_id) + .cloned() + .unwrap_or_else(|| "running".to_string()); + run_view(&binding.run_id, &status) + } + "runs_cancel" => { + self.run_status + .lock() + .unwrap() + .insert(binding.run_id.clone(), "cancelled".to_string()); + run_view(&binding.run_id, "cancelled") + } _ => { let mut pages = self.event_pages.lock().unwrap(); let page = pages.entry(binding.run_id.clone()).or_insert(0); *page += 1; - events_page(&binding.run_id, *page) + let page = *page; + drop(pages); + if page > 1 { + self.run_status + .lock() + .unwrap() + .insert(binding.run_id.clone(), "completed".to_string()); + } + events_page(&binding.run_id, page) } }) } @@ -453,18 +514,46 @@ impl TwoRuntimes { } async fn create_run(&self, request_id: &str, offer_id: &str) -> Value { + self.create_run_as( + request_id, + offer_id, + "assistant", + json!({ "prompt": "hello" }), + ) + .await + } + + async fn create_run_as( + &self, + request_id: &str, + offer_id: &str, + capsule_id: &str, + input: Value, + ) -> Value { self.call( "runs_create", json!({ "op": "runs_create", "offer_id": offer_id, "operation": "text.generate", - "input": { "prompt": "hello" }, "request_id": request_id, - "remote_model": self.remote(SEED_PRINCIPAL, "assistant"), + "input": input, "request_id": request_id, + "remote_model": self.remote(SEED_PRINCIPAL, capsule_id), "_runtime_invocation": { "schema": "elastos.provider.invocation/v1" }, }), ) .await } + async fn run_events(&self, run_id: &str, after_sequence: u64) -> Value { + self.call( + "runs_events", + json!({ + "op": "runs_events", "run_id": run_id, "after_sequence": after_sequence, + "request_id": format!("events:{after_sequence}"), + "remote_model": self.remote(SEED_PRINCIPAL, "assistant"), + }), + ) + .await + } + async fn run_operation(&self, op: &str, run_id: &str, principal_id: &str) -> Value { self.call( op, @@ -493,6 +582,14 @@ impl TwoRuntimes { self.provider.calls.lock().await.last().cloned().unwrap() } + async fn create_count(&self) -> usize { + self.provider_ops() + .await + .iter() + .filter(|op| *op == "runs_create") + .count() + } + async fn shutdown(mut self) { self.owner_service.shutdown().await.unwrap(); self.seed_node.shutdown().await; @@ -632,7 +729,10 @@ impl TwoRuntimes { "grant_id": self.grant_id, "source_endpoint_did": self.seed_did, "requester_principal_id": SEED_PRINCIPAL, "remote_principal_id": self.remote_principal(), "capsule_id": "assistant", - "offer_id": "qwen-local", "request_id": request_id, "created_at": now_ts(), + "offer_id": "qwen-local", "operation": "text.generate", + "request_id": request_id, + "input_hash": model_input_hash(&json!({ "prompt": "hello" })).unwrap(), + "created_at": now_ts(), }) }; for n in 0..count { @@ -654,6 +754,16 @@ impl TwoRuntimes { .unwrap(); } + fn strip_record_input_hash(&self, run_id: &str) { + let path = self.owner.path().join("services-model-runs.json"); + let mut index: Value = serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + index["runs"][run_id] + .as_object_mut() + .unwrap() + .remove("input_hash"); + std::fs::write(path, serde_json::to_vec(&index).unwrap()).unwrap(); + } + fn run_index_len(&self) -> usize { let path = self.owner.path().join("services-model-runs.json"); let index: Value = serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap(); @@ -717,7 +827,12 @@ async fn a_recorded_request_retries_through_a_full_index_without_a_new_slot() { retried["result"]["data"]["run_id"], run_id_for(&fx.remote_principal(), "seed-req-lost") ); - assert_eq!(fx.last_provider_call().await["op"], "runs_create"); + assert_eq!(fx.last_provider_call().await["op"], "runs_get"); + assert_eq!( + fx.create_count().await, + 0, + "a recorded retry must not create again" + ); assert_eq!(fx.run_index_len(), 512, "the retry consumed no capacity"); // An unrelated new request is still refused. let refused = fx.create_run("seed-req-new", "qwen-local").await; @@ -730,7 +845,7 @@ mod consumer_path { use super::*; use crate::api::gateway::gateway_model_remote::{ remote_offer_id, remote_run_route, route_run_operation, ConsumerModelGrant, - RemoteRouteError, + RemoteRouteError, REMOTE_RUN_INDEX_MAX, }; fn ticket_for(endpoint: &iroh::EndpointAddr) -> String { @@ -766,19 +881,69 @@ mod consumer_path { grants: &[ConsumerModelGrant], op: &str, normalized: Value, + ) -> Result, RemoteRouteError> { + self.route_as("assistant", grants, op, normalized).await + } + + async fn route_as( + &self, + capsule_id: &str, + grants: &[ConsumerModelGrant], + op: &str, + normalized: Value, ) -> Result, RemoteRouteError> { route_run_operation( self.seed_registry.clone(), self.seed.path(), grants, &seed_context(), - "assistant", + capsule_id, op, &normalized, now_ts(), ) .await } + + fn fill_consumer_run_index( + &self, + count: usize, + recorded_request: Option<&str>, + capsule_id: &str, + ) { + let mut runs = serde_json::Map::new(); + let record = |request_id: String| { + json!({ + "principal_id": SEED_PRINCIPAL, + "grant_id": self.grant_id, + "peer_did": "did:key:z6Mkpeer", + "connect_ticket": "ticket", + "display_name": "Mac", + "offer_id": remote_offer_id(&self.grant_id, "qwen-local"), + "request_id": request_id, + "capsule_id": capsule_id, + "created_at": now_ts(), + }) + }; + for n in 0..count { + runs.insert(format!("run:sha256:{n:064x}"), record(format!("open-{n}"))); + } + if let Some(request_id) = recorded_request { + runs.insert( + format!("run:sha256:{:064x}", 0), + record(request_id.to_string()), + ); + } + std::fs::write( + self.seed.path().join("services-model-remote-runs.json"), + serde_json::to_vec(&json!({ + "schema": "elastos.services.model-remote-runs/v1", + "runs": runs, + })) + .unwrap(), + ) + .unwrap(); + } } fn access(op: &str, run_id: &str, request_id: &str) -> Value { @@ -871,6 +1036,40 @@ mod consumer_path { assert!(local.is_none()); fx.shutdown().await; } + + #[tokio::test] + async fn a_full_consumer_index_refuses_a_second_capsule_before_any_model_work() { + let fx = TwoRuntimes::start().await; + let grant = fx.consumer_grant(); + fx.fill_consumer_run_index(REMOTE_RUN_INDEX_MAX, Some("seed-req-shared"), "assistant"); + let ops_before = fx.provider_ops().await; + let creates_before = fx.create_count().await; + let create = json!({ + "op": "runs_create", + "offer_id": remote_offer_id(&fx.grant_id, "qwen-local"), + "operation": "text.generate", + "input": { "prompt": "hello" }, + "runtime_binding": { + "principal_id": SEED_PRINCIPAL, + "request_id": "seed-req-shared", + }, + }); + let refused = fx + .route_as( + "home-agent", + std::slice::from_ref(&grant), + "runs_create", + create, + ) + .await; + assert!( + matches!(refused, Err(RemoteRouteError::Rejected { ref code }) if code == "rate_limited"), + "{refused:?}" + ); + assert_eq!(fx.provider_ops().await, ops_before); + assert_eq!(fx.create_count().await, creates_before); + fx.shutdown().await; + } } // Denial authority: the sweep follows only a denial this principal made and @@ -1011,7 +1210,14 @@ async fn a_pruned_run_settles_from_the_owners_record() { .await; assert_eq!(page["ok"], true, "{page}"); assert_eq!(page["result"]["data"]["has_more"], false); - assert_eq!(page["result"]["data"]["events"], json!([])); + let events = page["result"]["data"]["events"].as_array().unwrap(); + assert_eq!(events.len(), 1, "{page}"); + assert_eq!(events[0]["kind"], "completed"); + assert_eq!(events[0]["data"]["output_retained"], false); + assert_eq!(events[0]["terminal"], true); + assert_eq!(page["result"]["data"]["output_retained"], false); + assert_eq!(page["result"]["data"]["next_cursor"], 1); + assert_eq!(page["result"]["data"]["settlement_source"], "runtime_index"); let unknown = fx.run_operation("runs_get", &run_b, SEED_PRINCIPAL).await; assert_eq!(unknown["ok"], true, "{unknown}"); @@ -1020,6 +1226,10 @@ async fn a_pruned_run_settles_from_the_owners_record() { unknown["result"]["data"]["terminal"]["status"], "settlement_unknown" ); + assert_eq!( + fx.run_record(&run_b)["terminal_status"], + "settlement_unknown" + ); // Another principal still learns nothing about either run. let foreign = fx @@ -1028,3 +1238,297 @@ async fn a_pruned_run_settles_from_the_owners_record() { assert_eq!(foreign["code"], "denied", "{foreign}"); fx.shutdown().await; } + +#[tokio::test] +async fn a_completed_run_does_not_dispatch_again_after_the_journal_is_pruned() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-replay", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-replay"); + for _ in 0..2 { + let page = fx + .run_operation("runs_events", &run_id, SEED_PRINCIPAL) + .await; + assert_eq!(page["ok"], true, "{page}"); + } + assert_eq!(fx.run_record(&run_id)["terminal_status"], "completed"); + let creates_before = fx.create_count().await; + assert_eq!(creates_before, 1); + fx.provider.run_owners.lock().unwrap().clear(); + + let retried = fx.create_run("seed-req-replay", "qwen-local").await; + assert_eq!(retried["ok"], true, "{retried}"); + assert_eq!(retried["result"]["data"]["run_id"], run_id); + assert_eq!(retried["result"]["data"]["status"], "completed"); + assert_eq!( + retried["result"]["data"]["settlement_source"], + "runtime_index" + ); + assert_eq!( + fx.create_count().await, + creates_before, + "replay after journal cleanup must not dispatch again" + ); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_recorded_request_rejects_a_conflicting_offer_on_retry() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-bind", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-bind"); + let path = fx.owner.path().join("services-model-runs.json"); + let mut index: Value = serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + index["runs"][&run_id]["offer_id"] = json!("other-local"); + std::fs::write(&path, serde_json::to_vec(&index).unwrap()).unwrap(); + let retried = fx.create_run("seed-req-bind", "qwen-local").await; + assert_eq!(retried["ok"], false, "{retried}"); + assert_eq!(retried["code"], "denied", "{retried}"); + assert_eq!(fx.create_count().await, 1); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_provider_read_fault_stays_recoverable_instead_of_settling() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-fault", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-fault"); + for code in ["journal_corrupt", "internal_error", "not_initialized"] { + *fx.provider.next_error.lock().unwrap() = Some(code.to_string()); + let get = fx.run_operation("runs_get", &run_id, SEED_PRINCIPAL).await; + assert_eq!(get["ok"], false, "{code}: {get}"); + assert_eq!(get["code"], "provider_failure", "{code}: {get}"); + assert!( + fx.run_record(&run_id)["terminal_status"].is_null(), + "{code} must leave the owner record open" + ); + *fx.provider.next_error.lock().unwrap() = Some(code.to_string()); + let events = fx + .run_operation("runs_events", &run_id, SEED_PRINCIPAL) + .await; + assert_eq!(events["ok"], false, "{code}: {events}"); + assert_eq!(events["code"], "provider_failure", "{code}: {events}"); + } + fx.shutdown().await; +} +#[tokio::test] +async fn a_lost_create_reply_recovers_completed_output_from_the_journal() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-output", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-output"); + for _ in 0..2 { + let page = fx.run_events(&run_id, 0).await; + assert_eq!(page["ok"], true, "{page}"); + } + assert_eq!(fx.run_record(&run_id)["terminal_status"], "completed"); + let creates_before = fx.create_count().await; + assert_eq!(creates_before, 1); + + let retried = fx.create_run("seed-req-output", "qwen-local").await; + assert_eq!(retried["ok"], true, "{retried}"); + assert_eq!(retried["result"]["data"]["run_id"], run_id); + assert_eq!(retried["result"]["data"]["status"], "completed"); + assert_eq!( + retried["result"]["data"]["terminal"]["output"]["text"], + "hello" + ); + assert_ne!( + retried["result"]["data"]["settlement_source"], + "runtime_index" + ); + assert_eq!(fx.last_provider_call().await["op"], "runs_get"); + assert_eq!( + fx.create_count().await, + creates_before, + "a lost create reply must not dispatch again while the journal exists" + ); + fx.shutdown().await; +} + +#[tokio::test] +async fn an_attached_events_cursor_settles_after_the_journal_is_pruned() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-cursor", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-cursor"); + let first = fx.run_events(&run_id, 0).await; + assert_eq!(first["ok"], true, "{first}"); + assert_eq!(first["result"]["data"]["events"][1]["sequence"], 2); + fx.provider.run_owners.lock().unwrap().clear(); + + let settled = fx.run_events(&run_id, 2).await; + assert_eq!(settled["ok"], true, "{settled}"); + assert_eq!(settled["result"]["data"]["events"][0]["sequence"], 3); + assert_eq!(settled["result"]["data"]["events"][0]["terminal"], true); + assert_eq!(settled["result"]["data"]["next_cursor"], 3); + assert_eq!( + settled["result"]["data"]["settlement_source"], + "runtime_index" + ); + let high = fx.run_events(&run_id, 7).await; + assert_eq!(high["ok"], true, "{high}"); + assert_eq!(high["result"]["data"]["events"][0]["sequence"], 8); + assert_eq!(high["result"]["data"]["next_cursor"], 8); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_changed_input_is_rejected_before_and_after_journal_prune() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-input", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-input"); + assert!(!fx.run_record(&run_id)["input_hash"] + .as_str() + .unwrap() + .is_empty()); + let changed = fx + .create_run_as( + "seed-req-input", + "qwen-local", + "assistant", + json!({ "prompt": "other" }), + ) + .await; + assert_eq!(changed["ok"], false, "{changed}"); + assert_eq!(changed["code"], "denied", "{changed}"); + assert_eq!(fx.create_count().await, 1); + assert!(fx.run_record(&run_id)["terminal_status"].is_null()); + + fx.provider.run_owners.lock().unwrap().clear(); + let changed_again = fx + .create_run_as( + "seed-req-input", + "qwen-local", + "assistant", + json!({ "prompt": "other" }), + ) + .await; + assert_eq!(changed_again["ok"], false, "{changed_again}"); + assert_eq!(changed_again["code"], "denied", "{changed_again}"); + assert_eq!(fx.create_count().await, 1); + assert!(fx.run_record(&run_id)["terminal_status"].is_null()); + fx.shutdown().await; +} + +#[tokio::test] +async fn assistant_and_home_agent_keeps_separate_runs_for_the_same_request_id() { + let fx = TwoRuntimes::start().await; + let assistant = fx.create_run("seed-req-shared", "qwen-local").await; + let home_agent = fx + .create_run_as( + "seed-req-shared", + "qwen-local", + "home-agent", + json!({ "prompt": "hello" }), + ) + .await; + assert_eq!(assistant["ok"], true, "{assistant}"); + assert_eq!(home_agent["ok"], true, "{home_agent}"); + let assistant_id = run_id_for_capsule(&fx.remote_principal(), "seed-req-shared", "assistant"); + let home_id = run_id_for_capsule(&fx.remote_principal(), "seed-req-shared", "home-agent"); + assert_ne!(assistant_id, home_id); + assert_eq!(assistant["result"]["data"]["run_id"], assistant_id); + assert_eq!(home_agent["result"]["data"]["run_id"], home_id); + assert_eq!(fx.create_count().await, 2); + assert!(fx.run_record(&assistant_id)["terminal_status"].is_null()); + assert!(fx.run_record(&home_id)["terminal_status"].is_null()); + assert_eq!(fx.run_record(&assistant_id)["capsule_id"], "assistant"); + assert_eq!(fx.run_record(&home_id)["capsule_id"], "home-agent"); + + let result = crate::api::gateway::gateway_model_service::deny_grant_and_settle( + Some(fx.registry.clone()), + fx.owner.path(), + &HomeLaunchTokenContext { + principal_id: fx.authority.principal_id.clone(), + session_id: fx.authority.session_id.clone(), + proof_binding_id: Some(fx.authority.proof_binding_id.clone()), + grant_id: fx.authority.grant_id.clone(), + }, + None, + REQUEST_ID, + ) + .await; + assert!(result.is_err(), "delivery failure surfaces: {result:?}"); + let cancels = fx + .provider_ops() + .await + .iter() + .filter(|op| *op == "runs_cancel") + .count(); + assert_eq!(cancels, 2, "each capsule run must be cancelled"); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_legacy_record_refuses_changed_input_replay_while_the_journal_exists() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-legacy", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-legacy"); + fx.strip_record_input_hash(&run_id); + assert!(fx.run_record(&run_id).get("input_hash").is_none()); + + let changed = fx + .create_run_as( + "seed-req-legacy", + "qwen-local", + "assistant", + json!({ "prompt": "other" }), + ) + .await; + assert_eq!(changed["ok"], false, "{changed}"); + assert_eq!(changed["code"], "request_unbound", "{changed}"); + assert_eq!(fx.create_count().await, 1); + assert_eq!(fx.last_provider_call().await["op"], "offers_list"); + + let got = fx.run_operation("runs_get", &run_id, SEED_PRINCIPAL).await; + assert_eq!(got["ok"], true, "{got}"); + assert_eq!(got["result"]["data"]["status"], "running", "{got}"); + let page = fx.run_events(&run_id, 0).await; + assert_eq!(page["ok"], true, "{page}"); + let cancel = fx + .run_operation("runs_cancel", &run_id, SEED_PRINCIPAL) + .await; + assert_eq!(cancel["ok"], true, "{cancel}"); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_legacy_record_refuses_changed_input_replay_after_the_journal_is_pruned() { + let fx = TwoRuntimes::start().await; + let created = fx.create_run("seed-req-legacy-pruned", "qwen-local").await; + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-legacy-pruned"); + fx.strip_record_input_hash(&run_id); + fx.provider.run_owners.lock().unwrap().clear(); + + let changed = fx + .create_run_as( + "seed-req-legacy-pruned", + "qwen-local", + "assistant", + json!({ "prompt": "other" }), + ) + .await; + assert_eq!(changed["ok"], false, "{changed}"); + assert_eq!(changed["code"], "request_unbound", "{changed}"); + assert_eq!(fx.create_count().await, 1); + + let got = fx.run_operation("runs_get", &run_id, SEED_PRINCIPAL).await; + assert_eq!(got["ok"], true, "{got}"); + assert_eq!( + got["result"]["data"]["status"], "settlement_unknown", + "{got}" + ); + let page = fx.run_events(&run_id, 0).await; + assert_eq!(page["ok"], true, "{page}"); + assert_eq!( + page["result"]["data"]["events"][0]["kind"], + "settlement_unknown" + ); + fx.shutdown().await; +} diff --git a/scripts/assistant-studio-preservation.test.mjs b/scripts/assistant-studio-preservation.test.mjs index dbc8234fd..3026d1bf8 100644 --- a/scripts/assistant-studio-preservation.test.mjs +++ b/scripts/assistant-studio-preservation.test.mjs @@ -155,6 +155,65 @@ test("restored same-actor Studio run reads its exact cursor without another crea assert.deepEqual(f.app.snapshot().studioHistory[0].output, output); }); + +test("restored Studio run at cursor 7 settles from a retained missing-journal page", async t => { + const f = await fixture(t, { + studioState: { activeRun: restoredRun }, + readEvents: async body => { + assert.equal(body.after_sequence, 7); + return response({ status: "ok", data: { + schema: "elastos.model.run-events/v1", run_id: runId, next_cursor: 8, + has_more: false, settlement_source: "runtime_index", + events: [{ sequence: 8, kind: "settlement_unknown", terminal: true, data: {} }], + } }); + }, + }); + await flush(); + assert.deepEqual(f.operations().map(call => call.url), ["/api/provider/model/runs_events"]); + assert.equal(f.operations()[0].body.after_sequence, 7); + assert.equal(f.app.snapshot().activeRun.afterSequence, 8); + assert.equal(f.app.snapshot().activeRun.status, "settlement_unknown"); + assert.equal(f.app.snapshot().activeRun.terminal, true); + assert.equal(f.app.snapshot().activeRun.createRequestId, "original-request"); +}); + +test("a completed pruned journal states that Studio output was not retained", async t => { + const f = await fixture(t, { + studioState: { activeRun: restoredRun }, + readEvents: async body => { + assert.equal(body.after_sequence, 7); + return response({ status: "ok", data: { + schema: "elastos.model.run-events/v1", run_id: runId, next_cursor: 8, + has_more: false, settlement_source: "runtime_index", output_retained: false, + events: [{ sequence: 8, kind: "completed", terminal: true, data: { output_retained: false } }], + } }); + }, + }); + await flush(); + assert.deepEqual(f.operations().map(call => call.url), ["/api/provider/model/runs_events"]); + assert.equal(f.app.snapshot().activeRun.status, "completed"); + assert.equal(f.app.snapshot().activeRun.terminal, true); + assert.equal(f.app.snapshot().studioResult, null); + assert.equal(f.app.snapshot().statusMessage, "The run completed. This Runtime did not keep the output."); +}); + +test("a completed pruned create view states that Studio output was not retained", async t => { + const pending = defer(); + const started = defer(); + const f = await fixture(t, { studioState: { studioDraft: "Keep this draft" }, + create: body => { started.resolve(body); return pending.promise; } }); + const first = f.app.sendDraft(); + await started.promise; + pending.resolve(response({ status: "ok", data: { + schema: "elastos.model.run/v1", run_id: runId, status: "completed", sequence_cursor: 0, + terminal: { status: "completed" }, settlement_source: "runtime_index", output_retained: false, + } })); + assert.equal(await first, true); + assert.equal(f.app.snapshot().activeRun.status, "completed"); + assert.equal(f.app.snapshot().studioResult, null); + assert.equal(f.app.snapshot().statusMessage, "The run completed. This Runtime did not keep the output."); + assert.deepEqual(f.operations().map(call => call.url), ["/api/provider/model/runs_create"]); +}); for (const [label, binding] of [["foreign actor", { actorCapsule: "home-agent" }], ["history copy", { attachmentAllowed: false }]]) { test(`Studio ${label} retains history without events or cancellation`, async t => { diff --git a/scripts/home-agent-cancellation.test.mjs b/scripts/home-agent-cancellation.test.mjs index 12c350857..0042896fd 100644 --- a/scripts/home-agent-cancellation.test.mjs +++ b/scripts/home-agent-cancellation.test.mjs @@ -17,7 +17,7 @@ function fixtureWindow(location) { } globalThis.window = fixtureWindow({ href: "http://home.invalid/apps/assistant/", hash: "#home_token=fixture" }); const live = await import("../capsules/assistant/browser/agent-live.js"); -const { turnStoreGet } = await import("../capsules/assistant/browser/agent-context.js"); +const { turnStoreGet, cheapTurnSnapshot } = await import("../capsules/assistant/browser/agent-context.js"); const { recoverStalePersistedTurn } = await import("../capsules/assistant/browser/agent-stream-qos.js"); const controller = await import("../capsules/assistant/browser/agent-stream.js"); const workspace = await import("../capsules/assistant/browser/agent-workspace.js"); @@ -202,6 +202,168 @@ test("Runtime unknown settlement stays unknown rather than stopped", async () => assert.equal(recoverStalePersistedTurn({ lastTurn: result.turnManifest }).lastTurn.completedAt, result.turnManifest.completedAt); }); +test("an immediately failed create settles without polling events", async () => { + const calls = []; + globalThis.fetch = async (url) => { + const op = new URL(url).pathname.split("/").pop(); + calls.push(op); + if (op === "offers_list") return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + if (op === "runs_create") return { ok: true, json: async () => ({ + run_id: "run-full", sequence_cursor: 0, status: "failed", + terminal: { status: "failed", error: { code: "selection_unavailable", message: "model offer is not available" } }, + }) }; + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + await assert.rejects( + live.streamChatViaContract([{ role: "user", content: "fixture" }], { onAccepted() {}, onState() {} }), + (error) => error.code === "selection_unavailable", + ); + assert.deepEqual(calls.filter((op) => op.startsWith("runs_")), ["runs_create"]); +}); + +test("a retained unknown create settles from the create view without polling", async () => { + const calls = []; + globalThis.fetch = async (url) => { + const op = new URL(url).pathname.split("/").pop(); + calls.push(op); + if (op === "offers_list") return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + if (op === "runs_create") return { ok: true, json: async () => ({ + run_id: "run-pruned", sequence_cursor: 0, status: "settlement_unknown", + terminal: { status: "settlement_unknown" }, settlement_source: "runtime_index", + }) }; + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + const result = await live.streamChatViaContract([{ role: "user", content: "fixture" }], { onAccepted() {}, onState() {} }); + assert.equal(result.turnManifest.state, "settlement_unknown"); + assert.equal(result.turnManifest.providerRunId, "run-pruned"); + assert.ok(result.turnManifest.completedAt > 0); + assert.deepEqual(calls.filter((op) => op.startsWith("runs_")), ["runs_create"]); +}); + +test("reload of a retained settlement settles from get without polling events", async () => { + const calls = []; + globalThis.fetch = async (url) => { + const op = new URL(url).pathname.split("/").pop(); + calls.push(op); + if (op === "runs_get") return { ok: true, json: async () => ({ + run_id: "exact-run", status: "settlement_unknown", + terminal: { status: "settlement_unknown" }, settlement_source: "runtime_index", + }) }; + throw new Error(`unexpected operation ${op}`); + }; + const result = await live.streamChatViaContract([], { + turnManifest: { turnId: "saved", providerRunId: "exact-run", state: "settlement_unknown" }, + }); + assert.equal(result.turnManifest.state, "settlement_unknown"); + assert.ok(result.turnManifest.completedAt > 0); + assert.deepEqual(calls, ["runs_get"]); +}); + +test("an attached poll settles from a retained terminal event", async () => { + const f = await fixture(); + f.events.resolve({ events: [{ sequence: 1, kind: "settlement_unknown", terminal: true, data: {} }], next_cursor: 1, has_more: false }); + const result = await f.result; + assert.equal(result.turnManifest.state, "settlement_unknown"); + assert.ok(result.turnManifest.completedAt > 0); +}); + +test("an attached poll at cursor 7 settles from a retained terminal event", async () => { + const calls = []; + globalThis.fetch = async (url, init) => { + const op = new URL(url).pathname.split("/").pop(); + const body = init?.body ? JSON.parse(init.body) : null; + calls.push({ op, body }); + if (op === "offers_list") return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + if (op === "runs_create") return { ok: true, json: async () => ({ + run_id: "run-attached", sequence_cursor: 7, status: "running", + }) }; + if (op === "runs_events") { + assert.equal(body.after_sequence, 7); + return { ok: true, json: async () => ({ + schema: "elastos.model.run-events/v1", + run_id: "run-attached", + events: [{ sequence: 8, kind: "settlement_unknown", terminal: true, data: {} }], + next_cursor: 8, has_more: false, settlement_source: "runtime_index", + }) }; + } + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + const result = await live.streamChatViaContract([{ role: "user", content: "fixture" }], { onAccepted() {}, onState() {} }); + assert.equal(result.turnManifest.state, "settlement_unknown"); + assert.ok(result.turnManifest.completedAt > 0); + assert.equal(calls.filter((c) => c.op === "runs_events").length, 1); +}); + +test("a completed pruned journal states that live output was not retained", async () => { + const calls = []; + const deltas = []; + globalThis.fetch = async (url) => { + const op = new URL(url).pathname.split("/").pop(); + calls.push(op); + if (op === "offers_list") return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + if (op === "runs_create") return { ok: true, json: async () => ({ + run_id: "run-completed-pruned", sequence_cursor: 0, status: "completed", + terminal: { status: "completed" }, settlement_source: "runtime_index", + output_retained: false, + }) }; + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + const result = await live.streamChatViaContract([{ role: "user", content: "fixture" }], { + onAccepted() {}, onState() {}, onDelta: (delta) => deltas.push(delta), + }); + assert.equal(result.turnManifest.state, "completed"); + assert.equal(result.turnManifest.outputRetained, false); + assert.equal(deltas.length, 0); + assert.deepEqual(calls.filter((op) => op.startsWith("runs_")), ["runs_create"]); +}); + +test("an attached poll of a completed pruned journal states that output was not retained", async () => { + const calls = []; + const deltas = []; + globalThis.fetch = async (url, init) => { + const op = new URL(url).pathname.split("/").pop(); + const body = init?.body ? JSON.parse(init.body) : null; + calls.push({ op, body }); + if (op === "offers_list") return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + if (op === "runs_create") return { ok: true, json: async () => ({ + run_id: "run-completed-events", sequence_cursor: 2, status: "running", + }) }; + if (op === "runs_events") { + assert.equal(body.after_sequence, 2); + return { ok: true, json: async () => ({ + schema: "elastos.model.run-events/v1", + run_id: "run-completed-events", + events: [{ sequence: 3, kind: "completed", terminal: true, data: { output_retained: false } }], + next_cursor: 3, has_more: false, settlement_source: "runtime_index", + output_retained: false, + }) }; + } + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + const result = await live.streamChatViaContract([{ role: "user", content: "fixture" }], { + onAccepted() {}, onState() {}, onDelta: (delta) => deltas.push(delta), + }); + assert.equal(result.turnManifest.state, "completed"); + assert.equal(result.turnManifest.outputRetained, false); + assert.equal(deltas.length, 0); + assert.equal(calls.filter((c) => c.op === "runs_events").length, 1); +}); + test("navigation and reload retain the exact run without another create or cancel", async () => { const f = await fixture(); live.detachLiveChatStream(); @@ -344,6 +506,252 @@ function controllerFixture() { return { ctx, status }; } +function createDomNode(tag = "div") { + const node = { + tagName: String(tag).toUpperCase(), + className: "", + dataset: {}, + textContent: "", + hidden: false, + children: [], + style: {}, + isConnected: true, + classList: { + _names: new Set(), + add(...names) { names.forEach((name) => this._names.add(name)); node.className = [...this._names].join(" "); }, + remove(...names) { names.forEach((name) => this._names.delete(name)); node.className = [...this._names].join(" "); }, + toggle(name, force) { + if (force === true) this.add(name); + else if (force === false) this.remove(name); + else if (this._names.has(name)) this.remove(name); + else this.add(name); + }, + contains(name) { return this._names.has(name); }, + }, + append(...nodes) { + for (const child of nodes) { + if (child == null) continue; + const next = typeof child === "string" ? { textContent: child, children: [] } : child; + next.parent = this; + this.children.push(next); + } + }, + replaceChildren(...nodes) { this.children = []; this.append(...nodes); }, + remove() { + this.removed = true; + if (this.parent?.children) this.parent.children = this.parent.children.filter((child) => child !== this); + }, + querySelector(selector) { + const wanted = selector.startsWith(".") ? selector.slice(1) : ""; + const walk = (root) => { + for (const child of root.children || []) { + if (wanted && String(child.className || "").split(/\s+/).includes(wanted)) return child; + const found = walk(child); + if (found) return found; + } + return null; + }; + return walk(this); + }, + addEventListener() {}, + removeEventListener() {}, + setAttribute() {}, + scrollIntoView() {}, + contains() { return false; }, + querySelectorAll() { return []; }, + closest() { return null; }, + scrollHeight: 0, + scrollTop: 0, + clientHeight: 0, + }; + Object.defineProperty(node, "innerHTML", { + set(value) { this._html = String(value || ""); this.content = createDomNode("fragment"); }, + get() { return this._html || ""; }, + }); + return node; +} + +function streamControllerFixture() { + const { ctx, status } = controllerFixture(); + ctx.turnBusy = false; + ctx.sessions[0].messages = [{ role: "user", text: "fixture", modelText: "fixture" }]; + const stream = createDomNode("div"); + const viewport = createDomNode("div"); + const priorQuery = document.querySelector.bind(document); + document.createElement = (tag) => createDomNode(tag); + document.createTextNode = (value = "") => { + const text = String(value); + return { + data: text, + textContent: text, + nodeType: 3, + children: [], + appendData(extra) { + this.data += String(extra); + this.textContent = this.data; + }, + }; + }; + document.querySelector = (selector) => selector === "[data-agent-stream-status]" ? status : priorQuery(selector); + controller.bindAgentStream(ctx, { + streamEl: () => stream, + streamViewportEl: () => viewport, + streamScrollEl: () => stream, + titleEl: () => createDomNode("h1"), + clearEmptyState() {}, + scrollStreamToEnd() {}, + renderSessions() {}, + persistAgentWorkspaceSoon() {}, + }); + return { ctx, status, stream }; +} + +async function runUnretainedLiveTurn(t, { create, eventsPages = [] }) { + t.after(() => { live.detachLiveChatStream(); live.selectLiveOffer(""); }); + const { ctx, status, stream } = streamControllerFixture(); + controller.bindAgentStream(ctx, { + streamEl: () => stream, + streamViewportEl: () => createDomNode("div"), + streamScrollEl: () => stream, + titleEl: () => createDomNode("h1"), + clearEmptyState() {}, + scrollStreamToEnd() {}, + renderSessions() {}, + persistAgentWorkspaceSoon() {}, + }); + let offerLists = 0; + let eventPage = 0; + globalThis.fetch = async (url, init) => { + const op = new URL(url).pathname.split("/").pop(); + const body = init?.body ? JSON.parse(init.body) : null; + if (op === "offers_list") { + offerLists += 1; + return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + } + if (op === "runs_create") return { ok: true, json: async () => create }; + if (op === "runs_events") { + const page = eventsPages[Math.min(eventPage, Math.max(eventsPages.length - 1, 0))]; + eventPage += 1; + return { ok: true, json: async () => page.data }; + } + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + const offersAfterProbe = offerLists; + controller.startTurnForPrompt("fixture"); + const started = Date.now(); + while (Date.now() - started < 2000) { + if (ctx.sessions[0].lastTurn?.outputRetained === false) break; + await new Promise((resolve) => setTimeout(resolve, 20)); + } + if (ctx.sessions[0].lastTurn?.outputRetained !== false) { + throw new Error(`unretained turn missing: ${JSON.stringify(ctx.sessions[0].lastTurn)} status=${status.textContent}`); + } + await new Promise(setImmediate); + return { ctx, status, stream, offerLists, offersAfterProbe }; +} + +test("live chat states completed pruned output with no deltas and restores the warning", async t => { + const { ctx, status, offerLists, offersAfterProbe } = await runUnretainedLiveTurn(t, { + create: { + run_id: "run-completed-pruned", sequence_cursor: 0, status: "completed", + terminal: { status: "completed" }, settlement_source: "runtime_index", + output_retained: false, + }, + }); + assert.equal(status.textContent, controller.OUTPUT_UNRETAINED_STATUS); + assert.notEqual(status.textContent, "Model returned an empty reply"); + assert.equal(offerLists, offersAfterProbe, "completed pruned output must not force an inference probe"); + assert.equal(ctx.sessions[0].lastTurn.state, "completed"); + assert.equal(ctx.sessions[0].lastTurn.outputRetained, false); + assert.ok(!ctx.sessions[0].messages.some((message) => message.role === "agent")); + const snap = cheapTurnSnapshot(ctx.sessions[0].lastTurn); + assert.equal(snap.outputRetained, false); + assert.equal(cheapTurnSnapshot({ ...ctx.sessions[0].lastTurn, outputRetained: true }).outputRetained, undefined); + const saved = JSON.parse(JSON.stringify(workspace.serializeSessionForPersist(ctx.sessions[0]))); + assert.equal(saved.lastTurn.outputRetained, false); + ctx.sessions[0] = recoverStalePersistedTurn(saved); + status.textContent = ""; + controller.renderActiveSession(); + assert.equal(ctx.sessions[0].lastTurn.outputRetained, false); + assert.equal(status.textContent, controller.OUTPUT_UNRETAINED_STATUS); +}); + +test("live chat keeps partial deltas and restores completed-but-unavailable output", async t => { + const { ctx, status, offerLists, offersAfterProbe } = await runUnretainedLiveTurn(t, { + create: { run_id: "run-partial-pruned", sequence_cursor: 0, status: "running" }, + eventsPages: [ + { after: 0, data: { + schema: "elastos.model.run-events/v1", run_id: "run-partial-pruned", + events: [{ sequence: 1, kind: "text_delta", data: { text: "Partial prefix" } }], + next_cursor: 1, has_more: true, + } }, + { after: 1, data: { + schema: "elastos.model.run-events/v1", run_id: "run-partial-pruned", + events: [{ sequence: 2, kind: "completed", terminal: true, data: { output_retained: false } }], + next_cursor: 2, has_more: false, settlement_source: "runtime_index", + output_retained: false, + } }, + ], + }); + assert.equal(status.textContent, controller.OUTPUT_UNRETAINED_STATUS); + assert.equal(offerLists, offersAfterProbe); + assert.equal(ctx.sessions[0].lastTurn.state, "completed"); + assert.equal(ctx.sessions[0].lastTurn.outputRetained, false); + const agent = ctx.sessions[0].messages.filter((message) => message.role === "agent"); + assert.equal(agent.length, 1); + assert.equal(agent[0].text, "Partial prefix"); + assert.equal(agent[0].partial, undefined); + const saved = JSON.parse(JSON.stringify(workspace.serializeSessionForPersist(ctx.sessions[0]))); + assert.equal(saved.lastTurn.outputRetained, false); + assert.equal(saved.messages.at(-1).text, "Partial prefix"); + ctx.sessions[0] = recoverStalePersistedTurn(saved); + status.textContent = ""; + controller.renderActiveSession(); + assert.equal(ctx.sessions[0].messages.at(-1).text, "Partial prefix"); + assert.equal(status.textContent, controller.OUTPUT_UNRETAINED_STATUS); +}); + +function renderedAgentText(stream) { + const row = (stream.children || []).find((child) => child.dataset?.role === "agent"); + if (!row) return ""; + const parts = []; + const walk = (node) => { + if (!node || typeof node !== "object") return; + if (typeof node.data === "string") parts.push(node.data); + else if (String(node.className || "").includes("agent-stream-frozen-block") && node.textContent) { + parts.push(node.textContent); + } + for (const child of node.children || []) walk(child); + }; + walk(row); + return parts.join(""); +} + +test("live chat flushes two consecutive deltas before reopen of unretained output", async t => { + const { ctx, status, stream, offerLists, offersAfterProbe } = await runUnretainedLiveTurn(t, { + create: { run_id: "run-two-deltas", sequence_cursor: 0, status: "running" }, + eventsPages: [ + { after: 0, data: { + schema: "elastos.model.run-events/v1", run_id: "run-two-deltas", + events: [ + { sequence: 1, kind: "text_delta", data: { text: "Hel" } }, + { sequence: 2, kind: "text_delta", data: { text: "lo" } }, + { sequence: 3, kind: "completed", terminal: true, data: { output_retained: false } }, + ], + next_cursor: 3, has_more: false, settlement_source: "runtime_index", + output_retained: false, + } }, + ], + }); + assert.equal(status.textContent, controller.OUTPUT_UNRETAINED_STATUS); + assert.equal(offerLists, offersAfterProbe); + assert.equal(renderedAgentText(stream), "Hello", "both deltas must paint before reopen"); + assert.equal(ctx.sessions[0].messages.filter((message) => message.role === "agent").at(-1)?.text, "Hello"); +}); + test("refresh after a refused new turn requires another deliberate Send", async t => { t.after(() => live.selectLiveOffer("")); const { ctx } = controllerFixture(); ctx.turnBusy = false; diff --git a/scripts/home-agent-shell-smoke.mjs b/scripts/home-agent-shell-smoke.mjs index a2addc4e8..28fd9aa6d 100644 --- a/scripts/home-agent-shell-smoke.mjs +++ b/scripts/home-agent-shell-smoke.mjs @@ -387,6 +387,19 @@ assert.equal(page2.terminal.status, "completed"); assert.equal(contract.terminalOutputText(page2.terminal.output), "Hello"); assert.equal(contract.terminalOutputText({ schema: "other", text: "x" }), ""); +const prunedCompleted = contract.applyRunEventsPage( + { + events: [{ sequence: 3, kind: "completed", terminal: true, data: { output_retained: false } }], + next_cursor: 3, + has_more: false, + output_retained: false, + }, + 2, +); +assert.equal(prunedCompleted.terminal.status, "completed"); +assert.equal(prunedCompleted.terminal.outputRetained, false); +assert.equal(prunedCompleted.terminal.output, null); + const failed = contract.applyRunEventsPage( { events: [{ sequence: 1, kind: "failed", data: { code: "backend_down", message: "no backend" }, terminal: true }], next_cursor: 1 }, 0, From 50d835ed571aada25e71e7abc5d5e192b94abda8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 16:16:15 -0500 Subject: [PATCH 323/535] fix(services): let the owner revoke an approved model grant from Inbox An approved model grant appears as a read Inbox control with Revoke access. The existing deny route saves the denial and cancels open runs. Approved grants stay outside unread and pending counts. A 503 after a saved revoke still refreshes Inbox and keeps the delivery status. --- capsules/inbox/browser/index.html | 25 ++- .../src/api/gateway_home_system.rs | 48 ++++- .../src/api/gateway_tests/home_system.rs | 175 +++++++++++++++++- .../src/api/gateway_tests/remote_model.rs | 147 ++++++++++++++- scripts/inbox-product-behavior-smoke.mjs | 109 +++++++++++ 5 files changed, 495 insertions(+), 9 deletions(-) diff --git a/capsules/inbox/browser/index.html b/capsules/inbox/browser/index.html index 6e458adcc..9ea76b5ee 100644 --- a/capsules/inbox/browser/index.html +++ b/capsules/inbox/browser/index.html @@ -816,7 +816,9 @@

No requests

const response = await fetch(path, request); if (!response.ok) { const text = await response.text(); - throw new Error(publicInboxText(text, "Inbox could not complete the request.")); + const error = new Error(publicInboxText(text, "Inbox could not complete the request.")); + error.status = response.status; + throw error; } return response.json(); } @@ -930,10 +932,15 @@

No requests

renderInbox(); } + function pendingRequestEntries() { + return state.entries.filter((entry) => entry && entry.kind !== "service_access_grant"); + } + function renderInbox() { const entries = filteredEntries(); const attentionCount = state.entries.filter((entry) => entry && entry.severity === "attention").length; - elements.pendingCount.textContent = String(state.entries.length); + const pendingCount = pendingRequestEntries().length; + elements.pendingCount.textContent = String(pendingCount); elements.reviewCount.textContent = String(attentionCount); const requestedSelectionPresent = state.requestedSelectionId && entries.some((entry) => entryId(entry) === state.requestedSelectionId); @@ -968,7 +975,7 @@

No requests

state.filter === "review" && state.entries.length !== 0 ? "No requests need review" : "No requests"; - setStatus(state.entries.length === 0 ? "" : `${state.entries.length} pending.`); + setStatus(pendingCount === 0 ? "" : `${pendingCount} pending.`); notifyRailChrome(); } @@ -1052,6 +1059,8 @@

No requests

} else if (typeof actionId === "string" && actionId.startsWith("service-approve-request:")) { actions.appendChild(createActionButton("Approve", actionId, "primary")); actions.appendChild(createActionButton("Deny", "service-deny-request:" + actionId.slice("service-approve-request:".length), "danger")); + } else if (typeof actionId === "string" && actionId.startsWith("service-deny-request:")) { + actions.appendChild(createActionButton("Revoke access", actionId, "danger")); } else if (typeof actionId === "string" && actionId.startsWith("wallet-approve-request:")) { const requestId = actionId.slice("wallet-approve-request:".length); actions.appendChild(createButton("Approve", (button) => approveWalletRequest(button, requestId), "primary")); @@ -1074,7 +1083,7 @@

No requests

actions.appendChild(createActionButton("Reject", "wallet-price-http-deny:" + actionId.slice("wallet-price-http-approve:".length), "danger")); } - if (entry.kind !== "wallet_approval_request" && entry.kind !== "capability_request" && entry.kind !== "inspect_action_request" && entry.kind !== "external_http_request" && entry.kind !== "contact_request" && entry.kind !== "service_access_request") { + if (entry.kind !== "wallet_approval_request" && entry.kind !== "capability_request" && entry.kind !== "inspect_action_request" && entry.kind !== "external_http_request" && entry.kind !== "contact_request" && entry.kind !== "service_access_request" && entry.kind !== "service_access_grant") { actions.appendChild(createActionButton("Dismiss", "notification-dismiss:" + entry.id)); } } @@ -1151,6 +1160,14 @@

No requests

try { await inboxAction(actionId); await loadInbox(); + } catch (error) { + if (error && error.status === 503) { + try { + await loadInbox(); + } catch (_) { + } + } + throw error; } finally { setButtonsDisabled(false); } diff --git a/elastos/crates/elastos-server/src/api/gateway_home_system.rs b/elastos/crates/elastos-server/src/api/gateway_home_system.rs index 06c3a27b4..88eca6f39 100644 --- a/elastos/crates/elastos-server/src/api/gateway_home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_home_system.rs @@ -3962,19 +3962,33 @@ pub(super) fn append_home_service_access_notifications( let Ok(state) = home_services_requests_state(data_dir, context) else { return; }; - let mut requests = state + let now = now_ts(); + let mut pending = state .requests .values() .filter(|request| request.status == "pending") .cloned() .collect::>(); - requests.sort_by_key(|request| request.created_at); + pending.sort_by_key(|request| request.created_at); + let mut grants = state + .requests + .values() + .filter(|request| { + request.status == "approved" + && request.service_kind == super::MODEL_SERVICE_KIND + && request + .grant_expires_at + .is_some_and(|expires_at| expires_at > now) + }) + .cloned() + .collect::>(); + grants.sort_by_key(|request| request.updated_at); let existing_ids = notifications .entries .iter() .map(|entry| entry.id.clone()) .collect::>(); - for request in requests { + for request in pending { let id = format!("service-access-request:{}", request.request_id); if existing_ids.contains(&id) { continue; @@ -4004,6 +4018,34 @@ pub(super) fn append_home_service_access_notifications( created_at: request.created_at, }); } + for request in grants { + let id = format!("service-access-grant:{}", request.request_id); + if existing_ids.contains(&id) { + continue; + } + let (service_noun, _) = + home_services_request_notification_copy(&request.service_kind, &request.service_uri); + notifications.entries.push(HomeNotificationEntrySummary { + id, + source_app: SERVICES_CAPSULE_ID.to_string(), + kind: "service_access_grant".to_string(), + title: format!( + "{} may use your {service_noun}", + request.requester_display_name + ), + body: format!( + "{} may use {}. Revoke access to stop new runs and cancel open runs.", + request.requester_display_name, request.service_display_name + ), + action_ref: Some(HomeNotificationActionSummary { + app: SERVICES_CAPSULE_ID.to_string(), + action_id: format!("service-deny-request:{}", request.request_id), + }), + severity: "info".to_string(), + read: true, + created_at: request.updated_at, + }); + } } pub(super) fn approve_home_service_access_request( diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs b/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs index 5393ad2dc..d3a246095 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs @@ -26,7 +26,7 @@ impl Drop for EnvRestore { } } -async fn home_test_get_json( +pub(super) async fn home_test_get_json( app: &axum::Router, uri: &str, token: &str, @@ -8712,6 +8712,179 @@ async fn test_services_inbox_request_copy_matches_engine_or_exit_and_exact_appro } } +#[tokio::test] +async fn test_inbox_projects_approved_model_grant_outside_pending_counts() { + let dir = tempfile::tempdir().unwrap(); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + } + let authority = passkey_authority_with_profile(dir.path(), "Mac"); + let now = crate::auth::now_ts(); + let localhost_root = crate::auth::principal_localhost_root(&authority.principal_id); + write_home_principal_object_json_for_authority( + dir.path(), + &authority, + "services-requests.json", + json!({ + "schema": "elastos.services.requests/v1", + "principal_id": authority.principal_id, + "localhost_root": localhost_root, + "updated_at": now, + "requests": { + "pending-exit-1": { + "request_id": "pending-exit-1", + "offer_id": "local:provider:browser-exit", + "service_uri": "elastos://peer/browser-exit", + "service_kind": "remote_exit", + "service_display_name": "Mac Browser Exit", + "requester_peer_id": "peer-alice", + "requester_did": "did:key:alice", + "requester_principal_id": "person:local:alice", + "requester_display_name": "Alice", + "created_at": now, + "updated_at": now, + "status": "pending", + "authenticated_request": true + }, + "approved-model-1": { + "request_id": "approved-model-1", + "offer_id": "local:provider:model", + "service_uri": "elastos://peer/model", + "service_kind": "remote_model", + "service_display_name": "Mac model", + "requester_peer_id": "peer-seed", + "requester_did": "did:key:seed", + "requester_principal_id": "seed-principal-a", + "requester_display_name": "Seed", + "created_at": now, + "updated_at": now, + "status": "approved", + "authenticated_request": true, + "grant_expires_at": now + 3600 + }, + "expired-model-1": { + "request_id": "expired-model-1", + "offer_id": "local:provider:model", + "service_uri": "elastos://peer/model", + "service_kind": "remote_model", + "service_display_name": "Mac model", + "requester_peer_id": "peer-old", + "requester_did": "did:key:old", + "requester_principal_id": "seed-principal-old", + "requester_display_name": "Old", + "created_at": now - 10, + "updated_at": now - 10, + "status": "approved", + "authenticated_request": true, + "grant_expires_at": now - 1 + }, + "approved-engine-1": { + "request_id": "approved-engine-1", + "offer_id": "local:provider:browser-engine", + "service_uri": "elastos://peer/browser-engine", + "service_kind": "browser_engine", + "service_display_name": "Mac Browser Engine", + "requester_peer_id": "peer-engine", + "requester_did": "did:key:engine", + "requester_principal_id": "seed-principal-engine", + "requester_display_name": "Engine", + "created_at": now, + "updated_at": now, + "status": "approved", + "authenticated_request": true, + "grant_expires_at": now + 3600 + } + } + }), + ); + let app = gateway_router(test_state(dir.path())); + let (status, _) = home_test_post_json( + &app, + "/api/apps/home/launch", + &authority.home_token, + "http://localhost:61180", + json!({ "target": INBOX_CAPSULE_ID }), + ) + .await; + assert_eq!(status, StatusCode::OK); + let token = app_token_for_authority(dir.path(), INBOX_CAPSULE_ID, &authority); + let (status, inbox) = home_test_get_json(&app, "/api/apps/inbox/summary", &token, "null").await; + assert_eq!(status, StatusCode::OK, "{inbox}"); + let notifications = &inbox["notifications"]; + assert_eq!(notifications["unread_count"], 1, "{inbox}"); + assert_eq!(notifications["attention_count"], 1, "{inbox}"); + let entries = notifications["entries"].as_array().unwrap(); + let pending = entries + .iter() + .filter(|entry| entry["kind"] == "service_access_request") + .collect::>(); + let grants = entries + .iter() + .filter(|entry| entry["kind"] == "service_access_grant") + .collect::>(); + assert_eq!(pending.len(), 1, "{inbox}"); + assert_eq!(grants.len(), 1, "{inbox}"); + let grant = grants[0]; + assert_eq!(grant["id"], "service-access-grant:approved-model-1"); + assert_eq!(grant["source_app"], SERVICES_CAPSULE_ID); + assert_eq!(grant["title"], "Seed may use your AI model"); + assert_eq!( + grant["body"], + "Seed may use Mac model. Revoke access to stop new runs and cancel open runs." + ); + assert_eq!( + grant["action_ref"]["action_id"], + "service-deny-request:approved-model-1" + ); + assert_eq!(grant["action_ref"]["app"], SERVICES_CAPSULE_ID); + assert_eq!(grant["read"], true); + assert_eq!(grant["severity"], "info"); + assert!( + !entries.iter().any(|entry| { + entry["id"] == "service-access-grant:expired-model-1" + || entry["id"] == "service-access-grant:approved-engine-1" + || entry["id"] == "service-access-request:approved-model-1" + }), + "{inbox}" + ); + + let other = passkey_authority_with_name_role( + dir.path(), + Some("other"), + crate::auth::RuntimePrincipalRole::Guest, + ); + let (status, _) = home_test_post_json( + &app, + "/api/apps/home/launch", + &other.home_token, + "http://localhost:61180", + json!({ "target": INBOX_CAPSULE_ID }), + ) + .await; + assert_eq!(status, StatusCode::OK); + let other_token = app_token_for_authority(dir.path(), INBOX_CAPSULE_ID, &other); + let (status, other_inbox) = + home_test_get_json(&app, "/api/apps/inbox/summary", &other_token, "null").await; + assert_eq!(status, StatusCode::OK, "{other_inbox}"); + assert_eq!( + other_inbox["notifications"]["unread_count"], 0, + "{other_inbox}" + ); + assert_eq!( + other_inbox["notifications"]["attention_count"], 0, + "{other_inbox}" + ); + let other_entries = other_inbox["notifications"]["entries"].as_array().unwrap(); + assert!( + !other_entries.iter().any(|entry| { + entry["kind"] == "service_access_grant" || entry["kind"] == "service_access_request" + }), + "{other_inbox}" + ); +} + async fn services_contact_pending_engine_request( left: &std::path::Path, right: &std::path::Path, diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index ce3de9e42..2e3acdaca 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -1,5 +1,6 @@ use super::home_system::{ - configured_discovery_network_profile_for_test, write_home_principal_object_json_for_authority, + configured_discovery_network_profile_for_test, home_test_get_json, home_test_post_json, + write_home_principal_object_json_for_authority, }; use super::*; use crate::api::gateway::gateway_model_service::{ @@ -1110,6 +1111,37 @@ mod denial_authority { .unwrap() .to_path_buf() } + + fn owner_inbox_app(&self) -> axum::Router { + let mut state = test_state(self.owner.path()); + state.provider_registry = Some(self.registry.clone()); + gateway_router(state) + } + + async fn post_inbox_action( + app: &axum::Router, + token: &str, + action_id: &str, + ) -> (StatusCode, String) { + let response = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri("/api/apps/inbox/actions") + .header("x-elastos-home-token", token) + .header(CONTENT_TYPE, "application/json") + .body(Body::from(json!({ "action_id": action_id }).to_string())) + .unwrap(), + ) + .await + .unwrap(); + let status = response.status(); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + (status, String::from_utf8_lossy(&body).into_owned()) + } } #[tokio::test] @@ -1170,6 +1202,119 @@ mod denial_authority { assert_eq!(refused["code"], "denied", "{refused}"); fx.shutdown().await; } + + #[tokio::test] + async fn inbox_revoke_cancels_the_open_run_after_the_owner_saves_denial() { + let fx = TwoRuntimes::start().await; + let open = fx.create_run("seed-req-open", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let open_run_id = run_id_for(&fx.remote_principal(), "seed-req-open"); + let app = fx.owner_inbox_app(); + let (status, _) = home_test_post_json( + &app, + "/api/apps/home/launch", + &fx.authority.home_token, + "http://localhost:61180", + json!({ "target": INBOX_CAPSULE_ID }), + ) + .await; + assert_eq!(status, StatusCode::OK); + let token = app_token_for_authority(fx.owner.path(), INBOX_CAPSULE_ID, &fx.authority); + let (status, inbox) = + home_test_get_json(&app, "/api/apps/inbox/summary", &token, "null").await; + assert_eq!(status, StatusCode::OK, "{inbox}"); + assert_eq!(inbox["notifications"]["unread_count"], 0, "{inbox}"); + assert_eq!(inbox["notifications"]["attention_count"], 0, "{inbox}"); + let grant = inbox["notifications"]["entries"] + .as_array() + .unwrap() + .iter() + .find(|entry| entry["kind"] == "service_access_grant") + .expect("owner Inbox should show the approved model grant"); + assert_eq!(grant["read"], true); + assert_eq!( + grant["action_ref"]["action_id"], + format!("service-deny-request:{REQUEST_ID}") + ); + let (status, body) = TwoRuntimes::post_inbox_action( + &app, + &token, + &format!("service-deny-request:{REQUEST_ID}"), + ) + .await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{body}"); + assert!( + body.contains("service access request delivery failed"), + "{body}" + ); + let cancel = fx.last_provider_call().await; + assert_eq!(cancel["op"], "runs_cancel"); + assert_eq!(cancel["run_id"], open_run_id); + assert_eq!(cancel["runtime_binding"]["grant_id"], fx.grant_id); + let refused = fx.create_run("seed-req-after", "qwen-local").await; + assert_eq!(refused["code"], "denied", "{refused}"); + let (status, inbox) = + home_test_get_json(&app, "/api/apps/inbox/summary", &token, "null").await; + assert_eq!(status, StatusCode::OK, "{inbox}"); + assert!( + !inbox["notifications"]["entries"] + .as_array() + .unwrap() + .iter() + .any(|entry| entry["kind"] == "service_access_grant"), + "{inbox}" + ); + fx.shutdown().await; + } + + #[tokio::test] + async fn another_principal_cannot_revoke_the_owners_grant_from_inbox() { + let fx = TwoRuntimes::start().await; + let open = fx.create_run("seed-req-open", "qwen-local").await; + assert_eq!(open["ok"], true, "{open}"); + let ops_before = fx.provider_ops().await; + let app = fx.owner_inbox_app(); + let other = passkey_authority_with_name_role( + fx.owner.path(), + Some("other"), + crate::auth::RuntimePrincipalRole::Guest, + ); + let (status, _) = home_test_post_json( + &app, + "/api/apps/home/launch", + &other.home_token, + "http://localhost:61180", + json!({ "target": INBOX_CAPSULE_ID }), + ) + .await; + assert_eq!(status, StatusCode::OK); + let token = app_token_for_authority(fx.owner.path(), INBOX_CAPSULE_ID, &other); + let (status, inbox) = + home_test_get_json(&app, "/api/apps/inbox/summary", &token, "null").await; + assert_eq!(status, StatusCode::OK, "{inbox}"); + assert!( + !inbox["notifications"]["entries"] + .as_array() + .unwrap() + .iter() + .any(|entry| entry["kind"] == "service_access_grant"), + "{inbox}" + ); + let (status, body) = TwoRuntimes::post_inbox_action( + &app, + &token, + &format!("service-deny-request:{REQUEST_ID}"), + ) + .await; + assert_ne!(status, StatusCode::OK, "{body}"); + assert_eq!(fx.provider_ops().await, ops_before, "no cancel may follow"); + assert_eq!( + fx.create_run("seed-req-still-open", "qwen-local").await["ok"], + true, + "the grant stays approved" + ); + fx.shutdown().await; + } } // The provider journal prunes a run after its retention; the owning Runtime's diff --git a/scripts/inbox-product-behavior-smoke.mjs b/scripts/inbox-product-behavior-smoke.mjs index 916e58a51..1bf7d3202 100644 --- a/scripts/inbox-product-behavior-smoke.mjs +++ b/scripts/inbox-product-behavior-smoke.mjs @@ -90,6 +90,7 @@ function descendants(node) { function jsonResponse(payload) { return { ok: true, + status: 200, async json() { return payload; }, @@ -99,6 +100,19 @@ function jsonResponse(payload) { }; } +function textResponse(status, text) { + return { + ok: false, + status, + async json() { + throw new Error("error body is not json"); + }, + async text() { + return text; + }, + }; +} + function plainJson(value) { return JSON.parse(JSON.stringify(value)); } @@ -417,6 +431,101 @@ async function runInboxHomeChromeSmoke() { await settle(); } assert.deepEqual(serviceActions, ["service-approve-request:service-1", "service-deny-request:service-1"]); + + const grantActions = []; + let grantRevoked = false; + const grantEntry = { + id: "entry-grant", + kind: "service_access_grant", + title: "Seed may use your AI model", + body: "Seed may use Mac model. Revoke access to stop new runs and cancel open runs.", + severity: "info", + read: true, + source_app: "services", + action_ref: { action_id: "service-deny-request:grant-1" }, + }; + context.fetch = async (path, options = {}) => { + fetchCalls.push(path); + if (path === "/api/apps/inbox/actions") { + grantActions.push(JSON.parse(options.body).action_id); + grantRevoked = true; + return textResponse(503, "service access request delivery failed"); + } + return jsonResponse({ + notifications: { + attention_count: 0, + unread_count: 0, + entries: grantRevoked ? [] : [grantEntry], + }, + }); + }; + for (const callback of windowListeners.get("message") || []) { + callback({ origin: "null", source: parentFrame, + data: { type: "elastos:menu-command", cmd: "filter-all" } }); + } + await settle(); + const summariesBeforeRevoke = fetchCalls.filter((path) => path === "/api/apps/inbox/summary").length; + for (const callback of windowListeners.get("message") || []) { + callback({ origin: "null", source: parentFrame, + data: { type: "elastos:menu-command", cmd: "refresh" } }); + } + await settle(); + assert.equal(nodes.get("pending-count").textContent, "0"); + assert.equal(nodes.get("review-count").textContent, "0"); + const grantButtons = descendants(nodes.get("entry-rows")).filter(node => node.tagName === "BUTTON"); + assert.deepEqual(grantButtons.map(button => button.textContent), ["Open", "Revoke access"]); + assert.deepEqual(grantActions, [], "Viewing an approved grant must preserve its access"); + const revoke = grantButtons.find(button => button.textContent === "Revoke access"); + for (const callback of revoke.listeners.get("click") || []) callback(); + await settle(); + assert.deepEqual(grantActions, ["service-deny-request:grant-1"]); + assert.ok( + fetchCalls.filter((path) => path === "/api/apps/inbox/summary").length > summariesBeforeRevoke, + "503 after a saved revoke must refresh Inbox summary", + ); + assert.deepEqual( + descendants(nodes.get("entry-rows")).filter(node => node.tagName === "BUTTON").map(button => button.textContent), + [], + ); + assert.equal(nodes.get("pending-count").textContent, "0"); + assert.equal( + nodes.get("status-text").textContent, + "service access request delivery failed", + ); + + context.fetch = async () => jsonResponse({ notifications: { + attention_count: 1, + unread_count: 1, + entries: [ + { + id: "entry-service", + kind: "service_access_request", + title: "Browser Exit access", + body: "A contact requests Browser Exit access.", + severity: "attention", + read: false, + source_app: "services", + action_ref: { action_id: "service-approve-request:service-1" }, + }, + { + id: "entry-grant", + kind: "service_access_grant", + title: "Seed may use your AI model", + body: "Seed may use Mac model. Revoke access to stop new runs and cancel open runs.", + severity: "info", + read: true, + source_app: "services", + action_ref: { action_id: "service-deny-request:grant-1" }, + }, + ], + } }); + for (const callback of windowListeners.get("message") || []) { + callback({ origin: "null", source: parentFrame, + data: { type: "elastos:menu-command", cmd: "refresh" } }); + } + await settle(); + assert.equal(nodes.get("pending-count").textContent, "1"); + assert.equal(nodes.get("review-count").textContent, "1"); } async function runInboxLaunchSelectionSmoke() { From 4949ca63bf3160484762c977c1fe2faffc824a09 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 18:14:29 -0500 Subject: [PATCH 324/535] fix(assistant): show a failed model run after save and reload A capacity overflow create already settled as selection_unavailable, but Assistant stored a failed lastTurn and painted an empty banner. --- capsules/assistant/browser/agent-live.js | 5 ++- capsules/assistant/browser/agent-stream.js | 12 +++++ scripts/home-agent-cancellation.test.mjs | 51 ++++++++++++++++++++++ 3 files changed, 66 insertions(+), 2 deletions(-) diff --git a/capsules/assistant/browser/agent-live.js b/capsules/assistant/browser/agent-live.js index c659e8b79..c09586020 100644 --- a/capsules/assistant/browser/agent-live.js +++ b/capsules/assistant/browser/agent-live.js @@ -501,10 +501,11 @@ export async function streamChatViaContract( if (terminal.status === "settlement_unknown") { return finish({ settlementUnknown: true }); } - patch({ state: TurnState.FAILED, completedAt: Date.now() }); const detail = terminal.error && typeof terminal.error === "object" ? terminal.error : {}; + const code = String(detail.code || terminal.status || "run_failed"); + patch({ state: TurnState.FAILED, completedAt: Date.now(), error: code.slice(0, 120) }); throw contractError( - String(detail.code || terminal.status || "run_failed"), + code, String(detail.message || `run ${terminal.status}`), ); }; diff --git a/capsules/assistant/browser/agent-stream.js b/capsules/assistant/browser/agent-stream.js index d3bdee53a..22a47880c 100644 --- a/capsules/assistant/browser/agent-stream.js +++ b/capsules/assistant/browser/agent-stream.js @@ -535,6 +535,9 @@ export function formatStreamError(err) { if (code === "no_model_offers") { return NO_MODEL_OFFER_STATUS; } + if (code === "selection_unavailable") { + return `Model run failed: ${String(err?.message || "model offer is not available").slice(0, 160)}`; + } return err?.message ? `Model run failed: ${String(err.message).slice(0, 160)}` : "Model run failed"; @@ -589,6 +592,12 @@ function showRunSettlement(turn, detail = turn?.error === "run_not_found" el.append(" ", button); } +function showRunFailure(turn, detail) { + setStreamStatus(detail || formatStreamError({ + code: String(turn?.error || "run_failed"), + }), { tone: "error" }); +} + function showOutputUnretained(turn) { if (turn?.outputRetained !== false) { return; @@ -901,6 +910,8 @@ export function renderActiveSession() { void startLiveTurnForPrompt("", { resumeTurn: session.lastTurn }); } else if (session.lastTurn?.state === TurnState.SETTLEMENT_UNKNOWN) { showRunSettlement(session.lastTurn); + } else if (session.lastTurn?.state === TurnState.FAILED) { + showRunFailure(session.lastTurn); } else if (session.lastTurn?.outputRetained === false) { showOutputUnretained(session.lastTurn); } @@ -2337,6 +2348,7 @@ async function startLiveTurnForPrompt(userText, { resumeTurn = null } = {}) { if (generation !== ctx.streamGeneration) return; if (next.state === TurnState.CANCEL_PENDING) setStreamStatus("Cancel requested. Waiting for Runtime."); if (next.state === TurnState.SETTLEMENT_UNKNOWN) showRunSettlement(next); + if (next.state === TurnState.FAILED) showRunFailure(next); }, onAccepted: ({ run_id } = {}) => { persistTurn(turnStoreGet(turnId)); diff --git a/scripts/home-agent-cancellation.test.mjs b/scripts/home-agent-cancellation.test.mjs index 0042896fd..636dd0ae3 100644 --- a/scripts/home-agent-cancellation.test.mjs +++ b/scripts/home-agent-cancellation.test.mjs @@ -714,6 +714,57 @@ test("live chat keeps partial deltas and restores completed-but-unavailable outp assert.equal(status.textContent, controller.OUTPUT_UNRETAINED_STATUS); }); +test("immediate failed create shows the provider result and restores it after save", async t => { + t.after(() => { live.detachLiveChatStream(); live.selectLiveOffer(""); }); + const { ctx, status } = streamControllerFixture(); + const calls = []; + globalThis.fetch = async (url) => { + const op = new URL(url).pathname.split("/").pop(); + calls.push(op); + if (op === "offers_list") return { ok: true, json: async () => ({ offers: [ + { id: "fixture", title: "Fixture", operation: "text", input_modalities: ["text/plain"], output_modalities: ["text/plain"] }, + ] }) }; + if (op === "runs_create") return { ok: true, json: async () => ({ + run_id: "run-overflow", sequence_cursor: 2, status: "failed", + terminal: { status: "failed", error: { class: "selection_unavailable", code: "selection_unavailable", message: "model offer is not available" } }, + }) }; + throw new Error(`unexpected operation ${op}`); + }; + await live.probeLiveInference({ force: true }); + controller.startTurnForPrompt("Reply with the word overflow"); + const started = Date.now(); + while (Date.now() - started < 2000) { + if (ctx.sessions[0].lastTurn?.state === "failed" && status.textContent) break; + await new Promise((resolve) => setTimeout(resolve, 20)); + } + const visible = controller.formatStreamError({ + code: "selection_unavailable", + message: "model offer is not available", + }); + assert.equal(ctx.sessions[0].lastTurn.state, "failed"); + assert.equal(ctx.sessions[0].lastTurn.error, "selection_unavailable"); + assert.equal(ctx.sessions[0].lastTurn.providerRunId, "run-overflow"); + assert.ok(ctx.sessions[0].lastTurn.completedAt > 0); + assert.equal(status.textContent, visible); + assert.match(status.textContent, /model offer is not available/); + assert.doesNotMatch(status.textContent, /busy/i); + assert.equal(live.unresolvedModelTurn(ctx.sessions[0].lastTurn), false); + assert.equal(controller.canSubmitNewTurn(), true); + assert.deepEqual(calls.filter((op) => op.startsWith("runs_")), ["runs_create"]); + const saved = JSON.parse(JSON.stringify(workspace.serializeSessionForPersist(ctx.sessions[0]))); + assert.equal(saved.lastTurn.state, "failed"); + assert.equal(saved.lastTurn.error, "selection_unavailable"); + ctx.sessions[0] = recoverStalePersistedTurn(saved); + status.textContent = ""; + controller.renderActiveSession(); + assert.equal(ctx.sessions[0].lastTurn.state, "failed"); + assert.equal(status.textContent, visible); + assert.match(status.textContent, /model offer is not available/); + assert.doesNotMatch(status.textContent, /busy/i); + assert.equal(controller.canSubmitNewTurn(), true); + assert.deepEqual(calls.filter((op) => op.startsWith("runs_")), ["runs_create"]); +}); + function renderedAgentText(stream) { const row = (stream.children || []).find((child) => child.dataset?.role === "agent"); if (!row) return ""; From 8444ff513f594a8577c52ce7ac66a580d396c542 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 18:31:47 -0500 Subject: [PATCH 325/535] fix(model): return Busy when a live offer is at capacity Capacity now journals RateLimited/model_busy and returns before dispatch, so Assistant can show a Busy sentence while unavailable offers stay selection_unavailable. --- capsules/assistant/browser/agent-stream.js | 3 + capsules/model-provider/src/execution.rs | 5 +- capsules/model-provider/src/state.rs | 91 ++++++++++++++-------- scripts/home-agent-cancellation.test.mjs | 27 +++++++ 4 files changed, 90 insertions(+), 36 deletions(-) diff --git a/capsules/assistant/browser/agent-stream.js b/capsules/assistant/browser/agent-stream.js index 22a47880c..36fcc2260 100644 --- a/capsules/assistant/browser/agent-stream.js +++ b/capsules/assistant/browser/agent-stream.js @@ -535,6 +535,9 @@ export function formatStreamError(err) { if (code === "no_model_offers") { return NO_MODEL_OFFER_STATUS; } + if (code === "model_busy") { + return "Model is busy."; + } if (code === "selection_unavailable") { return `Model run failed: ${String(err?.message || "model offer is not available").slice(0, 160)}`; } diff --git a/capsules/model-provider/src/execution.rs b/capsules/model-provider/src/execution.rs index bae8cb8e0..29f5f1752 100644 --- a/capsules/model-provider/src/execution.rs +++ b/capsules/model-provider/src/execution.rs @@ -3451,10 +3451,7 @@ mod tests { let blocked_binding = create_binding("request:local-timeout-blocked", &offer, &blocked_input); let blocked = create_run(&provider, &offer, &blocked_binding, &blocked_input); - assert_eq!( - blocked["data"]["terminal"]["error"]["code"], - "selection_unavailable" - ); + assert_eq!(blocked["data"]["terminal"]["error"]["code"], "model_busy"); assert_eq!( wait_for_terminal(&provider, first_run_id, &access_binding(&first_binding))["data"] diff --git a/capsules/model-provider/src/state.rs b/capsules/model-provider/src/state.rs index 05529f586..2912f63fc 100644 --- a/capsules/model-provider/src/state.rs +++ b/capsules/model-provider/src/state.rs @@ -329,9 +329,9 @@ impl ModelProviderState
{ &offer, &mut run, RunError { - class: ErrorClass::SelectionUnavailable, - code: "selection_unavailable".to_string(), - message: "model offer is not available".to_string(), + class: ErrorClass::RateLimited, + code: "model_busy".to_string(), + message: "Model is busy.".to_string(), }, )?; self.journal.store_run(&run)?; @@ -3263,51 +3263,78 @@ mod tests { fn concurrency_limit_is_enforced() { let root = temp_root("concurrency"); let adapters = FakeAdapters { - dispatch_results: Arc::new(Mutex::new(vec![ - Ok(DispatchResult::Running { - events: vec![EventSeed { - kind: "dispatched", - data: serde_json::json!({}), - }], - backend_state: serde_json::json!({"job_id":"job-1"}), - }), - Ok(DispatchResult::Running { - events: vec![EventSeed { - kind: "dispatched", - data: serde_json::json!({}), - }], - backend_state: serde_json::json!({"job_id":"job-2"}), - }), - ])), + dispatch_results: Arc::new(Mutex::new(vec![Ok(DispatchResult::Running { + events: vec![EventSeed { + kind: "dispatched", + data: serde_json::json!({}), + }], + backend_state: serde_json::json!({"job_id":"job-1"}), + })])), ..Default::default() }; - let mut state = init_state(&root, vec![offer("local-text")], adapters); + let mut state = init_state(&root, vec![offer("local-text")], adapters.clone()); + assert_eq!(offer("local-text").policy.concurrency_limit, 1); let input1 = serde_json::json!({"schema":"elastos.model.input.text/v1","prompt":"one"}); let input2 = serde_json::json!({"schema":"elastos.model.input.text/v1","prompt":"two"}); + let first_binding = create_binding("request:one", "local-text", &input1); + let second_binding = create_binding("request:two", "local-text", &input2); let first = state .handle_runs_create(RunsCreateRequest { op: "runs_create".to_string(), offer_id: "local-text".to_string(), operation: "text.generate".to_string(), input: input1.clone(), - runtime_binding: create_binding("request:one", "local-text", &input1), - }) - .unwrap(); - let second = state - .handle_runs_create(RunsCreateRequest { - op: "runs_create".to_string(), - offer_id: "local-text".to_string(), - operation: "text.generate".to_string(), - input: input2.clone(), - runtime_binding: create_binding("request:two", "local-text", &input2), + runtime_binding: first_binding, }) .unwrap(); + let second_request = RunsCreateRequest { + op: "runs_create".to_string(), + offer_id: "local-text".to_string(), + operation: "text.generate".to_string(), + input: input2.clone(), + runtime_binding: second_binding.clone(), + }; + let second = state.handle_runs_create(second_request.clone()).unwrap(); assert_eq!(first["data"]["status"], "running"); assert_eq!(second["data"]["status"], "failed"); + assert_eq!(second["data"]["terminal"]["error"]["class"], "rate_limited"); + assert_eq!(second["data"]["terminal"]["error"]["code"], "model_busy"); assert_eq!( - second["data"]["terminal"]["error"]["class"], - "selection_unavailable" + second["data"]["terminal"]["error"]["message"], + "Model is busy." ); + assert_eq!(*adapters.dispatch_calls.lock().unwrap(), 1); + + let replay = state.handle_runs_create(second_request).unwrap(); + assert_eq!(replay["data"]["status"], "failed"); + assert_eq!(replay["data"]["run_id"], second["data"]["run_id"]); + assert_eq!(replay["data"]["terminal"]["error"]["class"], "rate_limited"); + assert_eq!(replay["data"]["terminal"]["error"]["code"], "model_busy"); + assert_eq!( + replay["data"]["terminal"]["error"]["message"], + "Model is busy." + ); + assert_eq!(*adapters.dispatch_calls.lock().unwrap(), 1); + + let events = state + .handle_runs_events(RunsEventsRequest { + op: "runs_events".to_string(), + run_id: second["data"]["run_id"].as_str().unwrap().to_string(), + after_sequence: Some(0), + runtime_binding: access_binding(&second_binding), + }) + .unwrap(); + let terminal = events["data"]["events"] + .as_array() + .unwrap() + .iter() + .find(|event| event["terminal"] == true) + .expect("capacity failure keeps a terminal event"); + assert_eq!(terminal["kind"], "failed"); + assert_eq!(terminal["data"]["class"], "rate_limited"); + assert_eq!(terminal["data"]["code"], "model_busy"); + assert_eq!(terminal["data"]["message"], "Model is busy."); + assert_eq!(*adapters.dispatch_calls.lock().unwrap(), 1); } #[test] diff --git a/scripts/home-agent-cancellation.test.mjs b/scripts/home-agent-cancellation.test.mjs index 636dd0ae3..5554b495f 100644 --- a/scripts/home-agent-cancellation.test.mjs +++ b/scripts/home-agent-cancellation.test.mjs @@ -765,6 +765,33 @@ test("immediate failed create shows the provider result and restores it after sa assert.deepEqual(calls.filter((op) => op.startsWith("runs_")), ["runs_create"]); }); +test("saved model_busy lastTurn shows Busy after restore", () => { + const { ctx, status } = streamControllerFixture(); + const saved = { + id: "one", + title: "overflow", + group: "Today", + messages: [{ role: "user", text: "Reply with the word overflow", modelText: "Reply with the word overflow" }], + lastTurn: { + state: "failed", + error: "model_busy", + providerRunId: "run-busy", + completedAt: 1, + }, + }; + ctx.sessions[0] = recoverStalePersistedTurn(saved); + status.textContent = ""; + controller.renderActiveSession(); + const visible = controller.formatStreamError({ code: "model_busy" }); + assert.equal(visible, "Model is busy."); + assert.equal(ctx.sessions[0].lastTurn.state, "failed"); + assert.equal(ctx.sessions[0].lastTurn.error, "model_busy"); + assert.equal(status.textContent, visible); + assert.match(status.textContent, /busy/i); + assert.doesNotMatch(status.textContent, /model offer is not available/); + assert.doesNotMatch(status.textContent, /Model run failed/); +}); + function renderedAgentText(stream) { const row = (stream.children || []).find((child) => child.dataset?.role === "agent"); if (!row) return ""; From 4ab99128b5d9491ef0d1c3536954f519aeee0def Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 18:45:24 -0500 Subject: [PATCH 326/535] fix(browser): give viewer-reload a separate 10s baseline budget Attempt 100 aborted during baseline acquisition under the shared 5s window, so actual reload never started. Keep the reload deadline at 5s and record failure_phase. --- scripts/lib/browser-journey-viewer-reload.mjs | 5 ++-- .../browser-journey-viewer-reload.test.mjs | 30 ++++++++++++++++++- 2 files changed, 32 insertions(+), 3 deletions(-) diff --git a/scripts/lib/browser-journey-viewer-reload.mjs b/scripts/lib/browser-journey-viewer-reload.mjs index 1488bb72f..7bc7566ab 100644 --- a/scripts/lib/browser-journey-viewer-reload.mjs +++ b/scripts/lib/browser-journey-viewer-reload.mjs @@ -1,7 +1,7 @@ import { createHash } from "node:crypto"; import { browserJourneyBinding, browserJourneyVideo } from "./browser-journey-recovery.mjs"; -const WINDOW_MS = 5000, POLL_MS = 250, SUFFIX = "-reload"; +const WINDOW_MS = 5000, BASELINE_MS = 10000, POLL_MS = 250, SUFFIX = "-reload"; const hash = value => `sha256:${createHash("sha256").update(String(value)).digest("hex").slice(0, 16)}`; const text = value => typeof value === "string" && value.length > 0 && value.length <= 512; const count = value => Number.isSafeInteger(value) && value >= 0; @@ -151,7 +151,7 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re return latest; } try { - const baselineDeadline = clock.now() + WINDOW_MS; + const baselineDeadline = clock.now() + BASELINE_MS; // Collect a successfully installed observer even if it reports a forbidden event during setup. stop = await within(baselineDeadline, "observer_deadline", budget => observeRequests(record, budget), true); requireEvidence(typeof stop === "function", "observer_missing"); @@ -223,6 +223,7 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re evidence.ok = !failure; if (failure) { evidence.failure = failure; + evidence.failure_phase = phase; const error = new Error(`Browser viewer reload diagnostic failed: ${failure}`); error.evidence = evidence; throw error; diff --git a/scripts/lib/browser-journey-viewer-reload.test.mjs b/scripts/lib/browser-journey-viewer-reload.test.mjs index 6ca6febff..12aee332e 100644 --- a/scripts/lib/browser-journey-viewer-reload.test.mjs +++ b/scripts/lib/browser-journey-viewer-reload.test.mjs @@ -90,7 +90,11 @@ function fixture(options = {}) { if (reads <= (options.pendingPageReads || 0)) raw.viewer.page_id = ""; if (!options.freezeNewVideo && !(sent && options.freezeAfterInput)) { frames++; bytes += 100; } if (options.bytesFreeze) bytes = 0; - } else { frames++; bytes += 100; } + } else { + if (options.baselineHangs) await new Promise(() => {}); + if (options.baselineStateDelay) await timer.wait(options.baselineStateDelay, budget.signal); + frames++; bytes += 100; + } raw.video = raw.viewer ? { present: true, hidden: reloaded && options.hidden === true, paused: false, ready_state: 4, video_width: 800, video_height: 600, client_width: 800, client_height: 600, decoded_frames: frames, ...(options.noBytes ? {} : { video_bytes_received: bytes }), token: secret } : null; @@ -287,6 +291,30 @@ test("reload callback, readiness, and input share one deadline from reload start test("readiness read hanging at the deadline aborts and removes the observer", async () => { const f = await fails({ stateDelay: 6000 }, "state_deadline"); assert.ok(f.calls.find(call => call.method === "state" && call.reloaded).budget.signal.aborted); + assert.equal(f.evidence.failure_phase, "reload"); + assert.equal(typeof f.evidence.reload_started_ms, "number"); +}); +test("slow baseline two serial samples can complete then reload stays on the five-second budget", async () => { + const f = fixture({ baselineStateDelay: 2534 }); + const proof = await f.run(); + assert.equal(proof.ok, true); + assert.ok(proof.reload_started_ms >= 5000); + assert.ok(proof.reload_ms <= 5000); + const reload = f.calls.find(call => call.method === "reload"); + const input = f.calls.find(call => call.method === "input"); + assert.equal(input.budget.deadlineMs, reload.at + 5000); + const baselineReads = f.calls.filter(call => call.method === "state" && !call.reloaded); + assert.ok(baselineReads.length >= 2); + assert.ok(baselineReads[0].budget.deadlineMs >= 10000); +}); +test("hung baseline is bounded and does not start reload", async () => { + const f = await fails({ baselineHangs: true }, "state_deadline"); + assert.equal(f.evidence.failure_phase, "baseline"); + assert.equal(f.evidence.reload_started_ms, undefined); + assert.equal(f.calls.some(call => call.method === "reload"), false); + const state = f.calls.find(call => call.method === "state"); + assert.ok(state.budget.signal.aborted); + assert.ok(state.budget.deadlineMs <= 10000); }); for (const kind of ["opening", "closing"]) { test(`${kind} immediately interrupts a hanging reload and aborts its budget`, async () => { From 57c102cfb9215125f836b78ad5d4fd5ad50b9e9c Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 21:12:49 -0500 Subject: [PATCH 327/535] docs: reconcile remote services evidence and remaining acceptance --- TASKS.md | 72 +++++++++++++++++++++++++----------------- docs/MODEL_PROVIDER.md | 14 ++++---- state.md | 39 +++++++++++++++++++++++ 3 files changed, 90 insertions(+), 35 deletions(-) diff --git a/TASKS.md b/TASKS.md index 4c1d266c0..4d55850d7 100644 --- a/TASKS.md +++ b/TASKS.md @@ -24,7 +24,11 @@ Initial analysis source: [`8ac18bec65ca650615be879f7ab3f66799d9fc53`](https://github.com/Elacity/elastos-runtime/tree/8ac18bec65ca650615be879f7ab3f66799d9fc53). Revalidate those findings against the implementation base before each repair. -### Delivery queue and 24-hour checkpoint plan +### Historical delivery queue and 24-hour checkpoint plan + +This dated queue preserves the original scope and allocations. The Now section +owns current execution and resource assignments; these historical Active labels +do not grant a present lease. Execution revision: 2026-09-08. The user requests the full Browser outcome by 2026-09-09, approximately 13:34 UTC, 24 hours after the deadline instruction. @@ -167,32 +171,40 @@ verification separate from product acceptance and required human review. ## Now -Mission revision R2, 2026-09-10. The [approved five-journey plan](https://app.notion.com/p/wauio/ElastOS-0-7-1-release-plan-five-user-journeys-from-start-to-stop-3d6b682adcca81948f78d12abcd677b9) -owns release acceptance and decisions D1-D6. The latest user clarification makes -the first target the seed server running reviewed code from the active local -work, rather than a link to a private local Runtime. [Execution](docs/EXECUTION.md) -owns the verification recipes and monitoring decisions; [state.md](state.md) -owns source, installed and public facts. - -Session handover: [2026-09-10 execution handover](docs/audits/2026-09-10-execution-handover.md). -The combined automated installed checkpoint passes; AUTH-01 retains its full -human verdict. The reviewed combined source `06bf4e0f` is published through -`feat/0.7.1-integration` in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64). -The website checkpoint and later donor history are protected there; the redundant -website refs and checkout are retired. The active integration checkout is named -in the private checkpoint. Public preview `25966622` is now deployed with -verified artifacts and preserved account/user data; Anders confirmed existing-account -sign-in and saved work. Mac Homes retain their tested candidate. The [team report](docs/audits/2026-09-11-team-sync.md) -and [contributor review](docs/audits/2026-09-11-contributor-review.md) describe -included work and remaining gates. C3/J3 pauses after the bounded Mac checkpoint -and publication. Next: exact Marketplace-to-Assistant handoff and safe removal, -then cold Content/Carrier delivery. Browser follows its named queue. Preserve the -remaining 69 branches and 11 worktrees under their existing history/owner gates; -recurring monitoring stays paused. - -This is the current execution queue. Older references below are retained -follow-up obligations, not permission to bypass this order or the release gate. -The C1-C7 sequence in the approved plan controls integration and publication. +Mission revision R3, 2026-09-12 (reconciled 2026-09-13 UTC). The +[approved five-journey plan](https://app.notion.com/p/wauio/ElastOS-0-7-1-release-plan-five-user-journeys-from-start-to-stop-3d6b682adcca81948f78d12abcd677b9) +owns J1-J5 acceptance and D1-D6 decisions, including the remote inference +amendment. This section owns execution order; [state.md](state.md) owns source, +installed and public evidence. Earlier delivery dates and allocations are history. + +`feat/remote-services` is the active candidate, extending the published PR64 +checkpoint with the Browser donor and remote Qwen work. Keep its reviewed history +and the separate dirty updater slice. Public preview remains on its accepted +September 11 artifacts. The [team report](docs/audits/2026-09-11-team-sync.md) +and [contributor review](docs/audits/2026-09-11-contributor-review.md) retain their +5-11 September scope; current progress is recorded in state. + +Use one execution owner for both installed test Homes. The existing Cursor +Model and Browser tasks are paused references until one receives the next bounded +slice. A reviewer checks the result at the evidence boundary. Recurring monitoring +stays paused. Prepare the clean candidate for draft review; publication and public +cutover retain their separate approvals. + +Next Browser slice: use the fitted desktop and normal Engine/Exit selection to +capture the reported startup failure. Verify the exact selected service and grant, +then distinguish environment, harness and product failure before a repair. Earlier +automated Home focus failures remain unexplained; Anders did not reproduce them. +After launch works, prove actual viewer reload within five seconds, input/media +and close before performance or endurance campaigns. + +The independent Qwen slice is one immediate Busy replay and retry after capacity +frees. Review the prepared helper only when that slice starts. Preserve accepted +reply, replay and revoke evidence. Complete restart during an active run and +second-principal denial, then join accepted U8/U9 in U10. Marketplace handoff, +safe removal and cold Content/Carrier delivery remain required under J3; AI1-AI6 +in the approved plan separate that work from later provider and business stages. + +The C1-C7 sequence below retains the full release gate. | ID | Outcome | Status / owner | Required input and next proof | | --- | --- | --- | --- | @@ -203,8 +215,10 @@ The C1-C7 sequence in the approved plan controls integration and publication. | C1-delivery | Signed three-platform candidate installation | Functional preparation complete; final assembly after C5 freeze / delivery owner | Native inputs, bootstrap and publisher import have bounded source/artifact proof. Keep working builds for functional tests and rebuild only affected components. Complete atomic promotion, final installer stamping, platform regeneration and signed fresh installs at the final release boundary. | | J1 / C2 | Install, passkey, Recovery Kit, Profile and usable Home | Human passkey proof pending / Home and installer owners | Registration binding and atomic/conflict-safe identity persistence are integrated with 66 focused tests passing. Durable owner intent and Profile-inclusive recovery now pass source and UI checks. Real signed Mac installation and seed export pass. Source repairs for kit-first recovery, gateway shutdown and media-tool reuse are accepted. Both targets pass installed browser recovery/reload/retry, complete owned shutdown/restart, full Home setup/media reuse and Carrier cleanup/reuse. User-review account label, Recovery/Advanced spacing and cross-port session isolation are repaired on `1e320578`; shared-browser Mac/Linux proof passes. Next: AUTH-01 final user acceptance on the updated preserved Homes; concurrent-save and window/selection UI; W2-W5; rerun the smoke with System entry through the ElastOS menu, then complete the app matrix; full Linux/Mac staging. Publish J1 for review first under D1. | | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | -| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Paused after checkpoint publication / coordinator; independent review | Continue on `feat/0.7.1-integration` after a new implementation instruction. Draft PR64 contains the complete combined checkpoint; the human Mac Home now has one Assistant and admitted Qwen. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Anders confirms Assistant works. Current `3c2f9a80` repairs pending Keep intent, normal Marketplace Models category/details, theme tokens and duplicate activity/headings. Source/rendered checks and installed Mac retention readback, Qwen reply/save/reload and human model-view observation pass, with matching receipts and user data preserved. Pending preparation itself uses source/rendered proof; the installed model was reused. Linux still needs disk headroom and a reviewed startup profile. Next required work is exact Marketplace-to-Assistant handoff, safe removal and cold Carrier delivery. Broader execution and monitoring pause at this user-feedback checkpoint; full human J3 remains required. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional, remote inference and Jetson Later. | -| J4 / C4 | Browser works in the two agreed placements | Planned / Browser owner | Retain canonical B01-B16 on the Browser branch; Mac A/A/A and Linux Home/Exit with Mac Engine A/B/A, including reload. Full claimed-role proof remains C6. | +| J3 / C3 | Obtain a model through Content/Carrier and use it locally | Paused after checkpoint publication / coordinator; independent review | The local-model checkpoint is preserved in draft PR64. Resume its remaining work from the active candidate named in Now. Draft PR64 contains the complete combined checkpoint; the human Mac Home now has one Assistant and admitted Qwen. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Anders confirms Assistant works. Current `3c2f9a80` repairs pending Keep intent, normal Marketplace Models category/details, theme tokens and duplicate activity/headings. Source/rendered checks and installed Mac retention readback, Qwen reply/save/reload and human model-view observation pass, with matching receipts and user data preserved. Pending preparation itself uses source/rendered proof; the installed model was reused. Linux still needs disk headroom and a reviewed startup profile. Next required work is exact Marketplace-to-Assistant handoff, safe removal and cold Carrier delivery. Broader execution and monitoring pause at this user-feedback checkpoint; full human J3 remains required. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional; remote inference is Required under C3R/M01-M06, and Jetson remains Later. | +| C3R / U8 | Approved seed user runs the Mac model | Partial installed acceptance / next execution owner | Accepted results are in state. Next: immediate Busy replay and successful retry, restart during a run without another dispatch, and denial of an unapproved second principal. AI1 adds exact package/offer/terms identity and Marketplace handoff before final M01-M06 acceptance. | +| J4 / C4 / U9 | Browser works in the two agreed placements | Functional proof open / next execution owner | Browser donor is integrated. Start with the observed Engine startup failure on the fitted desktop. Prove A/A/A and A/B/A, actual reload within five seconds, audio/video/input and all 13 close effects. Then finish authority, recovery, state and daily-use checks; B11 performance and B16 qualification retain their gates. | +| U10 | Qwen and Browser work together | Waiting for U8 and U9 / execution owner | One seed principal uses both on the same Mac; each keeps its own acceptance criteria. Include AI1 when its UX/identity changes enter the candidate. | | J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | PR60 original history is integrated in the published candidate; coordinate its open PR disposition with Irzhy. The focused video preparation repair adopted during C1 comes from the foundation inherited by PR62, rather than its four follow-up commits. Review PR59 authority cutover against final callers, then take PR62 work by readiness. J5 installed/human acceptance remains Required; extensions enter by readiness before freeze. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | | C6 | Accept the combined signed candidate | Planned / independent reviewer and human testers | All required journeys, shared Home/app regression, Browser qualification and first-hop update proof on matching artifacts. | diff --git a/docs/MODEL_PROVIDER.md b/docs/MODEL_PROVIDER.md index d3345e95f..555d02e3d 100644 --- a/docs/MODEL_PROVIDER.md +++ b/docs/MODEL_PROVIDER.md @@ -40,16 +40,18 @@ placement are independent: | Hosted API | Operator-configured hosted inference | Share after remote acceptance and upstream terms, privacy, and cost review | Both consumption paths use the existing model operations and service-offer -contract. Cross-Runtime model use remains planned work. Carrier currently -excludes `model` from its provider target allowlist. `RuntimeCreateBinding` and -`RuntimeAccessBinding` describe local-channel authority and carry no -authenticated remote issuer. +contract. Carrier admits the `model` target through the destination Runtime's +remote-model authorization path. The destination verifies the authenticated +source endpoint and service grant, then constructs its own local provider binding +for the consumer principal, capsule and run. `RuntimeCreateBinding` and +`RuntimeAccessBinding` remain local-channel authority records. Installed +acceptance and remaining remote-service work are recorded in [state.md](../state.md). An operator explicitly selects a configured capability for publication under Runtime policy. The owning Runtime publishes it as an `elastos.service.offer/v1` service and owns grants, quotas, selection, audit, -and routing. Before reusing provider invocation for remote model work, the -destination Runtime must verify the signed offer and grant, then map the +and routing. For remote model work, the +destination Runtime verifies the signed offer and grant, then maps the authenticated source Runtime and consumer principal, capsule, and run into destination-owned authority. Adding an allowlist entry alone cannot establish this binding. Carrier authenticates and transports the route that Runtime diff --git a/state.md b/state.md index a99b72cb9..03e8b4b9a 100644 --- a/state.md +++ b/state.md @@ -1,5 +1,44 @@ # State +## Current remote-services checkpoint, 2026-09-12 + +Reconciled 2026-09-13 UTC. The unpublished `feat/remote-services` candidate is +`4ab99128b5d9491ef0d1c3536954f519aeee0def`, tree +`2ae65c6737fe13986d0a7b50eaf346deae896f71`, with 161 commits beyond fetched +`origin/feat/0.7.1-integration` (`093b51ca`) and none missing from that base. +This identifies the tested source before the documentation reconciliation. +Product changes end at `8444ff51`; `4ab99128` changes the Browser test's baseline +budget to ten seconds while keeping actual viewer reload at five seconds. +The Browser donor `bf846cd8` is included. Its separate dirty updater work retains +its owner and has not entered this candidate. + +The isolated Mac and Linux Runtime binaries remain `50d835ed`; the Mac model +provider and seed Assistant use `8444ff51` components. Installation receipts bind +that mixed set explicitly. A fresh read confirms the Runtime/provider hashes and +available test Homes. Public Runtime retains its September 11 binary and responds +on the public Home route. These checks preserve earlier acceptance; they are not +a new full installed campaign. + +| Surface | Accepted evidence | Remaining proof | +| --- | --- | --- | +| Remote Qwen | Real reply and saved chat; exact retained replay without another dispatch; owner Revoke during a run and original-chat recovery with both identifiers; honest Outcome unknown; keyboard Assistant activation; saved historic failure display; fresh Busy classification | Immediate Busy replay and successful retry after capacity frees; restart during a run; unapproved second-principal denial; complete M01-M06 and AI1 identity/Marketplace acceptance | +| Browser | Bounded local run 92; remote run 100 navigation, media, input and 13 close effects | Actual remote reload and full U9 remain open. Attempts 101-103 stopped at automated Home focus before Engine selection; Anders did not reproduce that focus failure. Manual inspection separately shows Engine startup failure. | +| Test desktop | Manual helper now uses the native viewport; Anders confirms desktop fit | The viewport correction proves fit only. It does not establish the cause of earlier automated pointer/focus failures. | + +Expired Busy replay preserves the run identity and terminal status after the +provider journal expires. The Runtime's status-only fallback is intentional; +missing expired error details are not a demonstrated defect. Earlier failed +receipts remain historical evidence. Stop and revoke report the backend's +`settlement_unknown` as Outcome unknown; confirmed cancellation is a separate claim. + +The source can enter bounded draft review after documentation and basic gates +pass. Public cutover needs U8, U9 and U10, the selected public configuration, and +approval of the exact candidate. Full J1-J5, Browser B01-B16, model M01-M06, +Marketplace AI1, cold model delivery, signed installation/update and contributor +acceptance remain open under the approved plan. Follow the current Now section in +[TASKS.md](TASKS.md); previous checkpoints below retain their dated scope. + + ## Current public preview and review checkpoint, 2026-09-11 `feat/0.7.1-integration` is the active working/review branch in From 9283711e4b3cb2808a569a9ceaec86afbed046de Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 22:50:07 -0500 Subject: [PATCH 328/535] fix(browser): keep transfer and ingress errors in the open toast Ordinary Automatic open already returns the Runtime placement sentence. The toast replaced it with a generic pre-effect failure, so the user could not select the owning Engine. --- capsules/browser/browser/browser-status.js | 6 ++++++ capsules/browser/browser/browser-status.test.mjs | 16 ++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/capsules/browser/browser/browser-status.js b/capsules/browser/browser/browser-status.js index 86948540a..6576a5d53 100644 --- a/capsules/browser/browser/browser-status.js +++ b/capsules/browser/browser/browser-status.js @@ -98,6 +98,12 @@ export function friendlyOpenError(error) { if (error.status === 403) { return "This page was blocked by your Exit Node settings."; } + if (/This Browser profile requires an approved transfer/i.test(text)) { + return text; + } + if (/Runtime viewer ingress capacity unavailable/i.test(text)) { + return text; + } return "Browser Engine failed to start cleanly. No Browser page or VM was acquired."; } if (outcome?.state === "terminal_post_effect_cleanup") { diff --git a/capsules/browser/browser/browser-status.test.mjs b/capsules/browser/browser/browser-status.test.mjs index ba1fd5399..904384568 100644 --- a/capsules/browser/browser/browser-status.test.mjs +++ b/capsules/browser/browser/browser-status.test.mjs @@ -62,6 +62,22 @@ test("pre-effect open failure never claims a missing terminal close", () => { assert.doesNotMatch(friendlyOpenError(error), /terminal close/i); }); +test("pre-effect profile placement keeps the Runtime transfer sentence", () => { + const message = "This Browser profile requires an approved transfer from its owning Runtime"; + const error = openError("terminal_pre_effect_failure"); + error.message = message; + error.payload.message = message; + assert.equal(friendlyOpenError(error), message); +}); + +test("pre-effect viewer ingress capacity keeps the Runtime capacity sentence", () => { + const message = "Runtime viewer ingress capacity unavailable"; + const error = openError("terminal_pre_effect_failure"); + error.message = message; + error.payload.message = message; + assert.equal(friendlyOpenError(error), message); +}); + test("compatibility denial explains the required repair before launch", () => { for (const [code, message] of [ ["incompatible_engine_protocol", "Browser Engine and Runtime versions are incompatible. Update them to a compatible release."], From adc5dd2db512c1cedf2cfe4430a0260abd1c9037 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Sat, 12 Sep 2026 22:50:07 -0500 Subject: [PATCH 329/535] docs: record ordinary A/B/A open 110 and the remaining reload gate The selected Mac Engine opened ela.city on the installed pair. Home still needs to close that page before the five-second viewer reload proof. --- TASKS.md | 13 ++++++------- state.md | 2 +- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/TASKS.md b/TASKS.md index 4d55850d7..9f875cf96 100644 --- a/TASKS.md +++ b/TASKS.md @@ -190,12 +190,11 @@ slice. A reviewer checks the result at the evidence boundary. Recurring monitori stays paused. Prepare the clean candidate for draft review; publication and public cutover retain their separate approvals. -Next Browser slice: use the fitted desktop and normal Engine/Exit selection to -capture the reported startup failure. Verify the exact selected service and grant, -then distinguish environment, harness and product failure before a repair. Earlier -automated Home focus failures remain unexplained; Anders did not reproduce them. -After launch works, prove actual viewer reload within five seconds, input/media -and close before performance or endurance campaigns. +Next Browser slice: close the live 110 page through Home, then prove actual +viewer reload within five seconds, input/media and all 13 close effects on +Engine `remote-engine-ec54fd5eab67a5c5913513a47a797fc8` with seed Exit +`source-home-browser-exit`. Ordinary Automatic open fails at profile placement +while that grant is valid. Select the owning Mac Engine through Settings. The independent Qwen slice is one immediate Busy replay and retry after capacity frees. Review the prepared helper only when that slice starts. Preserve accepted @@ -217,7 +216,7 @@ The C1-C7 sequence below retains the full release gate. | J2 | Ordinary update preserves user state | Planned / delivery owner and independent reviewer | Coherent update commits on J1; Mac platform selection is repaired in 52fc231e with 11 updater tests; establish the actual stamped 0.7.0 starting updater. The published tag and local 0.7.0 tag differ; a source-home replacement cannot close first-hop proof. Candidate/interruption acceptance remains C6. | | J3 / C3 | Obtain a model through Content/Carrier and use it locally | Paused after checkpoint publication / coordinator; independent review | The local-model checkpoint is preserved in draft PR64. Resume its remaining work from the active candidate named in Now. Draft PR64 contains the complete combined checkpoint; the human Mac Home now has one Assistant and admitted Qwen. Bootstrap is accepted. The full corrected donor `6972e165` is merged by `b318bfda`, and Sash shelf ancestry is reconciled by `37c82d4f`. Irzhy foundation `617796a9` is merged by `dd21d8bd`; further intake is frozen for one installed Qwen journey. Mac catalog, Use/cancel/retry, exact admission, cold startup, actual reply and saved conversation pass on installed Runtime `fbf1a4b0`. Reply/draft/exact selection survive reload without implicit dispatch. Stop shows honest unknown, as allowed by Step 5; confirmed cancellation remains unproven. Owned shutdown, restart and second reply pass with identical package files, restored draft and zero received Bitswap payload. Installed model-menu empty-state/placement and workspace-preservation checks pass on `8f28b6e3`. The source-install metadata helper repair and its regression checks pass. The adapted URUX/UIUX candidate already entered through Irzhy reconstruction; the full old-tip merge recommendation is superseded. The approved one-Assistant implementation now preserves Sash’s UI in the canonical capsule, adopts all three stores into protected v2 storage, retains complete history/drafts, and preserves concurrent edits and run ownership. Source and rendered checks pass. Candidate `fa297cb5` now also passes installed one-Assistant catalog/legacy launch, real Qwen reply/save/reload and full Runtime restart with the original draft and exact model intact. The human Mac Home now has an enrolled passkey account and a verified locally cached Qwen package. Its System/Marketplace launch metadata defect is repaired by the `822c4e3d` installer guard and corrected receipts; both apps open in the actual browser. Anders confirms Assistant works. Current `3c2f9a80` repairs pending Keep intent, normal Marketplace Models category/details, theme tokens and duplicate activity/headings. Source/rendered checks and installed Mac retention readback, Qwen reply/save/reload and human model-view observation pass, with matching receipts and user data preserved. Pending preparation itself uses source/rendered proof; the installed model was reused. Linux still needs disk headroom and a reviewed startup profile. Next required work is exact Marketplace-to-Assistant handoff, safe removal and cold Carrier delivery. Broader execution and monitoring pause at this user-feedback checkpoint; full human J3 remains required. See [the convergence check](docs/audits/2026-09-11-assistant-convergence.md). URUX and Irzhy authority-cutover → follow-up work retain their Required gates and dependency order. Track every adapted or pending feature in [the preservation check](docs/audits/2026-09-11-integration-preservation.md). Preserve current recovery, sessions, media reuse and shutdown. The full merge includes storage/window source; retain their separate installed J1 acceptance. Prove normal controls and one bounded test-owned Home preparation attempt before large transfers. Complete acceptance steps 1, 2, 3 and 5 in section 6; hosted/Codex remains Optional; remote inference is Required under C3R/M01-M06, and Jetson remains Later. | | C3R / U8 | Approved seed user runs the Mac model | Partial installed acceptance / next execution owner | Accepted results are in state. Next: immediate Busy replay and successful retry, restart during a run without another dispatch, and denial of an unapproved second principal. AI1 adds exact package/offer/terms identity and Marketplace handoff before final M01-M06 acceptance. | -| J4 / C4 / U9 | Browser works in the two agreed placements | Functional proof open / next execution owner | Browser donor is integrated. Start with the observed Engine startup failure on the fitted desktop. Prove A/A/A and A/B/A, actual reload within five seconds, audio/video/input and all 13 close effects. Then finish authority, recovery, state and daily-use checks; B11 performance and B16 qualification retain their gates. | +| J4 / C4 / U9 | Browser works in the two agreed placements | Functional proof open / next execution owner | Ordinary A/B/A open 110 completed after Engine grant renew and Mac Engine selection. Close that live page, then prove actual reload within five seconds, audio/video/input and all 13 close effects. Then finish authority, recovery, state and daily-use checks; B11 performance and B16 qualification retain their gates. | | U10 | Qwen and Browser work together | Waiting for U8 and U9 / execution owner | One seed principal uses both on the same Mac; each keeps its own acceptance criteria. Include AI1 when its UX/identity changes enter the candidate. | | J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | PR60 original history is integrated in the published candidate; coordinate its open PR disposition with Irzhy. The focused video preparation repair adopted during C1 comes from the foundation inherited by PR62, rather than its four follow-up commits. Review PR59 authority cutover against final callers, then take PR62 work by readiness. J5 installed/human acceptance remains Required; extensions enter by readiness before freeze. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | diff --git a/state.md b/state.md index 03e8b4b9a..58f6c69e2 100644 --- a/state.md +++ b/state.md @@ -22,7 +22,7 @@ a new full installed campaign. | Surface | Accepted evidence | Remaining proof | | --- | --- | --- | | Remote Qwen | Real reply and saved chat; exact retained replay without another dispatch; owner Revoke during a run and original-chat recovery with both identifiers; honest Outcome unknown; keyboard Assistant activation; saved historic failure display; fresh Busy classification | Immediate Busy replay and successful retry after capacity frees; restart during a run; unapproved second-principal denial; complete M01-M06 and AI1 identity/Marketplace acceptance | -| Browser | Bounded local run 92; remote run 100 navigation, media, input and 13 close effects | Actual remote reload and full U9 remain open. Attempts 101-103 stopped at automated Home focus before Engine selection; Anders did not reproduce that focus failure. Manual inspection separately shows Engine startup failure. | +| Browser | Bounded local run 92; remote run 100 navigation, media, input and 13 close effects; ordinary open 110 completed on selected Mac Engine `remote-engine-ec54fd5eab67a5c5913513a47a797fc8` with seed Exit and `https://ela.city/` | Actual remote reload and full U9 remain open. Automatic open still fails at profile placement. Attempts 101-103 stopped at automated Home focus; Anders did not reproduce that focus failure. Open 110 left a live page that still needs Home close. | | Test desktop | Manual helper now uses the native viewport; Anders confirms desktop fit | The viewport correction proves fit only. It does not establish the cause of earlier automated pointer/focus failures. | Expired Busy replay preserves the run identity and terminal status after the From f4a04cfafa225d82fd6a0635ba1db8a8c4535bc8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 15 Sep 2026 16:41:35 -0500 Subject: [PATCH 330/535] fix(browser): keep restore page_status errors and visible recovery input HTTP and JSON Engine errors after restore stay errors, cut may reuse the last good binding, and recovery clicks the field the person already sees. --- scripts/lib/browser-journey-page-status.mjs | 52 ++++ .../lib/browser-journey-page-status.test.mjs | 99 +++++++ .../lib/browser-journey-recovered-input.mjs | 14 + .../browser-journey-recovered-input.test.mjs | 26 ++ scripts/lib/browser-journey-recovery.mjs | 100 ++++++- scripts/lib/browser-journey-recovery.test.mjs | 259 ++++++++++++++++++ 6 files changed, 543 insertions(+), 7 deletions(-) create mode 100644 scripts/lib/browser-journey-page-status.mjs create mode 100644 scripts/lib/browser-journey-page-status.test.mjs create mode 100644 scripts/lib/browser-journey-recovered-input.mjs create mode 100644 scripts/lib/browser-journey-recovered-input.test.mjs diff --git a/scripts/lib/browser-journey-page-status.mjs b/scripts/lib/browser-journey-page-status.mjs new file mode 100644 index 000000000..dc9fd5496 --- /dev/null +++ b/scripts/lib/browser-journey-page-status.mjs @@ -0,0 +1,52 @@ +export function isPageStatusTimeout(error) { + return error?.name === "AbortError" || error?.name === "TimeoutError"; +} + +export function pageStatusHttpError(status) { + const error = new Error(`Recovery page status failed: ${status}`); + error.name = "PageStatusHttpError"; + error.status = status; + return error; +} + +export function pageStatusJsonError(cause) { + const error = new Error("Recovery page status JSON failed"); + error.name = "PageStatusJsonError"; + error.cause = cause; + return error; +} + +// Timeout reuse keeps the last matching Engine status. HTTP and JSON errors +// stay visible so a later 403/404/500 cannot pass as a fresh observation. +export async function readBrowserPageStatus({ + fetchImpl, + url, + headers, + signal, + budgetMs, + lastPageStatus, +}) { + const statusController = new AbortController(); + const onAbort = () => statusController.abort(); + signal?.addEventListener("abort", onAbort, { once: true }); + const timer = budgetMs != null ? setTimeout(() => statusController.abort(), budgetMs) : null; + try { + const response = await fetchImpl(url, { headers, signal: statusController.signal }); + if (!response.ok) throw pageStatusHttpError(response.status); + let pageStatus; + try { + pageStatus = await response.json(); + } catch (error) { + throw pageStatusJsonError(error); + } + return { page_status: pageStatus, page_status_fresh: true }; + } catch (error) { + if (isPageStatusTimeout(error) && lastPageStatus) { + return { page_status: lastPageStatus, page_status_fresh: false }; + } + throw error; + } finally { + if (timer != null) clearTimeout(timer); + signal?.removeEventListener("abort", onAbort); + } +} diff --git a/scripts/lib/browser-journey-page-status.test.mjs b/scripts/lib/browser-journey-page-status.test.mjs new file mode 100644 index 000000000..1c0b45fd0 --- /dev/null +++ b/scripts/lib/browser-journey-page-status.test.mjs @@ -0,0 +1,99 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { + readBrowserPageStatus, +} from "./browser-journey-page-status.mjs"; + +const lastPageStatus = { + schema: "elastos.browser.page-status/v1", + page_id: "page-current", + actual_url: "http://localhost/nav", +}; +const freshPageStatus = { + ...lastPageStatus, + actual_url: "http://localhost/nav?fresh=1", +}; + +function httpResponse(status, body) { + return { + ok: status >= 200 && status < 300, + status, + async json() { + if (body instanceof Error) throw body; + return body; + }, + }; +} + +test("a fresh Engine status replaces lastPageStatus", async () => { + const result = await readBrowserPageStatus({ + fetchImpl: async () => httpResponse(200, freshPageStatus), + url: "http://runtime/status", + lastPageStatus, + budgetMs: 800, + }); + assert.equal(result.page_status_fresh, true); + assert.equal(result.page_status.actual_url, freshPageStatus.actual_url); +}); + +for (const status of [403, 404, 500]) { + test(`HTTP ${status} after restore stays an error`, async () => { + await assert.rejects(readBrowserPageStatus({ + fetchImpl: async () => httpResponse(status, { error: "denied" }), + url: "http://runtime/status", + lastPageStatus, + budgetMs: 800, + }), error => { + assert.equal(error.name, "PageStatusHttpError"); + assert.equal(error.status, status); + return true; + }); + }); +} + +test("invalid JSON after restore stays an error", async () => { + await assert.rejects(readBrowserPageStatus({ + fetchImpl: async () => httpResponse(200, new SyntaxError("bad json")), + url: "http://runtime/status", + lastPageStatus, + budgetMs: 800, + }), error => { + assert.equal(error.name, "PageStatusJsonError"); + return true; + }); +}); + +test("an 800 ms abort reuses lastPageStatus and marks it stale", async () => { + const result = await readBrowserPageStatus({ + fetchImpl: (_url, options) => new Promise((_, reject) => { + options.signal.addEventListener("abort", () => { + const error = new Error("The operation was aborted"); + error.name = "AbortError"; + reject(error); + }, { once: true }); + }), + url: "http://runtime/status", + lastPageStatus, + budgetMs: 20, + }); + assert.equal(result.page_status_fresh, false); + assert.equal(result.page_status, lastPageStatus); +}); + +test("a timeout without lastPageStatus stays an abort", async () => { + await assert.rejects(readBrowserPageStatus({ + fetchImpl: (_url, options) => new Promise((_, reject) => { + options.signal.addEventListener("abort", () => { + const error = new Error("The operation was aborted"); + error.name = "AbortError"; + reject(error); + }, { once: true }); + }), + url: "http://runtime/status", + lastPageStatus: null, + budgetMs: 20, + }), error => { + assert.equal(error.name, "AbortError"); + return true; + }); +}); diff --git a/scripts/lib/browser-journey-recovered-input.mjs b/scripts/lib/browser-journey-recovered-input.mjs new file mode 100644 index 000000000..1e2790680 --- /dev/null +++ b/scripts/lib/browser-journey-recovered-input.mjs @@ -0,0 +1,14 @@ +export function recoveredInputLiftPx(rect) { + if (!rect || !Number.isFinite(rect.y) || !Number.isFinite(rect.height)) return 0; + if (rect.y + rect.height * 0.5 >= 0) return 0; + return Math.min(1920, Math.max(640, Math.ceil(-(rect.y + rect.height * 0.5) / 640) * 640)); +} + +export function recoveredInputClickRect(lastRect, restRect) { + const lift = recoveredInputLiftPx(lastRect); + if (!lift) return { rect: lastRect, lift: 0 }; + return { + lift, + rect: restRect ? { ...restRect } : { ...lastRect, y: Math.max(0, lastRect.y + lift) }, + }; +} diff --git a/scripts/lib/browser-journey-recovered-input.test.mjs b/scripts/lib/browser-journey-recovered-input.test.mjs new file mode 100644 index 000000000..7071672f8 --- /dev/null +++ b/scripts/lib/browser-journey-recovered-input.test.mjs @@ -0,0 +1,26 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { recoveredInputClickRect, recoveredInputLiftPx } from "./browser-journey-recovered-input.mjs"; + +const field = { x: 96, y: 192, width: 240, height: 40 }; + +test("a visible rest field needs no lift", () => { + const result = recoveredInputClickRect(field, field); + assert.equal(result.lift, 0); + assert.equal(result.rect.y, 192); +}); + +test("a field below the viewport lifts once to the rest rectangle", () => { + const last = { ...field, y: -767.6875 }; + const result = recoveredInputClickRect(last, field); + assert.equal(result.lift, 1280); + assert.equal(result.rect.y, 192); + assert.equal(recoveredInputLiftPx(last), 1280); +}); + +test("a guessed lift without a rest rectangle stays at or above y 0", () => { + const last = { ...field, y: -173.6875 }; + const result = recoveredInputClickRect(last, null); + assert.equal(result.lift, 640); + assert.equal(result.rect.y, 466.3125); +}); diff --git a/scripts/lib/browser-journey-recovery.mjs b/scripts/lib/browser-journey-recovery.mjs index 48af131ee..2eab79c31 100644 --- a/scripts/lib/browser-journey-recovery.mjs +++ b/scripts/lib/browser-journey-recovery.mjs @@ -123,6 +123,15 @@ export async function diagnoseBrowserJourneyRecovery({ }; let phase = "baseline", stop, failure, forbidden = false; let original, initialReceipt, input, receiptSequence, initialVideo, cutVideo, bytesRequired; + let lastGoodBinding, lastBindingMs = 0, lastBindingAt = null, lastBindingPhase = null; + let bindingRefreshMissed = false, recoveryBindingFresh = false; + function setPhase(next) { + if (phase === next) return; + phase = next; + bindingRefreshMissed = false; + lastBindingMs = 0; + if (next === "recovery") recoveryBindingFresh = false; + } const pending = new Set(); const elapsed = () => Math.round(clock.now() - started); const pause = ms => new Promise(resolve => clock.setTimeout(resolve, ms)); @@ -157,6 +166,12 @@ export async function diagnoseBrowserJourneyRecovery({ ]); requireEvidence(clock.now() <= deadlineMs, code); return result; + } catch (error) { + if (error instanceof RecoveryFailure) throw error; + // A late AbortError from the expired signal is the deadline, not a + // fixture observation crash. + if (controller.signal.aborted) throw new RecoveryFailure(code); + throw error; } finally { clock.clearTimeout(timer); } @@ -176,7 +191,49 @@ export async function diagnoseBrowserJourneyRecovery({ request_hash: key, ...(count(event.status) && event.status <= 599 ? { status: event.status } : {}) }); } async function sample(deadline) { - const raw = await within(deadline, "binding_deadline", readBinding); + let raw, bindingSource = "fresh"; + const remaining = deadline - clock.now(); + const requireFresh = phase === "recovery" && !recoveryBindingFresh; + const samePhaseReuse = (phase === "baseline" || phase === "recovery") && lastBindingPhase === phase; + // Cut proves stall from video samples. Engine page_status through the + // offline viewer may time out; reuse the last good binding so that read + // does not consume the stall clock. + const cutReuse = phase === "cut"; + const mayReuse = !requireFresh && lastGoodBinding && (samePhaseReuse || cutReuse); + const reuseBinding = mayReuse && + (bindingRefreshMissed || remaining <= lastBindingMs + POLL_MS); + if (reuseBinding) { + raw = lastGoodBinding; + bindingSource = "cached"; + } else { + const startedAt = clock.now(); + const refreshDeadline = mayReuse + ? Math.min(deadline, startedAt + lastBindingMs + POLL_MS) + : deadline; + try { + raw = await within(refreshDeadline, "binding_deadline", readBinding); + lastBindingMs = Math.max(lastBindingMs, clock.now() - startedAt); + lastBindingAt = clock.now(); + lastBindingPhase = phase; + if (phase === "recovery") { + recoveryBindingFresh = true; + if (evidence.binding_observation_ms == null) { + evidence.binding_observation_ms = Math.round(clock.now() - startedAt); + } + } + } catch (error) { + // Same-phase reuse keeps the media budget. Recovery still requires + // one completed post-restore binding read before any cache. + if (mayReuse && error instanceof RecoveryFailure && error.message === "binding_deadline") { + bindingRefreshMissed = true; + lastBindingMs = Math.max(lastBindingMs, clock.now() - startedAt); + raw = lastGoodBinding; + bindingSource = "cached"; + } else { + throw error; + } + } + } const state = raw?.sessions?.recoverable_page?.state; evidence.last_runtime = { owner_state: ["active", "cleanup_pending"].includes(state) ? state : "absent_or_unknown", @@ -185,11 +242,21 @@ export async function diagnoseBrowserJourneyRecovery({ .map(key => [key, raw.sessions[key]])), }; const value = checkedBinding(raw); + lastGoodBinding = raw; evidence.changed_binding_fields = BINDING_FIELDS.filter((_, i) => value[i] !== original[i]); requireEvidence(evidence.changed_binding_fields.length === 0, "binding_changed"); const media = video(await within(deadline, "video_deadline", readVideo), bytesRequired); requireEvidence(Object.hasOwn(media, "video_bytes_received") === bytesRequired, "byte_metrics_changed"); - evidence.samples.push({ at_ms: elapsed(), phase, binding_matches: true, ...media }); + const pageStatusFresh = raw?.page_status_fresh !== false; + evidence.samples.push({ + at_ms: elapsed(), phase, binding_source: bindingSource, + binding_matches: bindingSource === "fresh" && pageStatusFresh, + page_status_fresh: pageStatusFresh, ...media, + ...(bindingSource === "cached" ? { + binding_cached_phase: lastBindingPhase, + ...(lastBindingAt != null ? { binding_age_ms: Math.round(clock.now() - lastBindingAt) } : {}), + } : {}), + }); if (evidence.samples.length > 64) evidence.samples.shift(); return media; } @@ -207,7 +274,13 @@ export async function diagnoseBrowserJourneyRecovery({ const deadline = clock.now() + WINDOW_MS; stop = await within(deadline, "observer_deadline", budget => observeRequests(record, budget)); requireEvidence(typeof stop === "function", "observer_missing"); - original = checkedBinding(await within(deadline, "binding_deadline", readBinding)); + const originalStarted = clock.now(); + const originalRaw = await within(deadline, "binding_deadline", readBinding); + lastBindingMs = Math.max(lastBindingMs, clock.now() - originalStarted); + original = checkedBinding(originalRaw); + lastGoodBinding = originalRaw; + lastBindingPhase = "baseline"; + lastBindingAt = clock.now(); evidence.binding_hashes = Object.fromEntries(BINDING_FIELDS.map((key, i) => [key, hash(String(original[i]))])); initialReceipt = validReceipt(await within(deadline, "receipt_deadline", readReceipt)); input = initialReceipt.events.findLast(event => event.type === "input"); @@ -237,7 +310,7 @@ export async function diagnoseBrowserJourneyRecovery({ await within(deadline, "media_cut_deadline", budget => mediaInterruption.cut(budget)); evidence.media_cut_acknowledged = true; } - phase = "cut"; + setPhase("cut"); const cutStarted = clock.now(), cutDeadline = cutStarted + WINDOW_MS; let unchangedSince = cutStarted; await within(cutDeadline, "viewer_probe_deadline", probeViewerRequest); @@ -268,7 +341,7 @@ export async function diagnoseBrowserJourneyRecovery({ failure = error instanceof RecoveryFailure ? error.message : "observation_failed"; throw error; } finally { - phase = "restoring"; + setPhase("restoring"); evidence.restore.attempted = true; try { const restoreDeadline = clock.now() + WINDOW_MS; @@ -293,9 +366,22 @@ export async function diagnoseBrowserJourneyRecovery({ } } - phase = "recovery"; + setPhase("recovery"); + const observeStarted = clock.now(); + const observedRaw = await within(observeStarted + WINDOW_MS, "binding_deadline", readBinding); + evidence.binding_observation_ms = Math.round(clock.now() - observeStarted); + evidence.page_status_fresh = observedRaw.page_status_fresh !== false; + requireEvidence(evidence.page_status_fresh, "page_status_stale"); + lastBindingMs = Math.max(lastBindingMs, evidence.binding_observation_ms); + lastBindingAt = clock.now(); + lastBindingPhase = "recovery"; + lastGoodBinding = observedRaw; + checkedBinding(observedRaw); + recoveryBindingFresh = true; + bindingRefreshMissed = true; + // Media and input keep the five-second criterion. Binding observation + // latency is recorded separately and does not shrink this window. const restored = clock.now(), deadline = restored + WINDOW_MS; - // Every observation and the input effect share this one restoration deadline. const restoredVideo = await sample(deadline); let sent = false, beforeInputVideo; while (true) { diff --git a/scripts/lib/browser-journey-recovery.test.mjs b/scripts/lib/browser-journey-recovery.test.mjs index 6de3a54d6..a175e3256 100644 --- a/scripts/lib/browser-journey-recovery.test.mjs +++ b/scripts/lib/browser-journey-recovery.test.mjs @@ -295,6 +295,265 @@ test("bounded evidence still rejects a close after event overflow", async () => assert.ok(evidence.dropped_requests > 0); }); test("a new page load cannot be hidden by an input receipt", () => fails({ reload: true }, "page_reloaded")); +test("baseline reuses the last matching binding when the next binding read misses the shared deadline", async () => { + const f = fixture(); + const readBinding = f.args.readBinding, readVideo = f.args.readVideo; + let videos = 0, bindings = 0; + f.args.readVideo = async budget => { + const value = await readVideo(budget); + videos++; + if (videos <= 2) { + return { ...value, decoded_frames: 10, video_bytes_received: 100 }; + } + return value; + }; + f.args.readBinding = async budget => { + bindings++; + if (bindings === 3) { + await new Promise(resolve => { + const timer = f.args.clock.setTimeout(resolve, 10_000); + budget.signal.addEventListener("abort", () => { + f.args.clock.clearTimeout(timer); + resolve(); + }, { once: true }); + }); + } + return readBinding(budget); + }; + const evidence = await f.run(); + assert.equal(evidence.ok, true); + assert.ok(evidence.cut_ms >= 5000); + assert.ok(evidence.recovery_ms <= 5000); + assert.ok(bindings >= 3); + assertRestoredAndDetached(f); +}); +test("recovery reuses the last matching binding when remaining time cannot finish another binding read", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + let recoveryReads = 0; + f.args.readBinding = async budget => { + if (f.calls.some(call => call.params?.offline === false)) { + recoveryReads++; + if (recoveryReads === 1) { + await new Promise(resolve => { + const timer = f.args.clock.setTimeout(resolve, 4000); + budget.signal.addEventListener("abort", () => { + f.args.clock.clearTimeout(timer); + resolve(); + }, { once: true }); + }); + } + assert.equal(recoveryReads, 1, "must reuse the first recovery binding"); + } + return readBinding(budget); + }; + const evidence = await f.run(); + assert.equal(evidence.ok, true); + assert.equal(recoveryReads, 1); + assert.ok(evidence.binding_observation_ms >= 4000); + assert.ok(evidence.recovery_ms <= 5000); + assertRestoredAndDetached(f); +}); +test("a completed post-restore binding observation leaves a full five-second media window", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + const extendInput = f.args.extendInput; + f.args.readBinding = async budget => { + if (f.calls.some(call => call.params?.offline === false)) { + await new Promise(resolve => { + const timer = f.args.clock.setTimeout(resolve, 4000); + budget.signal.addEventListener("abort", () => { + f.args.clock.clearTimeout(timer); + resolve(); + }, { once: true }); + }); + } + return readBinding(budget); + }; + f.args.extendInput = async (suffix, budget) => { + assert.ok(budget.timeoutMs >= 4000, "observation latency must not shrink the media window"); + return extendInput(suffix, budget); + }; + const evidence = await f.run(); + assert.equal(evidence.ok, true); + assert.ok(evidence.binding_observation_ms >= 4000); + assert.ok(evidence.recovery_ms <= 5000); + assertRestoredAndDetached(f); +}); +test("a baseline binding timeout cannot hide a changed owner after restore", async () => { + const f = fixture({ changeBinding(value) { + value.page_id = value.sessions.recoverable_page.page_id = "replacement"; + } }); + const readBinding = f.args.readBinding, readVideo = f.args.readVideo; + let videos = 0, bindings = 0; + f.args.readVideo = async budget => { + const value = await readVideo(budget); + videos++; + if (videos <= 2) return { ...value, decoded_frames: 10, video_bytes_received: 100 }; + return value; + }; + f.args.readBinding = async budget => { + bindings++; + if (bindings === 3) { + await new Promise(resolve => { + const timer = f.args.clock.setTimeout(resolve, 10_000); + budget.signal.addEventListener("abort", () => { + f.args.clock.clearTimeout(timer); + resolve(); + }, { once: true }); + }); + } + return readBinding(budget); + }; + await assert.rejects(f.run(), error => error.evidence.failure === "binding_changed" || + error.evidence.failure === "binding_unavailable_or_cleanup"); + assertRestoredAndDetached(f); +}); +test("an aborted binding read after the deadline is binding_deadline", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + f.args.readBinding = async budget => { + if (f.calls.some(call => call.params?.offline === false)) { + await new Promise((_, reject) => { + const fail = () => { + const error = new Error("The operation was aborted"); + error.name = "AbortError"; + reject(error); + }; + if (budget.signal.aborted) { + fail(); + return; + } + budget.signal.addEventListener("abort", fail, { once: true }); + }); + } + return readBinding(budget); + }; + await assert.rejects(f.run(), error => { + assert.equal(error.evidence.failure, "binding_deadline"); + assert.notEqual(error.evidence.failure, "observation_failed"); + return true; + }); + assertRestoredAndDetached(f); +}); +test("a reused Engine status after restore is page_status_stale", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + f.args.readBinding = async budget => { + const value = await readBinding(budget); + if (f.calls.some(call => call.params?.offline === false)) { + return { ...value, page_status_fresh: false }; + } + return value; + }; + await assert.rejects(f.run(), error => { + assert.equal(error.evidence.failure, "page_status_stale"); + assert.equal(error.evidence.page_status_fresh, false); + return true; + }); + assertRestoredAndDetached(f); +}); +test("HTTP page_status errors after restore stay observation failures", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + f.args.readBinding = async budget => { + if (f.calls.some(call => call.params?.offline === false)) { + const error = new Error("Recovery page status failed: 403"); + error.name = "PageStatusHttpError"; + error.status = 403; + throw error; + } + return readBinding(budget); + }; + await assert.rejects(f.run(), error => { + assert.equal(error.evidence.failure, "observation_failed"); + return true; + }); + assertRestoredAndDetached(f); +}); +test("a later reused Engine status does not count as a matching binding", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + let recoveryReads = 0; + f.args.readBinding = async budget => { + const value = await readBinding(budget); + if (!f.calls.some(call => call.params?.offline === false)) return value; + recoveryReads++; + return { ...value, page_status_fresh: recoveryReads === 1 }; + }; + const evidence = await f.run(); + assert.equal(evidence.ok, true); + assert.equal(evidence.page_status_fresh, true); + for (const sample of evidence.samples) { + if (sample.page_status_fresh === false) assert.equal(sample.binding_matches, false); + } + assertRestoredAndDetached(f); +}); +test("a hung cut binding still proves the media stall from video samples", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + f.args.readBinding = async budget => { + if (f.calls.some(call => call.params?.offline === true) && + !f.calls.some(call => call.params?.offline === false)) { + await new Promise(resolve => { + const timer = f.args.clock.setTimeout(resolve, 10_000); + budget.signal.addEventListener("abort", () => { + f.args.clock.clearTimeout(timer); + resolve(); + }, { once: true }); + }); + } + return readBinding(budget); + }; + const evidence = await f.run(); + assert.equal(evidence.ok, true); + assert.ok(evidence.stall_ms >= 1000); + assert.ok(evidence.samples.some(sample => sample.phase === "cut" && sample.binding_source === "cached")); + assertRestoredAndDetached(f); +}); +test("a completed cut binding that changes the owner still fails", async () => { + const f = fixture(); + const readBinding = f.args.readBinding; + f.args.readBinding = async budget => { + const value = await readBinding(budget); + if (f.calls.some(call => call.params?.offline === true) && + !f.calls.some(call => call.params?.offline === false)) { + value.page_id = value.sessions.recoverable_page.page_id = "replacement"; + } + return value; + }; + await assert.rejects(f.run(), error => error.evidence.failure === "binding_changed" || + error.evidence.failure === "binding_unavailable_or_cleanup"); + assertRestoredAndDetached(f); +}); +test("a hung first recovery binding read cannot pass on the cut binding", async () => { + const f = fixture({ changeBinding(value) { + value.page_id = value.sessions.recoverable_page.page_id = "replacement"; + } }); + const readBinding = f.args.readBinding; + f.args.readBinding = async budget => { + if (f.calls.some(call => call.params?.offline === false)) { + await new Promise(resolve => { + const timer = f.args.clock.setTimeout(resolve, 10_000); + budget.signal.addEventListener("abort", () => { + f.args.clock.clearTimeout(timer); + resolve(); + }, { once: true }); + }); + } + return readBinding(budget); + }; + await assert.rejects(f.run(), error => error.evidence.failure === "binding_deadline"); + assertRestoredAndDetached(f); +}); +test("a completed first recovery binding still rejects a changed owner", async () => { + const f = fixture({ changeBinding(value) { + value.page_id = value.sessions.recoverable_page.page_id = "replacement"; + } }); + await assert.rejects(f.run(), error => error.evidence.failure === "binding_changed" || + error.evidence.failure === "binding_unavailable_or_cleanup"); + assertRestoredAndDetached(f); +}); test("input receives only the remaining shared recovery budget", async () => { const f = fixture(); const readVideo = f.args.readVideo; From 6aa7c5aeee85d60987cdc8a28a42e8f60c55be20 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Tue, 15 Sep 2026 16:56:54 -0500 Subject: [PATCH 331/535] fix(browser): sample audio ICE-pair bytes on the inbound RTP clock Receiver getStats already had inbound and remote-outbound counters. Add the nominated candidate-pair so a TURN hold can be compared with inbound RTP without changing tone acceptance. --- scripts/lib/browser-journey-audio.mjs | 64 +++++++++++++++++-- scripts/lib/browser-journey-audio.test.mjs | 72 ++++++++++++++++++++++ 2 files changed, 130 insertions(+), 6 deletions(-) diff --git a/scripts/lib/browser-journey-audio.mjs b/scripts/lib/browser-journey-audio.mjs index 926b39986..20e377283 100644 --- a/scripts/lib/browser-journey-audio.mjs +++ b/scripts/lib/browser-journey-audio.mjs @@ -1,6 +1,9 @@ // Test-only observation of the product receiver. The Engine fixture generates // the tone; this probe reads decoded PCM and never creates playback audio. -export function installBrowserJourneyAudioProbe() { +export function installBrowserJourneyAudioProbe(options) { + const windowMs = Number.isFinite(options?.windowMs) ? Math.max(500, options.windowMs) : 2500; + const maxSamples = Math.max(60, Math.floor(windowMs / 50) + 4); + const rtpLimit = Math.max(13, Math.floor(windowMs / 200) + 1); const elements = []; const peers = []; const NativePeer = window.RTCPeerConnection; @@ -30,10 +33,14 @@ export function installBrowserJourneyAudioProbe() { const context = new AudioContext(); let source, analyser, resumeTimer; const samples = [], started = performance.now(); - const result = { ok: false, samples, track_id: track.id, + const result = { ok: false, samples, track_id: track.id, window_ms: windowMs, receiver_metrics_before: window.__elastosBrowserRemoteDisplayMetrics || null }; const rtp = result.inbound_audio_rtp = { status: "receiver_unavailable", receiver_match_count: 0, requests: 0, samples: [] }; + const outbound = result.remote_outbound_audio_rtp = { status: "receiver_unavailable", + requests: 0, samples: [] }; + const ice = result.audio_ice_pair = { status: "receiver_unavailable", + requests: 0, samples: [] }; let binding, rtpStopped = false, rtpPending = false, nextRtpAt = 0, reportId; try { const matches = peers.flatMap(peer => peer.connectionState === "closed" ? [] : @@ -54,23 +61,33 @@ export function installBrowserJourneyAudioProbe() { const sampleRtp = async () => { const requestedAt = performance.now() - started; if (!binding || rtpStopped || rtpPending || requestedAt < nextRtpAt || - requestedAt >= 2500 || rtp.requests >= 13) return; + requestedAt >= windowMs || rtp.requests >= rtpLimit) return; nextRtpAt = requestedAt + 200; rtp.requests++; rtpPending = true; try { if (!receiverCurrent()) { rtp.status = "receiver_changed"; binding = null; return; } const reports = await binding.receiver.getStats(); - if (rtpStopped || performance.now() - started >= 2500) return; + if (rtpStopped || performance.now() - started >= windowMs) return; if (!receiverCurrent()) { rtp.status = "receiver_changed"; binding = null; return; } const inbound = [...reports.values()].filter(item => item.type === "inbound-rtp" && (item.kind === "audio" || item.mediaType === "audio")); + const remoteOut = [...reports.values()].filter(item => item.type === "remote-outbound-rtp" && + (item.kind === "audio" || item.mediaType === "audio")); + const pairs = [...reports.values()].filter(item => item.type === "candidate-pair" && + item.state === "succeeded" && item.nominated === true); const sample = { requested_at_ms: requestedAt, at_ms: performance.now() - started }; + const sent = { requested_at_ms: requestedAt, at_ms: sample.at_ms }; + const iceSample = { requested_at_ms: requestedAt, at_ms: sample.at_ms }; const item = inbound[0]; if (inbound.length !== 1 || (item.trackIdentifier !== undefined && item.trackIdentifier !== track.id)) { sample.status = "report_unavailable"; + sent.status = "report_unavailable"; + iceSample.status = "report_unavailable"; } else if (reportId !== undefined && item.id !== reportId) { sample.status = "report_changed"; + sent.status = "report_changed"; + iceSample.status = "report_changed"; binding = null; } else { reportId = item.id; @@ -82,11 +99,46 @@ export function installBrowserJourneyAudioProbe() { "silentConcealedSamples", "concealmentEvents", "jitterBufferDelay", "jitterBufferEmittedCount"]) { if (typeof item[key] === "number" && Number.isFinite(item[key])) sample[key] = item[key]; } + if (remoteOut.length !== 1) { + sent.status = "report_unavailable"; + } else { + const remote = remoteOut[0]; + sent.status = "observed"; + for (const key of ["timestamp", "remoteTimestamp", "bytesSent", "packetsSent"]) { + if (typeof remote[key] === "number" && Number.isFinite(remote[key])) sent[key] = remote[key]; + } + } + if (pairs.length !== 1) { + iceSample.status = "report_unavailable"; + } else { + const pair = pairs[0]; + iceSample.status = "observed"; + for (const key of ["bytesReceived", "bytesSent", "packetsReceived", "packetsSent", + "currentRoundTripTime", "availableIncomingBitrate"]) { + if (typeof pair[key] === "number" && Number.isFinite(pair[key])) iceSample[key] = pair[key]; + } + const remoteCandidate = [...reports.values()].find(entry => + entry.type === "remote-candidate" && entry.id === pair.remoteCandidateId); + if (Number.isInteger(remoteCandidate?.port) && remoteCandidate.port > 0 && + remoteCandidate.port <= 65535) { + iceSample.remote_port = remoteCandidate.port; + } + if (typeof remoteCandidate?.candidateType === "string" && + remoteCandidate.candidateType.length <= 16) { + iceSample.remote_candidate_type = remoteCandidate.candidateType; + } + } } rtp.samples.push(sample); rtp.status = sample.status; + outbound.samples.push(sent); + outbound.status = sent.status; + outbound.requests = rtp.requests; + ice.samples.push(iceSample); + ice.status = iceSample.status; + ice.requests = rtp.requests; } catch { - if (!rtpStopped && performance.now() - started < 2500) rtp.status = "get_stats_failed"; + if (!rtpStopped && performance.now() - started < windowMs) rtp.status = "get_stats_failed"; } finally { rtpPending = false; } }; try { @@ -101,7 +153,7 @@ export function installBrowserJourneyAudioProbe() { source.connect(analyser); const pcm = new Float32Array(analyser.fftSize); const spectrum = new Float32Array(analyser.frequencyBinCount); - while (performance.now() - started < 2500 && samples.length < 60) { + while (performance.now() - started < windowMs && samples.length < maxSamples) { if (audio.srcObject !== stream || track.readyState !== "live") { throw new Error("product audio receiver changed"); } diff --git a/scripts/lib/browser-journey-audio.test.mjs b/scripts/lib/browser-journey-audio.test.mjs index 52c2c49fa..c2d1bb039 100644 --- a/scripts/lib/browser-journey-audio.test.mjs +++ b/scripts/lib/browser-journey-audio.test.mjs @@ -148,6 +148,78 @@ function rtpReport(state, patch = {}) { sdp: "PRIVATE-SDP", candidate: "PRIVATE-CANDIDATE", codecId: "PRIVATE-CODEC", ...patch }]]); } +test("remote-outbound RTP is sampled from the same getStats report and never changes tone acceptance", async () => { + const state = audioProbeFixture({ stats(state) { + const reports = rtpReport(state); + reports.set("remote-out", { id: "remote-out", type: "remote-outbound-rtp", kind: "audio", + timestamp: 100000 + state.now, remoteTimestamp: 200000 + state.now, + bytesSent: 2000 + state.now, packetsSent: 80 + state.now / 10, + sdp: "PRIVATE-SDP", candidate: "PRIVATE-CANDIDATE", codecId: "PRIVATE-CODEC" }); + return reports; + } }); + const result = await state.run(); + assert.equal(controlledTonePresent(result), true); + const outbound = result.remote_outbound_audio_rtp; + assert.equal(outbound.status, "observed"); + assert.equal(outbound.requests, 13); + assert.equal(outbound.samples.length, 13); + assert.ok(outbound.samples.at(-1).bytesSent > outbound.samples[0].bytesSent); + assert.doesNotMatch(JSON.stringify(outbound), /PRIVATE|remote-out|codecId|candidate|sdp/); +}); + +test("a missing remote-outbound report stays diagnostic and keeps the 440 Hz gate unchanged", async () => { + const state = audioProbeFixture(); + const result = await state.run(); + assert.equal(controlledTonePresent(result), true); + assert.equal(result.remote_outbound_audio_rtp.status, "report_unavailable"); + assert.ok(result.remote_outbound_audio_rtp.samples.every(sample => !Object.hasOwn(sample, "bytesSent"))); +}); + +test("audio ICE pair bytes come from the same getStats report and never change tone acceptance", async () => { + const state = audioProbeFixture({ stats(state) { + const reports = rtpReport(state); + reports.set("pair", { id: "pair", type: "candidate-pair", state: "succeeded", nominated: true, + bytesReceived: 8000 + state.now, bytesSent: 400 + state.now, + packetsReceived: 40 + state.now / 10, packetsSent: 4, + currentRoundTripTime: 0.02, availableIncomingBitrate: 120000, + remoteCandidateId: "remote-cand", localCandidateId: "local-cand", + ip: "203.0.113.9", address: "203.0.113.9", candidate: "PRIVATE-CANDIDATE" }); + reports.set("remote-cand", { id: "remote-cand", type: "remote-candidate", port: 49160, + candidateType: "relay", address: "203.0.113.9", ip: "203.0.113.9", + relatedAddress: "198.51.100.4", usernameFragment: "PRIVATE-ICE" }); + return reports; + } }); + const result = await state.run(); + assert.equal(controlledTonePresent(result), true); + const ice = result.audio_ice_pair; + assert.equal(ice.status, "observed"); + assert.equal(ice.requests, 13); + assert.equal(ice.samples.length, 13); + assert.equal(ice.samples[0].remote_port, 49160); + assert.equal(ice.samples[0].remote_candidate_type, "relay"); + assert.ok(ice.samples.at(-1).bytesReceived > ice.samples[0].bytesReceived); + assert.doesNotMatch(JSON.stringify(ice), /PRIVATE|203\.0\.113|198\.51\.100|pair|remote-cand|usernameFragment|relatedAddress/); +}); + +test("missing or ambiguous ICE pairs stay diagnostic and keep the 440 Hz gate unchanged", async () => { + for (const kind of ["missing", "ambiguous"]) { + const state = audioProbeFixture({ stats(state) { + const reports = rtpReport(state); + if (kind === "ambiguous") { + reports.set("pair-a", { id: "pair-a", type: "candidate-pair", state: "succeeded", + nominated: true, bytesReceived: 1 }); + reports.set("pair-b", { id: "pair-b", type: "candidate-pair", state: "succeeded", + nominated: true, bytesReceived: 2 }); + } + return reports; + } }); + const result = await state.run(); + assert.equal(controlledTonePresent(result), true); + assert.equal(result.audio_ice_pair.status, "report_unavailable"); + assert.ok(result.audio_ice_pair.samples.every(sample => !Object.hasOwn(sample, "bytesReceived"))); + } +}); + test("RTP samples use the exact observed receiver, preserve zero counters and leave PCM cadence unchanged", async () => { const state = audioProbeFixture({ setup(state) { new state.window.RTCPeerConnection([{ track: { ...state.track }, getStats() { From 2fef812c0fc6fad29a9a7b6d4131aa26278105d8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 03:19:06 -0500 Subject: [PATCH 332/535] fix(publisher): promote the release publication head-last from verified staging The Publisher export copied release-head.json before release.json, install.sh and the artifacts. A failed or colliding copy left the new head advertised over the previous release bytes, and a directory at release.json was copied into instead of rejected. export_release_publication now verifies the prepared set against its own metadata, stages a byte-checked copy, then promotes by per-file rename with release-head.json last. A failed rename restores the previous files from hard links kept in the attempt scratch. --- scripts/publish-platform-artifacts-test.py | 442 +++++++++++++++++++++ scripts/publish-release.sh | 228 ++++++++++- 2 files changed, 664 insertions(+), 6 deletions(-) diff --git a/scripts/publish-platform-artifacts-test.py b/scripts/publish-platform-artifacts-test.py index 7d0e38211..e847e4ffe 100644 --- a/scripts/publish-platform-artifacts-test.py +++ b/scripts/publish-platform-artifacts-test.py @@ -6,6 +6,7 @@ import importlib.util import json from pathlib import Path +import shutil import subprocess import tempfile import tarfile @@ -19,6 +20,114 @@ integrity = importlib.util.module_from_spec(spec) spec.loader.exec_module(integrity) +RELEASE_PLATFORMS = {"x86_64-linux": "linux-amd64", "aarch64-linux": "linux-arm64", "aarch64-darwin": "darwin-arm64"} +PUBLICATION_COMMANDS = ("elastos", "ipfs-provider", "ipfs", "curl", "cargo", "cloudflared", "git") + + +def sha256(data): + return hashlib.sha256(data).hexdigest() + + +def publication_fixture(prepared, salt=b"", version="0.7.1", channel="stable"): + """Write a tiny three-platform publication set and return the served files it advertises.""" + artifacts = prepared / "artifacts" + artifacts.mkdir(parents=True) + files = {"home.tar.gz": b"universal app" + salt, "shell-capsule-metadata.tar.gz": b"provider metadata" + salt} + for platform, setup in RELEASE_PLATFORMS.items(): + files[f"elastos-{platform}"] = b"runtime " + platform.encode() + salt + files[f"shell-{setup}"] = b"native shell " + platform.encode() + salt + files[f"shell-{platform}.capsule.tar.gz"] = b"shell capsule " + platform.encode() + salt + + def descriptor(name, install_path, extract_path=None): + record = {"release_path": name, "install_path": install_path, "cid": "bafy-" + name, + "checksum": "sha256:" + sha256(files[name]), "size": len(files[name])} + if extract_path: + record["extract_path"] = extract_path + return record + + platforms = {} + for platform, setup in RELEASE_PLATFORMS.items(): + capsule = files[f"shell-{platform}.capsule.tar.gz"] + manifest = { + "schema": "elastos.components/v1", + "capsules": {"shell": {"cid": "bafy-shell", "sha256": sha256(capsule), "size": len(capsule), + "platforms": [platform]}}, + "external": { + "home": {"platforms": {"*": descriptor("home.tar.gz", "capsules/home", "home")}}, + "shell": {"provider_runtime": {}, + "platforms": {setup: descriptor(f"shell-{setup}", "bin/shell")}, + "capsule_metadata": {"install_path": "capsules/shell", "platforms": { + "*": descriptor("shell-capsule-metadata.tar.gz", "capsules/shell", "shell")}}}, + "kubo": {"platforms": {"*": {"url": "https://example.invalid/kubo.tar.gz", "install_path": "bin/ipfs", + "checksum": "sha256:" + "d" * 64, "size": 100}}}, + }, + "profiles": {"home": {"components": ["home", "shell", "kubo"]}}, + } + files[f"components-{platform}.json"] = json.dumps(manifest, indent=2).encode() + platforms[platform] = { + "binary": {"cid": "bafy-" + platform, "sha256": sha256(files[f"elastos-{platform}"]), + "size": len(files[f"elastos-{platform}"])}, + "components": {"cid": "bafy-components-" + platform, "sha256": sha256(files[f"components-{platform}.json"]), + "size": len(files[f"components-{platform}.json"])}, + } + for name, data in files.items(): + (artifacts / name).write_bytes(data) + release = json.dumps({"payload": {"schema": "elastos.release/v1", "channel": channel, "version": version, + "released_at": 1, "prev_release_cid": None, "platforms": platforms}, + "signature": "fixture", "signer_did": "did:key:fixture"}).encode() + head = json.dumps({"payload": {"schema": "elastos.release.head/v1", "channel": channel, "version": version, + "latest_release_cid": "bafy-release", "release_sha256": sha256(release), + "signer_did": "did:key:fixture"}, + "signature": "fixture", "signer_did": "did:key:fixture"}).encode() + install = b"#!/bin/sh\necho install" + salt + b"\n" + (prepared / "release.json").write_bytes(release) + (prepared / "release-head.json").write_bytes(head) + (prepared / "install.sh").write_bytes(install) + served = {"release.json": release, "release-head.json": head, "install.sh": install} + served.update({"artifacts/" + name: data for name, data in files.items()}) + return served + + +def rebind_head(prepared, **payload_changes): + """Rebind release-head.json to the current release.json bytes, then apply payload edits.""" + head = json.loads((prepared / "release-head.json").read_bytes()) + head["payload"]["release_sha256"] = sha256((prepared / "release.json").read_bytes()) + head["payload"].update(payload_changes) + (prepared / "release-head.json").write_bytes(json.dumps(head).encode()) + + +def rewrite_release(prepared, edit): + release = json.loads((prepared / "release.json").read_bytes()) + edit(release) + (prepared / "release.json").write_bytes(json.dumps(release).encode()) + rebind_head(prepared) + + +def snapshot(root): + """Byte-level view of a tree without following links, so unchanged means identical.""" + view = {} + + def visit(directory): + for entry in sorted(directory.iterdir()): + key = str(entry.relative_to(root)) + if entry.is_symlink(): + view[key] = ("link", os.readlink(entry)) + elif entry.is_dir(): + view[key] = ("dir",) + visit(entry) + else: + view[key] = ("file", entry.read_bytes()) + + if root.is_dir() and not root.is_symlink(): + visit(root) + return view + + +def served_view(files): + view = {"artifacts": ("dir",)} + view.update({name: ("file", data) for name, data in files.items()}) + return view + class PlatformArtifactExportTest(unittest.TestCase): def test_source_provider_metadata_archives_pass_admission(self): @@ -532,6 +641,339 @@ def test_host_and_cross_manifests_keep_their_platform_names(self): if cross: self.assertEqual((outputs / f"components-{cross}.json").read_text(), f"cross:{cross}") + def run_publication_export(self, prepared, publisher_root, stubs=""): + """Export a prepared set into a disposable Publisher root with real commands stubbed out.""" + with tempfile.TemporaryDirectory() as commands: + commands = Path(commands) + effects = commands / "effects" + for command in PUBLICATION_COMMANDS: + stub = commands / command + stub.write_text('#!/bin/sh\nprintf "%s\\n" "$0" >> "$TEST_EFFECT_LOG"\nexit 93\n') + stub.chmod(0o755) + result = subprocess.run( + ["bash", "-euc", 'source "$1"; shift\n' + stubs + ''' +export_release_publication "$1" "$2/release-head.json" "$2/release.json" "$2/install.sh" "$2/artifacts" +''', "publication-test", str(PUBLISHER), str(publisher_root), str(prepared)], + cwd=PUBLISHER.parent.parent, + env={**os.environ, "PATH": str(commands) + os.pathsep + os.environ["PATH"], + "TEST_EFFECT_LOG": str(effects), "LC_ALL": "C"}, + capture_output=True, text=True, + ) + self.assertFalse(effects.exists(), "publication export invoked a real publication command") + return result + + def test_publication_export_promotes_exact_advertised_set_head_last(self): + source = PUBLISHER.read_text() + start = source.index("# Save release metadata to runtime-owned publisher state") + call_site = source[start:source.index('info "Saved release artifacts', start)] + self.assertIn('export_release_publication "$PUBLISHER_ROOT"', call_site) + self.assertNotIn("cp ", call_site) + with tempfile.TemporaryDirectory() as root: + root = Path(root) + publisher = root / "data/ElastOS/SystemServices/Publisher" + previous = publication_fixture(root / "previous", salt=b" previous") + result = self.run_publication_export(root / "previous", publisher) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(snapshot(publisher), served_view(previous)) + expected = publication_fixture(root / "prepared", salt=b" next") + moves = root / "moves" + for attempt in ("publish", "identical retry"): + with self.subTest(attempt=attempt): + moves.unlink(missing_ok=True) + result = self.run_publication_export(root / "prepared", publisher, + 'mv() { printf "%s\\n" "${@: -1}" >> "$TEST_MOVES"; command mv "$@"; }\nexport TEST_MOVES="' + + str(moves) + '"\n') + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stderr, "") + self.assertEqual(snapshot(publisher), served_view(expected)) + order = [os.path.relpath(line, publisher) for line in moves.read_text().splitlines()] + artifacts = sorted(name for name in expected if name.startswith("artifacts/")) + self.assertEqual(order, artifacts + ["install.sh", "release.json", "release-head.json"]) + + def test_publication_export_rejects_incomplete_or_mismatched_sets_before_mutation(self): + def replace(name, data): + return lambda prepared: (prepared / name).write_bytes(data) + + def corrupt(name): + def edit(prepared): + path = prepared / "artifacts" / name + path.write_bytes(bytes(len(path.read_bytes()))) + return edit + + def grow(name): + def edit(prepared): + path = prepared / "artifacts" / name + path.write_bytes(path.read_bytes() + b"!") + return edit + + def unknown_platform(release): + release["payload"]["platforms"]["riscv64-linux"] = release["payload"]["platforms"]["x86_64-linux"] + + def no_platforms(release): + release["payload"]["platforms"] = {} + + def unsigned(release): + del release["signature"] + + cases = { + "missing-app": ((lambda p: (p / "artifacts/home.tar.gz").unlink()), "home.tar.gz"), + "missing-runtime": ((lambda p: (p / "artifacts/elastos-aarch64-linux").unlink()), "elastos-aarch64-linux"), + "missing-components": ((lambda p: (p / "artifacts/components-x86_64-linux.json").unlink()), + "components-x86_64-linux.json"), + "missing-install": ((lambda p: (p / "install.sh").unlink()), "install.sh must be a regular file"), + "empty-install": (replace("install.sh", b""), "install.sh is empty"), + "malformed-head": (replace("release-head.json", b"{"), "release-head.json is not valid JSON"), + "malformed-release": (replace("release.json", b"not json"), "release.json is not valid JSON"), + "wrong-head-schema": (replace("release-head.json", json.dumps( + {"payload": {"schema": "elastos.release/v1"}, "signature": "s", "signer_did": "d"}).encode()), + "release-head.json is not a elastos.release.head/v1 envelope"), + "unsigned-release": ((lambda p: rewrite_release(p, unsigned)), "release.json is missing its signature"), + "stale-head-binding": ((lambda p: rebind_head(p, release_sha256="0" * 64)), "does not bind"), + "head-version": ((lambda p: rebind_head(p, version="0.7.2")), "version differ"), + "head-channel": ((lambda p: rebind_head(p, channel="canary")), "channel differ"), + "runtime-hash": (corrupt("elastos-x86_64-linux"), "advertised binary differs from its bytes"), + "runtime-size": (grow("elastos-aarch64-darwin"), "advertised binary differs from its bytes"), + "components-hash": (corrupt("components-aarch64-linux.json"), "advertised components differs"), + "app-hash": (corrupt("home.tar.gz"), "artifact checksum mismatch"), + "app-size": (grow("home.tar.gz"), "artifact size mismatch"), + "native-hash": (corrupt("shell-linux-arm64"), "artifact checksum mismatch"), + "capsule-hash": (corrupt("shell-aarch64-darwin.capsule.tar.gz"), "artifact checksum mismatch"), + "provider-metadata-size": (grow("shell-capsule-metadata.tar.gz"), "artifact size mismatch"), + "unadvertised-extra": (replace("artifacts/elastos-riscv64-linux", b"stray"), "not advertised"), + "unknown-platform": ((lambda p: rewrite_release(p, unknown_platform)), "unknown platform"), + "no-platforms": ((lambda p: rewrite_release(p, no_platforms)), "advertises no platforms"), + } + with tempfile.TemporaryDirectory() as root: + root = Path(root) + publisher = root / "publisher" + publication_fixture(root / "previous", salt=b" previous") + self.assertEqual(self.run_publication_export(root / "previous", publisher).returncode, 0) + before = snapshot(publisher) + for name, (mutate, message) in cases.items(): + with self.subTest(failure=name): + prepared = root / f"prepared-{name}" + publication_fixture(prepared, salt=b" next") + mutate(prepared) + result = self.run_publication_export(prepared, publisher) + self.assertNotEqual(result.returncode, 0, name) + self.assertIn("Release publication set rejected", result.stderr) + self.assertIn(message, result.stderr) + self.assertEqual(snapshot(publisher), before) + + def test_publication_export_rejects_unsafe_paths_and_collisions_before_mutation(self): + def link_artifact(prepared, publisher, outside): + path = prepared / "artifacts/home.tar.gz" + outside.write_bytes(path.read_bytes()) + path.unlink() + path.symlink_to(outside) + + def link_artifact_dir(prepared, publisher, outside): + (prepared / "artifacts").rename(prepared / "artifacts-real") + (prepared / "artifacts").symlink_to(prepared / "artifacts-real") + + def link_release(prepared, publisher, outside): + outside.write_bytes((prepared / "release.json").read_bytes()) + (prepared / "release.json").unlink() + (prepared / "release.json").symlink_to(outside) + + def nested_dir(prepared, publisher, outside): + (prepared / "artifacts/nested").mkdir() + + def hidden_file(prepared, publisher, outside): + (prepared / "artifacts/.hidden").write_bytes(b"hidden") + + def release_dir(prepared, publisher, outside): + (publisher / "release.json").unlink() + (publisher / "release.json").mkdir() + + def head_link(prepared, publisher, outside): + outside.write_bytes(b"outside head") + (publisher / "release-head.json").unlink() + (publisher / "release-head.json").symlink_to(outside) + + def artifacts_file(prepared, publisher, outside): + shutil.rmtree(publisher / "artifacts") + (publisher / "artifacts").write_bytes(b"not a directory") + + def artifacts_link(prepared, publisher, outside): + (publisher / "artifacts").rename(publisher / "elsewhere") + (publisher / "artifacts").symlink_to(publisher / "elsewhere") + + def artifact_dir_collision(prepared, publisher, outside): + (publisher / "artifacts/home.tar.gz").unlink() + (publisher / "artifacts/home.tar.gz").mkdir() + + def root_file(prepared, publisher, outside): + shutil.rmtree(publisher) + publisher.write_bytes(b"not a directory") + + cases = { + "symlink-artifact": (link_artifact, "plain regular files"), + "symlink-artifact-dir": (link_artifact_dir, "artifact directory must be a regular directory"), + "symlink-release": (link_release, "release.json must be a regular file"), + "nested-directory": (nested_dir, "plain regular files"), + "hidden-file": (hidden_file, "plain regular files"), + "destination-release-dir": (release_dir, "Publisher destination must be a regular file or absent"), + "destination-head-link": (head_link, "Publisher destination must be a regular file or absent"), + "destination-artifacts-file": (artifacts_file, "Publisher artifacts path must be a regular directory"), + "destination-artifacts-link": (artifacts_link, "Publisher artifacts path must be a regular directory"), + "destination-artifact-dir": (artifact_dir_collision, "Publisher destination must be a regular file or absent"), + "destination-root-file": (root_file, "Publisher root is not a directory"), + } + with tempfile.TemporaryDirectory() as root: + root = Path(root) + for name, (arrange, message) in cases.items(): + with self.subTest(failure=name): + publisher = root / f"publisher-{name}" + previous = root / f"previous-{name}" + publication_fixture(previous, salt=b" previous") + self.assertEqual(self.run_publication_export(previous, publisher).returncode, 0) + prepared = root / f"prepared-{name}" + publication_fixture(prepared, salt=b" next") + outside = root / f"outside-{name}" + arrange(prepared, publisher, outside) + untouched = lambda: {k: v for k, v in snapshot(root).items() if not k.startswith(f"prepared-{name}")} + before = untouched() + result = self.run_publication_export(prepared, publisher) + self.assertNotEqual(result.returncode, 0, name) + self.assertIn(message, result.stderr) + self.assertEqual(untouched(), before) + + def test_publication_staging_failure_keeps_publication_and_foreign_scratch(self): + stubs = { + "copy-error": 'cp() { case "$1" in */release.json) return 93;; esac; command cp "$@"; }\n', + "short-write": 'cp() { case "$1" in */elastos-aarch64-linux) printf short > "$2";; *) command cp "$@";; esac; }\n', + } + with tempfile.TemporaryDirectory() as root: + root = Path(root) + publisher = root / "publisher" + publication_fixture(root / "previous", salt=b" previous") + self.assertEqual(self.run_publication_export(root / "previous", publisher).returncode, 0) + foreign = publisher / ".publish-release.foreign/staged" + foreign.mkdir(parents=True) + (foreign / "release.json").write_bytes(b"another attempt") + before = snapshot(publisher) + publication_fixture(root / "prepared", salt=b" next") + for failure, stub in stubs.items(): + with self.subTest(failure=failure): + result = self.run_publication_export(root / "prepared", publisher, stub) + self.assertNotEqual(result.returncode, 0) + self.assertIn("the current publication is unchanged", result.stderr) + self.assertIn(".publish-release.foreign", result.stdout) + self.assertEqual(snapshot(publisher), before) + + def test_publication_promotion_failure_restores_previous_and_never_advertises_new_head(self): + boundaries = ("artifacts/components-aarch64-darwin.json", "artifacts/elastos-aarch64-linux", + "artifacts/shell-x86_64-linux.capsule.tar.gz", "install.sh", "release.json", "release-head.json") + with tempfile.TemporaryDirectory() as root: + root = Path(root) + publisher = root / "publisher" + publication_fixture(root / "previous", salt=b" previous") + self.assertEqual(self.run_publication_export(root / "previous", publisher).returncode, 0) + before = snapshot(publisher) + expected = served_view(publication_fixture(root / "prepared", salt=b" next")) + for boundary in boundaries: + with self.subTest(boundary=boundary): + result = self.run_publication_export(root / "prepared", publisher, + 'mv() { case "$2" in */staged/%s) return 93;; esac; command mv "$@"; }\n' % boundary) + self.assertNotEqual(result.returncode, 0) + self.assertIn("the previous publication was restored", result.stderr) + self.assertEqual(snapshot(publisher), before) + with self.subTest(boundary="hard link of the current file"): + result = self.run_publication_export(root / "prepared", publisher, 'ln() { return 93; }\n') + self.assertNotEqual(result.returncode, 0) + self.assertIn("the previous publication was restored", result.stderr) + self.assertEqual(snapshot(publisher), before) + with self.subTest(boundary="restore itself fails"): + # The attempt keeps its scratch, which still holds the previous bytes it could not put back. + result = self.run_publication_export(root / "prepared", publisher, + 'mv() { case "$2" in */staged/release.json|*/previous/artifacts/home.tar.gz) return 93;; esac;' + ' command mv "$@"; }\n') + self.assertNotEqual(result.returncode, 0) + self.assertIn("could not restore every previous file", result.stderr) + after = snapshot(publisher) + scratch = sorted(key for key in after if key.startswith(".publish-release.")) + self.assertTrue(scratch) + kept = [key for key in scratch if key.endswith("/previous/artifacts/home.tar.gz")] + self.assertEqual([after[key] for key in kept], [before["artifacts/home.tar.gz"]]) + self.assertEqual({key: value for key, value in after.items() if key not in scratch}, + {**before, "artifacts/home.tar.gz": expected["artifacts/home.tar.gz"]}) + for key in scratch[::-1]: + path = publisher / key + path.rmdir() if path.is_dir() else path.unlink() + shutil.copyfile(root / "previous/artifacts/home.tar.gz", publisher / "artifacts/home.tar.gz") + self.assertEqual(snapshot(publisher), before) + # A first publication has no previous files; a failed promotion removes what it created. + fresh = root / "fresh" + result = self.run_publication_export(root / "prepared", fresh, + 'mv() { case "$2" in */staged/release.json) return 93;; esac; command mv "$@"; }\n') + self.assertNotEqual(result.returncode, 0) + self.assertEqual(snapshot(fresh), {"artifacts": ("dir",)}) + + def test_promotion_records_pending_change_before_rename(self): + # Bookkeeping that fails after a rename must not hide the replaced file + # from restore; recording first keeps every replaced path restorable. + pending = 'printf() { if [[ "${2-}" == %s ]]; then return 93; fi; builtin printf "$@"; }\n' + with tempfile.TemporaryDirectory() as root: + root = Path(root) + publisher = root / "publisher" + publication_fixture(root / "previous", salt=b" previous") + self.assertEqual(self.run_publication_export(root / "previous", publisher).returncode, 0) + before = snapshot(publisher) + expected = served_view(publication_fixture(root / "prepared", salt=b" next")) + for relative in ("artifacts/home.tar.gz", "release.json", "release-head.json"): + with self.subTest(pending_record=relative): + result = self.run_publication_export(root / "prepared", publisher, pending % relative) + self.assertNotEqual(result.returncode, 0) + self.assertIn("the previous publication was restored", result.stderr) + self.assertEqual(snapshot(publisher), before) + with self.subTest(pending_record="restore of an earlier file fails"): + result = self.run_publication_export(root / "prepared", publisher, pending % "release.json" + + 'mv() { case "$2" in */previous/artifacts/home.tar.gz) return 93;; esac; command mv "$@"; }\n') + self.assertNotEqual(result.returncode, 0) + self.assertIn("could not restore every previous file", result.stderr) + after = snapshot(publisher) + scratch = sorted(key for key in after if key.startswith(".publish-release.")) + kept = [key for key in scratch if key.endswith("/previous/artifacts/home.tar.gz")] + self.assertEqual([after[key] for key in kept], [before["artifacts/home.tar.gz"]]) + self.assertEqual({key: value for key, value in after.items() if key not in scratch}, + {**before, "artifacts/home.tar.gz": expected["artifacts/home.tar.gz"]}) + self.assertEqual(after["release-head.json"], before["release-head.json"]) + + def test_killed_promotion_leaves_old_head_over_mixed_artifacts_until_retry(self): + # A shell error rolls back; a killed process does not. Per-file renames are + # atomic, so the observable state is the promotion order cut at the kill: + # replaced paths hold complete new bytes, the rest hold the previous bytes, + # and the previous head stays advertised. A reader holding that head fails + # closed on replaced artifact checksums until an identical retry completes. + with tempfile.TemporaryDirectory() as root: + root = Path(root) + publisher = root / "publisher" + previous = served_view(publication_fixture(root / "previous", salt=b" previous")) + expected = served_view(publication_fixture(root / "prepared", salt=b" next")) + order = sorted(name for name in expected if name.startswith("artifacts/")) + [ + "install.sh", "release.json", "release-head.json"] + for boundary in ("artifacts/elastos-aarch64-linux", "release-head.json"): + with self.subTest(boundary=boundary): + shutil.rmtree(publisher, ignore_errors=True) + self.assertEqual(self.run_publication_export(root / "previous", publisher).returncode, 0) + result = self.run_publication_export(root / "prepared", publisher, + 'mv() { case "$2" in */staged/%s) kill -9 $$;; esac; command mv "$@"; }\n' % boundary) + self.assertEqual(result.returncode, -9) + replaced = set(order[:order.index(boundary)]) + after = snapshot(publisher) + scratch = {key for key in after if key.startswith(".publish-release.")} + self.assertTrue(scratch, "the killed attempt leaves its staging behind") + self.assertEqual({key: value for key, value in after.items() if key not in scratch}, + {key: (expected if key in replaced else previous)[key] for key in previous}) + self.assertEqual(after["release-head.json"], previous["release-head.json"]) + result = self.run_publication_export(root / "prepared", publisher) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("interrupted publication attempt remains", result.stdout) + after = snapshot(publisher) + self.assertEqual({key: value for key, value in after.items() if key not in scratch}, expected) + self.assertEqual({key for key in after if key.startswith(".publish-release.")}, scratch) + if __name__ == "__main__": unittest.main() diff --git a/scripts/publish-release.sh b/scripts/publish-release.sh index 7f695ead9..32cb506df 100755 --- a/scripts/publish-release.sh +++ b/scripts/publish-release.sh @@ -1183,6 +1183,226 @@ publish_prepared_platform_inputs() { SHELL_CID='' SHELL_SHA256='' } +# The Publisher root serves stable paths: release-head.json, release.json, +# install.sh and artifacts/. A prepared set is checked against its own +# signed metadata, staged beside the current publication, then promoted by +# per-file rename with release-head.json last, so a reader sees the previous +# head until every advertised byte is in place. Renames are atomic per file, +# not per release: a reader that already holds the previous head can still meet +# replaced artifact bytes at a stable path and fails closed on its checksum. +verify_release_publication_set() { + local head="$1" release="$2" install="$3" artifact_dir="$4" + python3 - "$head" "$release" "$install" "$artifact_dir" <<'PY' +import hashlib +import importlib.util +import json +import stat +import sys +from pathlib import Path + +head_path, release_path, install_path, artifact_root = (Path(p) for p in sys.argv[1:5]) +spec = importlib.util.spec_from_file_location( + "components_integrity", "scripts/components-release-integrity-check.py") +integrity = importlib.util.module_from_spec(spec) +spec.loader.exec_module(integrity) + + +def regular(path, label): + if path.is_symlink() or not path.exists() or not stat.S_ISREG(path.stat().st_mode): + raise ValueError(f"{label} must be a regular file: {path}") + data = path.read_bytes() + if not data: + raise ValueError(f"{label} is empty: {path}") + return data + + +def envelope(data, schema, label): + try: + value = json.loads(data) + except ValueError as exc: + raise ValueError(f"{label} is not valid JSON: {exc}") + payload = value.get("payload") if isinstance(value, dict) else None + if not isinstance(payload, dict) or payload.get("schema") != schema: + raise ValueError(f"{label} is not a {schema} envelope") + if any(not isinstance(value.get(k), str) or not value[k] for k in ("signature", "signer_did")): + raise ValueError(f"{label} is missing its signature or signer") + return payload + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +try: + release_bytes = regular(release_path, "release.json") + head = envelope(regular(head_path, "release-head.json"), "elastos.release.head/v1", "release-head.json") + release = envelope(release_bytes, "elastos.release/v1", "release.json") + regular(install_path, "install.sh") + if head.get("release_sha256") != digest(release_bytes): + raise ValueError("release-head.json does not bind these release.json bytes") + for field in ("version", "channel"): + if not isinstance(head.get(field), str) or not head[field] or release.get(field) != head[field]: + raise ValueError(f"release-head.json and release.json {field} differ") + if artifact_root.is_symlink() or not artifact_root.is_dir(): + raise ValueError(f"artifact directory must be a regular directory: {artifact_root}") + present = {} + for entry in artifact_root.iterdir(): + if (entry.name.startswith(".") or "\\" in entry.name or entry.is_symlink() + or not stat.S_ISREG(entry.lstat().st_mode)): + raise ValueError(f"artifact entries must be plain regular files: {entry.name}") + present[entry.name] = entry + platforms = release.get("platforms") + if not isinstance(platforms, dict) or not platforms: + raise ValueError("release.json advertises no platforms") + referenced = set() + errors = [] + for platform, descriptor in sorted(platforms.items()): + setup = next((s for s, r in integrity.RELEASE_PLATFORMS.items() if r == platform), None) + if setup is None: + raise ValueError(f"release.json advertises an unknown platform: {platform}") + for kind, name in (("binary", f"elastos-{platform}"), ("components", f"components-{platform}.json")): + if name not in present: + raise ValueError(f"advertised artifact is missing: {name}") + data = present[name].read_bytes() + info = descriptor.get(kind) if isinstance(descriptor, dict) else None + if not isinstance(info, dict) or info.get("sha256") != digest(data) or info.get("size") != len(data): + raise ValueError(f"advertised {kind} differs from its bytes: {name}") + referenced.add(name) + try: + manifest = json.loads(present[f"components-{platform}.json"].read_bytes()) + except ValueError as exc: + raise ValueError(f"components-{platform}.json is not valid JSON: {exc}") + errors += integrity.audit_release_artifacts(manifest, [setup], artifact_root) + for component in (manifest.get("external") or {}).values(): + if not isinstance(component, dict): + continue + for entry in (component, component.get("capsule_metadata")): + if isinstance(entry, dict): + _, info = integrity.resolve_platform_info(entry, setup) + if isinstance(info, dict) and isinstance(info.get("release_path"), str): + referenced.add(info["release_path"]) + for name, entry in (manifest.get("capsules") or {}).items(): + if isinstance(entry, dict) and platform in (entry.get("platforms") or []): + referenced.add(f"{name}-{platform}.capsule.tar.gz") + if errors: + raise ValueError("; ".join(errors)) + extra = sorted(set(present) - referenced) + if extra: + raise ValueError(f"prepared artifacts are not advertised by this release: {extra}") +except (OSError, ValueError, TypeError, AttributeError) as exc: + raise SystemExit(f"Release publication set rejected: {exc}") +PY +} + +reject_release_publication_collision() { + if [[ -L "$1" ]] || [[ -e "$1" && ! -f "$1" ]]; then + die "Publisher destination must be a regular file or absent: $1" + fi +} + +check_release_publication_destinations() { + local publisher_root="$1" artifact_dir="$2" + local source destination + if [[ -e "$publisher_root" || -L "$publisher_root" ]] && [[ ! -d "$publisher_root" ]]; then + die "Publisher root is not a directory: ${publisher_root}" + fi + if [[ -L "${publisher_root}/artifacts" ]] || \ + [[ -e "${publisher_root}/artifacts" && ! -d "${publisher_root}/artifacts" ]]; then + die "Publisher artifacts path must be a regular directory: ${publisher_root}/artifacts" + fi + for source in "${artifact_dir}"/*; do + reject_release_publication_collision "${publisher_root}/artifacts/$(basename "$source")" + done + for destination in install.sh release.json release-head.json; do + reject_release_publication_collision "${publisher_root}/${destination}" + done +} + +copy_release_publication_file() { + cp "$1" "$2" || return + cmp -s "$1" "$2" +} + +stage_release_publication() { + local scratch="$1" head="$2" release="$3" install="$4" artifact_dir="$5" + local source + mkdir -p "${scratch}/staged/artifacts" "${scratch}/previous/artifacts" || return + for source in "${artifact_dir}"/*; do + copy_release_publication_file "$source" "${scratch}/staged/artifacts/$(basename "$source")" || return + done + copy_release_publication_file "$install" "${scratch}/staged/install.sh" || return + copy_release_publication_file "$release" "${scratch}/staged/release.json" || return + copy_release_publication_file "$head" "${scratch}/staged/release-head.json" || return +} + +# The current file keeps a hard link under the attempt's scratch so a failed +# rename can put it back without a moment where the path is absent. The path +# is recorded as pending before its rename, so restore always sees a replaced +# file even when bookkeeping fails afterwards. +promote_release_publication_file() { + local scratch="$1" relative="$2" destination="$3" + if [[ -f "$destination" ]]; then + ln "$destination" "${scratch}/previous/${relative}" || return + fi + printf '%s\n' "$relative" >> "${scratch}/promoted" || return + mv -f "${scratch}/staged/${relative}" "$destination" +} + +promote_release_publication() { + local scratch="$1" publisher_root="$2" + local staged relative + : > "${scratch}/promoted" || return + for staged in "${scratch}/staged/artifacts"/* "${scratch}/staged/install.sh" \ + "${scratch}/staged/release.json" "${scratch}/staged/release-head.json"; do + relative="${staged#"${scratch}/staged/"}" + promote_release_publication_file "$scratch" "$relative" "${publisher_root}/${relative}" || return + done +} + +restore_release_publication() { + local scratch="$1" publisher_root="$2" + local relative previous status=0 + [[ -f "${scratch}/promoted" ]] || return 0 + while IFS= read -r relative; do + previous="${scratch}/previous/${relative}" + if [[ -f "$previous" ]]; then + # A pending path whose rename never ran still shares the saved inode. + if [[ ! "$previous" -ef "${publisher_root}/${relative}" ]]; then + mv -f "$previous" "${publisher_root}/${relative}" || status=1 + fi + else + rm -f "${publisher_root}/${relative}" || status=1 + fi + done < "${scratch}/promoted" + return "$status" +} + +export_release_publication() { + local publisher_root="$1" head="$2" release="$3" install="$4" artifact_dir="$5" + local scratch stale + verify_release_publication_set "$head" "$release" "$install" "$artifact_dir" || return + check_release_publication_destinations "$publisher_root" "$artifact_dir" || return + mkdir -p "${publisher_root}/artifacts" || return + for stale in "${publisher_root}"/.publish-release.*; do + if [[ -e "$stale" ]]; then + warn "Staging from an interrupted publication attempt remains: ${stale}" + fi + done + scratch=$(mktemp -d "${publisher_root}/.publish-release.XXXXXX") || return + if ! stage_release_publication "$scratch" "$head" "$release" "$install" "$artifact_dir"; then + rm -rf "$scratch" + die "Failed to stage the release publication set; the current publication is unchanged" + fi + if ! promote_release_publication "$scratch" "$publisher_root"; then + if restore_release_publication "$scratch" "$publisher_root"; then + rm -rf "$scratch" + die "Failed to promote the release publication set; the previous publication was restored" + fi + die "Failed to promote the release publication set and could not restore every previous file; inspect ${scratch}" + fi + rm -rf "$scratch" +} + # Sourcing exposes local builders without invoking the publisher. if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then return 0 @@ -2441,12 +2661,8 @@ info "Installer CID: ${INSTALL_SCRIPT_CID}" # Save release metadata to runtime-owned publisher state for gateway serving. RUNTIME_DATA_DIR="${HOST_DATA_DIR}" PUBLISHER_ROOT="${RUNTIME_DATA_DIR}/ElastOS/SystemServices/Publisher" -PUBLISHER_ARTIFACTS_DIR="${PUBLISHER_ROOT}/artifacts" -mkdir -p "${PUBLISHER_ARTIFACTS_DIR}" -cp "${TMPDIR}/release-head.json" "${PUBLISHER_ROOT}/release-head.json" -cp "${TMPDIR}/release.json" "${PUBLISHER_ROOT}/release.json" -cp "$STAMPED_INSTALL" "${PUBLISHER_ROOT}/install.sh" -cp "${PREPARED_ARTIFACTS_DIR}"/* "${PUBLISHER_ARTIFACTS_DIR}/" +export_release_publication "$PUBLISHER_ROOT" \ + "${TMPDIR}/release-head.json" "${TMPDIR}/release.json" "$STAMPED_INSTALL" "$PREPARED_ARTIFACTS_DIR" info "Saved release artifacts to ${PUBLISHER_ROOT} for gateway serving" # ── Step 10: Start public gateway URL (best-effort) ────────────────── From aac9f26343acb6a2c642e222dae285ab57b42c8a Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 03:19:15 -0500 Subject: [PATCH 333/535] test(installer): prove artifact rejection before changes and a clean rerun run_offline_installer blocked every artifact request, so the complete installer tests stopped before the binary and components SHA-256 gates and never covered a failed download followed by a successful install. InstallerSandbox answers requests from response files in one disposable HOME. Truncated or wrong bytes fail at their SHA-256 stage with user data unchanged. The same sandbox then installs on a clean --install-only rerun. --- scripts/install-bootstrap-test.py | 336 +++++++++++++++++++++++++++--- 1 file changed, 309 insertions(+), 27 deletions(-) diff --git a/scripts/install-bootstrap-test.py b/scripts/install-bootstrap-test.py index 0d435f45e..929c103f2 100644 --- a/scripts/install-bootstrap-test.py +++ b/scripts/install-bootstrap-test.py @@ -13,6 +13,7 @@ import os from pathlib import Path import platform +import re import shlex import signal import subprocess @@ -55,6 +56,76 @@ def binding_fixture(): encode = lambda value: json.dumps(value, sort_keys=True, separators=(",", ":")).encode() + b"\n" return did, encode(head), encode(first), encode(second) + +TEST_SEED = bytes([7]) * 32 +BASE58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz" + + +def encode_point(point): + x, y, z, _ = point + inverse = pow(z, CRYPTO["FIELD"] - 2, CRYPTO["FIELD"]) + x, y = x * inverse % CRYPTO["FIELD"], y * inverse % CRYPTO["FIELD"] + return (y | (x & 1) << 255).to_bytes(32, "little") + + +def sign_envelope(payload, domain, seed=TEST_SEED): + """RFC 8032 signing over the installer's own verifier arithmetic; disposable test seed only.""" + expanded = hashlib.sha512(seed).digest() + scalar = int.from_bytes(expanded[:32], "little") & ((1 << 254) - 8) | 1 << 254 + public = encode_point(CRYPTO["point_mul"](scalar, CRYPTO["BASE"])) + canonical = json.dumps(payload, separators=(",", ":"), sort_keys=True, + ensure_ascii=False, allow_nan=False).encode("utf-8") + message = hashlib.sha256(domain.encode("utf-8") + b"\0" + canonical).digest() + nonce = int.from_bytes(hashlib.sha512(expanded[32:] + message).digest(), "little") % CRYPTO["ORDER"] + commitment = encode_point(CRYPTO["point_mul"](nonce, CRYPTO["BASE"])) + challenge = int.from_bytes(hashlib.sha512(commitment + public + message).digest(), "little") % CRYPTO["ORDER"] + signature = commitment + ((nonce + challenge * scalar) % CRYPTO["ORDER"]).to_bytes(32, "little") + number, encoded = int.from_bytes(b"\xed\x01" + public, "big"), "" + while number: + number, digit = divmod(number, 58) + encoded = BASE58[digit] + encoded + return {"payload": payload, "signer_did": "did:key:z" + encoded, "signature": signature.hex()} + + +def encode_envelope(envelope): + return json.dumps(envelope, sort_keys=True, separators=(",", ":")).encode() + b"\n" + + +# A harmless stand-in for the Runtime binary: it records every command it is +# asked to run and answers only the calls the installer is expected to make. +RUNTIME_STUB = b'''#!/bin/bash +printf '%s\\n' "$*" >> "${ELASTOS_TEST_CALLS:?fixture call log}" +case "${1:-}" in + --version) echo "elastos 0.7.1" ;; + principal-root-upgrade|setup|home) exit 0 ;; + *) exit 97 ;; +esac +''' +COMPONENTS = json.dumps({"schema": "elastos.components/v1", "capsules": {}, "external": {}, + "profiles": {}}, sort_keys=True).encode() + b"\n" +BOOTSTRAP = (b'{"schema":"elastos.carrier.bootstrap/v1","role":"publisher",' + b'"ticket":"fixture-ticket","node_id":"fixture-node"}\n') + + +def installable_fixture(runtime=RUNTIME_STUB, components=COMPONENTS, version="0.7.1"): + """Deterministic signed head/release advertising the fixture Runtime stub and manifest.""" + release_payload = { + "schema": "elastos.release/v1", "version": version, "channel": "stable", + "released_at": 1, "prev_release_cid": None, + "platforms": {"x86_64-linux": { + "binary": {"cid": "binary-a", "sha256": hashlib.sha256(runtime).hexdigest(), "size": len(runtime)}, + "components": {"cid": "components-a", "sha256": hashlib.sha256(components).hexdigest(), + "size": len(components)}}}} + signed_release = sign_envelope(release_payload, "elastos.release.v1") + release = encode_envelope(signed_release) + head_payload = { + "schema": "elastos.release.head/v1", "version": version, "channel": "stable", + "latest_release_cid": "release-a", "release_sha256": hashlib.sha256(release).hexdigest(), + "signer_did": signed_release["signer_did"], "updated_at": 1, "prev_head_cid": None} + head = encode_envelope(sign_envelope(head_payload, "elastos.release.head.v1")) + return signed_release["signer_did"], head, release + + # Public key, message and signature from RFC 8032 section 7.1, tests 1, 2, # 3 and SHA(abc). These fixed vectors exercise the implementation above. # https://www.rfc-editor.org/rfc/rfc8032.html#section-7.1 @@ -270,45 +341,107 @@ def test_release_identity_requires_matching_nonempty_strings(self): self.assertEqual(result.returncode == 0, accepted, result.stdout + result.stderr) -def run_offline_installer(head, release, did, transport="publisher", system="Linux", machine="x86_64"): - # Only transport and uname are replaced. The complete installer executes, - # and every artifact request fails before installation or external access. - with tempfile.TemporaryDirectory(prefix="installer-offline-") as directory: - root = Path(directory) - mockbin = root / "mocks" - mockbin.mkdir() - (root / "release-head.json").write_bytes(head) - (root / "release.json").write_bytes(release) - (root / "requests").write_text("") - (mockbin / "uname").write_text('#!/bin/sh\ncase "$1" in -s) echo "$MOCK_SYSTEM";; -m) echo "$MOCK_MACHINE";; esac\n') - (mockbin / "curl").write_text('''#!/bin/sh +class InstallerSandbox: + """Disposable HOME, data root, install dir, temp dir and logs for the complete installer. + + Only transport (curl) and platform reporting (uname) are replaced. Requests are + answered from the response files present under responses/; any other request, + and any request for a response that is absent, fails and is logged. + """ + + def __init__(self, head, release, did, system="Linux", machine="x86_64"): + self.directory = tempfile.TemporaryDirectory(prefix="installer-sandbox-") + self.root = Path(self.directory.name) + self.did, self.system, self.machine = did, system, machine + self.home = self.root / "home" + self.data = self.home / "xdg-data/elastos" + self.binary = self.home / ".local/bin/elastos" + self.calls = self.root / "calls" + self.responses = self.root / "responses" + for path in (self.home, self.root / "tmp", self.responses, self.root / "mocks"): + path.mkdir(parents=True) + (self.root / "requests").write_text("") + self.head_cid = "head-a" + self.release_cid = json.loads(head)["payload"]["latest_release_cid"] + platform_entry = json.loads(release)["payload"].get("platforms", {}).get("x86_64-linux", {}) + self.binary_cid = platform_entry.get("binary", {}).get("cid", "") + self.components_cid = platform_entry.get("components", {}).get("cid", "") + self.respond("release-head.json", head) + self.respond("release.json", release) + mocks = self.root / "mocks" + (mocks / "uname").write_text('#!/bin/sh\ncase "$1" in -s) echo "$MOCK_SYSTEM";; -m) echo "$MOCK_MACHINE";; esac\n') + (mocks / "curl").write_text('''#!/bin/sh destination="" while [ "$#" -gt 0 ]; do case "$1" in -o) destination="$2"; shift 2;; *) url="$1"; shift;; esac done printf '%s\\n' "$url" >> "$FIXTURES/requests" case "$url" in - */release-head.json|*/ipfs/head-a) cp "$FIXTURES/release-head.json" "$destination";; - */release.json|*/ipfs/"$FIXTURE_RELEASE_CID") cp "$FIXTURES/release.json" "$destination";; - *) echo artifact-request-blocked >&2; exit 93;; + */release-head.json|*/ipfs/"$FIXTURE_HEAD_CID") response=release-head.json;; + */release.json|*/ipfs/"$FIXTURE_RELEASE_CID") response=release.json;; + */artifacts/elastos-*|*/ipfs/"$FIXTURE_BINARY_CID") response=binary;; + */artifacts/components-*.json|*/ipfs/"$FIXTURE_COMPONENTS_CID") response=components;; + */.well-known/elastos/carrier-bootstrap.json*) response=bootstrap;; + *) echo unexpected-request-blocked >&2; exit 93;; esac +[ -f "$FIXTURES/responses/$response" ] || { echo artifact-request-blocked >&2; exit 93; } +if [ -n "$destination" ]; then cp "$FIXTURES/responses/$response" "$destination"; else cat "$FIXTURES/responses/$response"; fi ''') - for path in mockbin.iterdir(): + for path in mocks.iterdir(): path.chmod(0o755) - options = (["--publisher-gateway", "https://test.invalid"] if transport == "publisher" else - ["--gateway", "https://test.invalid", "--head-cid", "head-a"]) + + def __enter__(self): + return self + + def __exit__(self, *exc): + self.directory.cleanup() + + def respond(self, name, data): + (self.responses / name).write_bytes(data) + + def requests(self): + return (self.root / "requests").read_text().splitlines() + + def run(self, *options, transport="publisher"): + seen = len(self.requests()) + options = list(options) + (["--publisher-gateway", "https://test.invalid"] if transport == "publisher" + else ["--gateway", "https://test.invalid", "--head-cid", self.head_cid]) result = shell(''' -export HOME="$1/home" FIXTURES="$1" PATH="$1/mocks:$PATH" +export HOME="$1/home" XDG_DATA_HOME="$1/home/xdg-data" TMPDIR="$1/tmp" FIXTURES="$1" PATH="$1/mocks:$PATH" export ELASTOS_PUBLISHER_GATEWAY="" ELASTOS_HEAD_CID="" ELASTOS_IPFS_GATEWAYS="" -export ELASTOS_SOURCE_CONNECT_TICKET="" ELASTOS_PUBLISHER_NODE_ID="" -export MOCK_SYSTEM="$4" MOCK_MACHINE="$5" FIXTURE_RELEASE_CID="$6" -exec "$2" --noprofile --norc "$3" "${@:7}" -''', root, OPTIONS.bash, INSTALLER, system, machine, - json.loads(head)["payload"]["latest_release_cid"], - "--maintainer-did", did, *options) - if (root / "home/.local/bin/elastos").exists(): +export ELASTOS_SOURCE_CONNECT_TICKET="" ELASTOS_PUBLISHER_NODE_ID="" ELASTOS_INSTALL_ONLY="" +export ELASTOS_TEST_CALLS="$1/calls" MOCK_SYSTEM="$4" MOCK_MACHINE="$5" +export FIXTURE_HEAD_CID="$6" FIXTURE_RELEASE_CID="$7" FIXTURE_BINARY_CID="$8" FIXTURE_COMPONENTS_CID="$9" +exec "$2" --noprofile --norc "$3" "${@:10}" +''', self.root, OPTIONS.bash, INSTALLER, self.system, self.machine, self.head_cid, self.release_cid, + self.binary_cid, self.components_cid, "--maintainer-did", self.did, *options) + return result, self.requests()[seen:] + + def runtime_calls(self): + return self.calls.read_text().splitlines() if self.calls.exists() else [] + + def home_state(self): + """Bytes and mode of every file under HOME, so unchanged means identical.""" + state = {} + for path in sorted(self.home.rglob("*")): + key = str(path.relative_to(self.home)) + if path.is_symlink(): + state[key] = ("link", os.readlink(path)) + elif path.is_dir(): + state[key] = ("dir",) + else: + state[key] = ("file", path.stat().st_mode & 0o777, path.read_bytes()) + return state + + +def run_offline_installer(head, release, did, transport="publisher", system="Linux", machine="x86_64"): + # Only transport and uname are replaced. The complete installer executes, + # and every artifact request fails before installation or external access. + with InstallerSandbox(head, release, did, system, machine) as sandbox: + result, requests = sandbox.run(transport=transport) + if sandbox.binary.exists(): raise AssertionError("Offline fixture reached installation") - return result, (root / "requests").read_text().splitlines() + return result, requests class CompletionTests(unittest.TestCase): @@ -471,6 +604,155 @@ def test_publisher_hashes_the_final_envelope_bytes_into_head_payload(self): self.assertEqual(payload["latest_release_cid"], "release-a") +class InstallationTests(unittest.TestCase): + """The complete installer against served artifact bytes inside one disposable sandbox.""" + + REQUESTS = { + "publisher": ["https://test.invalid/release-head.json", "https://test.invalid/release.json", + "https://test.invalid/artifacts/elastos-x86_64-linux", + "https://test.invalid/artifacts/components-x86_64-linux.json", + "https://test.invalid/.well-known/elastos/carrier-bootstrap.json?role=publisher"], + "cid": ["https://test.invalid/ipfs/head-a", "https://test.invalid/ipfs/release-a", + "https://test.invalid/ipfs/binary-a", "https://test.invalid/ipfs/components-a"], + } + + def test_test_signer_reproduces_fixed_fixture_vectors(self): + # The signer must agree with the fixed OpenSSL vectors that anchor binding_fixture(), + # and every envelope it produces must pass the installer's own verifier. + did, head, first, second = binding_fixture() + for raw, domain in [(first, "elastos.release.v1"), (second, "elastos.release.v1"), + (head, "elastos.release.head.v1")]: + envelope = json.loads(raw) + signed = sign_envelope(envelope["payload"], domain) + self.assertEqual((signed["signer_did"], signed["signature"]), (did, envelope["signature"])) + signer, head, release = installable_fixture() + self.assertEqual(signer, did) + with tempfile.TemporaryDirectory(prefix="installer-signer-") as directory: + for name, data, domain in [("head.json", head, "elastos.release.head.v1"), + ("release.json", release, "elastos.release.v1")]: + Path(directory, name).write_bytes(data) + CRYPTO["verify_envelope"](Path(directory, name), domain, did) + self.assertEqual(json.loads(head)["payload"]["release_sha256"], hashlib.sha256(release).hexdigest()) + + def existing_installation(self, sandbox): + sandbox.binary.parent.mkdir(parents=True) + sandbox.binary.write_bytes(b"#!/bin/sh\necho previous runtime\n") + sandbox.binary.chmod(0o755) + files = { + "components.json": b'{"schema":"elastos.components/v1","capsules":{},"note":"previous"}\n', + "sources.json": b'{"schema":"elastos.trusted-sources/v1","note":"previous"}\n', + "Users/alice/notes.txt": b"user data stays\n", + "capsules/shell/cache.bin": b"cached capsule stays\n", + "ElastOS/SystemServices/Publisher/release-head.json": b"previous head\n", + "ElastOS/SystemServices/Publisher/release.json": b"previous release\n", + } + for name, data in files.items(): + (sandbox.data / name).parent.mkdir(parents=True, exist_ok=True) + (sandbox.data / name).write_bytes(data) + + def assert_no_installation_effects(self, sandbox, before, result): + self.assertNotEqual(result.returncode, 0) + self.assertNotIn("Stopping verified Runtime processes", result.stdout) + self.assertEqual(sandbox.runtime_calls(), []) + self.assertEqual(sandbox.home_state(), before) + self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + + def test_corrupt_artifacts_fail_before_changes_then_clean_rerun_installs(self): + did, head, release = installable_fixture() + cases = [ + ("truncated binary", RUNTIME_STUB[:-9], COMPONENTS, "binary"), + ("wrong binary bytes", RUNTIME_STUB.replace(b"0.7.1", b"0.7.2"), COMPONENTS, "binary"), + ("wrong components bytes", RUNTIME_STUB, COMPONENTS.replace(b"{}", b"{ }", 1), "components"), + ] + for transport in ["publisher", "cid"]: + with InstallerSandbox(head, release, did) as sandbox: + self.existing_installation(sandbox) + sandbox.respond("bootstrap", BOOTSTRAP) + before = sandbox.home_state() + expected = self.REQUESTS[transport] + for name, binary, components, stage in cases: + sandbox.respond("binary", binary) + sandbox.respond("components", components) + result, requests = sandbox.run("--install-only", transport=transport) + with self.subTest(transport=transport, case=name): + self.assertIn("SHA-256 mismatch", result.stderr) + if stage == "binary": + self.assertIn("Verifying binary SHA-256", result.stdout) + self.assertNotIn("Downloading components.json", result.stdout) + self.assertEqual(requests, expected[:3]) + else: + self.assertIn("Verifying components.json SHA-256", result.stdout) + self.assertEqual(requests, expected[:4]) + self.assert_no_installation_effects(sandbox, before, result) + sandbox.respond("binary", RUNTIME_STUB) + sandbox.respond("components", COMPONENTS) + result, requests = sandbox.run("--install-only", transport=transport) + with self.subTest(transport=transport, case="clean rerun"): + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(requests, expected) + self.assertIn("Runtime installed:", result.stdout) + self.assertNotIn("Setting up Home", result.stdout) + advertised = json.loads(release)["payload"]["platforms"]["x86_64-linux"] + installed = sandbox.binary.read_bytes() + self.assertEqual(installed, RUNTIME_STUB) + self.assertEqual(hashlib.sha256(installed).hexdigest(), advertised["binary"]["sha256"]) + self.assertTrue(sandbox.binary.stat().st_mode & 0o100) + self.assertTrue(os.access(sandbox.binary, os.X_OK)) + self.assertFalse((sandbox.binary.parent / ".elastos.install.tmp").exists()) + manifest = (sandbox.data / "components.json").read_bytes() + self.assertEqual(manifest, COMPONENTS) + self.assertEqual(hashlib.sha256(manifest).hexdigest(), advertised["components"]["sha256"]) + calls = sandbox.runtime_calls() + self.assertEqual(calls[0], "--version") + self.assertRegex(calls[1], "^principal-root-upgrade --data-dir %s --backup-dir %s/backups/principal-root-upgrade-[0-9]+-[0-9]+$" + % (re.escape(str(sandbox.data)), re.escape(str(sandbox.data)))) + self.assertEqual(len(calls), 2, "setup and Home launch stay out of --install-only") + sources = json.loads((sandbox.data / "sources.json").read_text()) + self.assertEqual(sources["schema"], "elastos.trusted-sources/v1") + source = sources["sources"][0] + self.assertEqual((source["publisher_dids"], source["channel"], source["installed_version"], source["install_path"]), + ([did], "stable", "0.7.1", str(sandbox.binary))) + self.assertEqual(source["discovery_uri"], + "elastos://source/stable/" + hashlib.sha256(did.encode()).hexdigest()[:32]) + registration = (source["gateways"], source["head_cid"], source["connect_ticket"], source["publisher_node_id"]) + self.assertEqual(registration, (["https://test.invalid"], "", "fixture-ticket", "fixture-node") + if transport == "publisher" else ([], "head-a", "", "")) + publisher = sandbox.data / "ElastOS/SystemServices/Publisher" + self.assertEqual((publisher / "release-head.json").read_bytes(), head) + self.assertEqual((publisher / "release.json").read_bytes(), release) + after = sandbox.home_state() + changed = {key for key in set(before) | set(after) if before.get(key) != after.get(key)} + self.assertEqual(changed, { + ".local/bin/elastos", "xdg-data/elastos/components.json", "xdg-data/elastos/sources.json", + "xdg-data/elastos/ElastOS/SystemServices/Publisher/release-head.json", + "xdg-data/elastos/ElastOS/SystemServices/Publisher/release.json"}) + self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + + def test_signature_and_binding_failures_reject_before_artifacts_in_populated_installation(self): + did, head, release = installable_fixture() + unbound = json.loads(head) + unbound["payload"]["release_sha256"] = "0" * 64 + unbound = encode_envelope(sign_envelope(unbound["payload"], "elastos.release.head.v1")) + cases = [ + ("tampered release", head, release.replace(b'"version":"0.7.1"', b'"version":"0.8.1"'), did, + "Signature verification FAILED", 2), + ("head bound to other release", unbound, release, did, "Release envelope differs from the signed head", 2), + ("foreign trust anchor", head, release, PUBLISHER_DID, "Signature verification FAILED", 1), + ] + for transport in ["publisher", "cid"]: + for name, served_head, served_release, anchor, message, request_count in cases: + with self.subTest(transport=transport, case=name), \ + InstallerSandbox(served_head, served_release, anchor) as sandbox: + self.existing_installation(sandbox) + for response, data in [("binary", RUNTIME_STUB), ("components", COMPONENTS), ("bootstrap", BOOTSTRAP)]: + sandbox.respond(response, data) + before = sandbox.home_state() + result, requests = sandbox.run("--install-only", transport=transport) + self.assertIn(message, result.stderr) + self.assertEqual(requests, self.REQUESTS[transport][:request_count]) + self.assert_no_installation_effects(sandbox, before, result) + + class DataPathTests(unittest.TestCase): def test_three_platform_release_keys(self): for system, machine, expected in [("Linux", "x86_64", "x86_64-linux"), From 4009015b945abca23a705e7cd3cb83a15afb649d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 03:19:24 -0500 Subject: [PATCH 334/535] fix(installer): probe the staged executable before replacing the binary The installer moved the downloaded binary over the current one and only then ran --version, so a signed release whose binary carried the right hash but reported another version, failed to run, or was not executable left the new bytes installed beside the previous components and sources. The staged copy now runs --version first. A refused probe removes only the staged copy and leaves the current installation untouched. --- scripts/install-bootstrap-test.py | 128 ++++++++++++++++++++---------- scripts/install.sh | 16 +++- 2 files changed, 100 insertions(+), 44 deletions(-) diff --git a/scripts/install-bootstrap-test.py b/scripts/install-bootstrap-test.py index 929c103f2..29fad3707 100644 --- a/scripts/install-bootstrap-test.py +++ b/scripts/install-bootstrap-test.py @@ -686,47 +686,95 @@ def test_corrupt_artifacts_fail_before_changes_then_clean_rerun_installs(self): self.assert_no_installation_effects(sandbox, before, result) sandbox.respond("binary", RUNTIME_STUB) sandbox.respond("components", COMPONENTS) - result, requests = sandbox.run("--install-only", transport=transport) with self.subTest(transport=transport, case="clean rerun"): - self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - self.assertEqual(requests, expected) - self.assertIn("Runtime installed:", result.stdout) - self.assertNotIn("Setting up Home", result.stdout) - advertised = json.loads(release)["payload"]["platforms"]["x86_64-linux"] - installed = sandbox.binary.read_bytes() - self.assertEqual(installed, RUNTIME_STUB) - self.assertEqual(hashlib.sha256(installed).hexdigest(), advertised["binary"]["sha256"]) - self.assertTrue(sandbox.binary.stat().st_mode & 0o100) - self.assertTrue(os.access(sandbox.binary, os.X_OK)) - self.assertFalse((sandbox.binary.parent / ".elastos.install.tmp").exists()) - manifest = (sandbox.data / "components.json").read_bytes() - self.assertEqual(manifest, COMPONENTS) - self.assertEqual(hashlib.sha256(manifest).hexdigest(), advertised["components"]["sha256"]) - calls = sandbox.runtime_calls() - self.assertEqual(calls[0], "--version") - self.assertRegex(calls[1], "^principal-root-upgrade --data-dir %s --backup-dir %s/backups/principal-root-upgrade-[0-9]+-[0-9]+$" - % (re.escape(str(sandbox.data)), re.escape(str(sandbox.data)))) - self.assertEqual(len(calls), 2, "setup and Home launch stay out of --install-only") - sources = json.loads((sandbox.data / "sources.json").read_text()) - self.assertEqual(sources["schema"], "elastos.trusted-sources/v1") - source = sources["sources"][0] - self.assertEqual((source["publisher_dids"], source["channel"], source["installed_version"], source["install_path"]), - ([did], "stable", "0.7.1", str(sandbox.binary))) - self.assertEqual(source["discovery_uri"], - "elastos://source/stable/" + hashlib.sha256(did.encode()).hexdigest()[:32]) - registration = (source["gateways"], source["head_cid"], source["connect_ticket"], source["publisher_node_id"]) - self.assertEqual(registration, (["https://test.invalid"], "", "fixture-ticket", "fixture-node") - if transport == "publisher" else ([], "head-a", "", "")) - publisher = sandbox.data / "ElastOS/SystemServices/Publisher" - self.assertEqual((publisher / "release-head.json").read_bytes(), head) - self.assertEqual((publisher / "release.json").read_bytes(), release) - after = sandbox.home_state() - changed = {key for key in set(before) | set(after) if before.get(key) != after.get(key)} - self.assertEqual(changed, { - ".local/bin/elastos", "xdg-data/elastos/components.json", "xdg-data/elastos/sources.json", - "xdg-data/elastos/ElastOS/SystemServices/Publisher/release-head.json", - "xdg-data/elastos/ElastOS/SystemServices/Publisher/release.json"}) - self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + self.assert_clean_install(sandbox, transport, did, head, release, before) + + def assert_clean_install(self, sandbox, transport, did, head, release, before): + """A valid --install-only run in a sandbox that already holds an installation.""" + sandbox.calls.unlink(missing_ok=True) + result, requests = sandbox.run("--install-only", transport=transport) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(requests, self.REQUESTS[transport]) + self.assertIn("Runtime installed:", result.stdout) + self.assertNotIn("Setting up Home", result.stdout) + advertised = json.loads(release)["payload"]["platforms"]["x86_64-linux"] + installed = sandbox.binary.read_bytes() + self.assertEqual(installed, RUNTIME_STUB) + self.assertEqual(hashlib.sha256(installed).hexdigest(), advertised["binary"]["sha256"]) + self.assertTrue(sandbox.binary.stat().st_mode & 0o100) + self.assertTrue(os.access(sandbox.binary, os.X_OK)) + self.assertFalse((sandbox.binary.parent / ".elastos.install.tmp").exists()) + manifest = (sandbox.data / "components.json").read_bytes() + self.assertEqual(manifest, COMPONENTS) + self.assertEqual(hashlib.sha256(manifest).hexdigest(), advertised["components"]["sha256"]) + calls = sandbox.runtime_calls() + self.assertEqual(calls[0], "--version") + self.assertRegex(calls[1], "^principal-root-upgrade --data-dir %s --backup-dir %s/backups/principal-root-upgrade-[0-9]+-[0-9]+$" + % (re.escape(str(sandbox.data)), re.escape(str(sandbox.data)))) + self.assertEqual(len(calls), 2, "setup and Home launch stay out of --install-only") + sources = json.loads((sandbox.data / "sources.json").read_text()) + self.assertEqual(sources["schema"], "elastos.trusted-sources/v1") + source = sources["sources"][0] + self.assertEqual((source["publisher_dids"], source["channel"], source["installed_version"], source["install_path"]), + ([did], "stable", "0.7.1", str(sandbox.binary))) + self.assertEqual(source["discovery_uri"], + "elastos://source/stable/" + hashlib.sha256(did.encode()).hexdigest()[:32]) + registration = (source["gateways"], source["head_cid"], source["connect_ticket"], source["publisher_node_id"]) + self.assertEqual(registration, (["https://test.invalid"], "", "fixture-ticket", "fixture-node") + if transport == "publisher" else ([], "head-a", "", "")) + publisher = sandbox.data / "ElastOS/SystemServices/Publisher" + self.assertEqual((publisher / "release-head.json").read_bytes(), head) + self.assertEqual((publisher / "release.json").read_bytes(), release) + after = sandbox.home_state() + changed = {key for key in set(before) | set(after) if before.get(key) != after.get(key)} + self.assertEqual(changed, { + ".local/bin/elastos", "xdg-data/elastos/components.json", "xdg-data/elastos/sources.json", + "xdg-data/elastos/ElastOS/SystemServices/Publisher/release-head.json", + "xdg-data/elastos/ElastOS/SystemServices/Publisher/release.json"}) + self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + + def test_staged_executable_refusal_preserves_installation_then_valid_retry_installs(self): + # These binaries carry the signed, advertised hash; only the staged + # executable's own behavior can reject them, and that must happen + # before the current binary is replaced. + cases = [ + ("wrong version", RUNTIME_STUB.replace(b"elastos 0.7.1", b"elastos 0.6.0"), + "version mismatch", ["--version"]), + ("nonzero exit with expected version in output", + RUNTIME_STUB.replace(b'echo "elastos 0.7.1" ;;', b'echo "elastos 0.7.1"; exit 3 ;;'), + "failed its version check (exit 3)", ["--version"]), + ("invalid executable", b"\x00\x01\x02 not an executable\n", "failed its version check", []), + ] + did, head, release = installable_fixture() + for transport in ["publisher", "cid"]: + with InstallerSandbox(head, release, did) as sandbox: + self.existing_installation(sandbox) + sandbox.respond("components", COMPONENTS) + sandbox.respond("bootstrap", BOOTSTRAP) + before = sandbox.home_state() + for name, binary, message, expected_calls in cases: + _, served_head, served_release = installable_fixture(runtime=binary) + sandbox.respond("release-head.json", served_head) + sandbox.respond("release.json", served_release) + sandbox.respond("binary", binary) + sandbox.calls.unlink(missing_ok=True) + result, requests = sandbox.run("--install-only", transport=transport) + with self.subTest(transport=transport, case=name): + self.assertNotEqual(result.returncode, 0) + self.assertIn(message, result.stderr) + self.assertIn("the current installation was preserved", result.stderr) + self.assertEqual(requests, self.REQUESTS[transport][:4]) + self.assertIn("Installing binary to", result.stdout) + self.assertNotIn("Installing components.json", result.stdout) + self.assertEqual(sandbox.runtime_calls(), expected_calls) + self.assertFalse((sandbox.binary.parent / ".elastos.install.tmp").exists()) + self.assertEqual(sandbox.home_state(), before) + self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + sandbox.respond("release-head.json", head) + sandbox.respond("release.json", release) + sandbox.respond("binary", RUNTIME_STUB) + with self.subTest(transport=transport, case="valid retry"): + self.assert_clean_install(sandbox, transport, did, head, release, before) def test_signature_and_binding_failures_reject_before_artifacts_in_populated_installation(self): did, head, release = installable_fixture() diff --git a/scripts/install.sh b/scripts/install.sh index b66a9b75b..625223b06 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -983,12 +983,20 @@ mkdir -p "$INSTALL_DIR" TMP_INSTALL_BIN="${INSTALL_DIR}/.elastos.install.tmp" cp "${TMPDIR}/elastos" "${TMP_INSTALL_BIN}" chmod +x "${TMP_INSTALL_BIN}" -mv -f "${TMP_INSTALL_BIN}" "${INSTALL_DIR}/elastos" -INSTALLED_VERSION_OUTPUT="$("${INSTALL_DIR}/elastos" --version 2>&1 || true)" -if ! printf '%s' "${INSTALLED_VERSION_OUTPUT}" | grep -Fq "${RELEASE_VERSION}"; then - die "Installed binary version mismatch at ${INSTALL_DIR}/elastos\n Expected: ${RELEASE_VERSION}\n Got: ${INSTALLED_VERSION_OUTPUT:-}" +# The staged executable must run and report the release version before it +# replaces the current binary. A refusal removes only this staged copy. +STAGED_VERSION_STATUS=0 +STAGED_VERSION_OUTPUT="$("${TMP_INSTALL_BIN}" --version 2>&1)" || STAGED_VERSION_STATUS=$? +if [[ "${STAGED_VERSION_STATUS}" -ne 0 ]]; then + rm -f "${TMP_INSTALL_BIN}" + die "Downloaded binary failed its version check (exit ${STAGED_VERSION_STATUS}); the current installation was preserved\n Output: ${STAGED_VERSION_OUTPUT:-}" +fi +if ! printf '%s' "${STAGED_VERSION_OUTPUT}" | grep -Fq "${RELEASE_VERSION}"; then + rm -f "${TMP_INSTALL_BIN}" + die "Downloaded binary version mismatch; the current installation was preserved\n Expected: ${RELEASE_VERSION}\n Got: ${STAGED_VERSION_OUTPUT:-}" fi +mv -f "${TMP_INSTALL_BIN}" "${INSTALL_DIR}/elastos" # New Runtime data is private; preserve the mode of an existing installation. (umask 077; mkdir -p "$DATA_DIR") From 49f0ac88610cd273eaf6c5d22f850d788464f6e7 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 20:17:41 -0500 Subject: [PATCH 335/535] feat(model): install a two-entry catalogue and copy it into managed Home Sash preview needs SmolLM2 and Qwen as one signed catalogue, with child Home startup installing that snapshot instead of a manual copy. --- capsules/_shared/model-management.css | 2 + capsules/_shared/model-management.js | 128 +- capsules/_shared/model-selection.js | 7 +- capsules/assistant/browser/assistant.js | 127 +- capsules/assistant/browser/harness-host.js | 2 +- capsules/assistant/browser/model-contract.js | 54 +- capsules/assistant/browser/model-selection.js | 7 +- capsules/home-gui/browser/shell-windows.js | 19 + capsules/home/browser/home-shell-host.js | 41 +- capsules/ipfs-provider/src/main.rs | 16 + capsules/marketplace/browser/marketplace.js | 82 + .../marketplace/browser/model-management.css | 21 +- .../marketplace/browser/model-management.js | 73 +- capsules/system/browser/model-management.css | 21 +- capsules/system/browser/model-management.js | 73 +- components.json | 23 +- docs/CAPSULE_MODEL.md | 21 + docs/CONTENT_CAPSULE_DISTRIBUTION.md | 56 +- docs/MODEL_PACKAGE_HANDOFF.md | 242 ++ docs/STORAGE_AND_ACCESS.md | 51 + elastos/crates/elastos-common/src/manifest.rs | 8 +- .../elastos-runtime/src/provider/registry.rs | 105 +- .../src/api/capsule_inventory.rs | 218 +- .../src/api/capsule_inventory/preparation.rs | 944 +++++++- ...proof.rs => pinned_model_process_proof.rs} | 311 ++- .../src/api/gateway_capsule_catalog.rs | 52 +- .../src/api/gateway_model_remote.rs | 79 +- .../src/api/gateway_model_service.rs | 370 ++- .../src/api/gateway_tests/remote_model.rs | 167 +- elastos/crates/elastos-server/src/carrier.rs | 2110 +++++++++++++++-- elastos/crates/elastos-server/src/content.rs | 217 +- .../elastos-server/src/provider_resource.rs | 3 + elastos/crates/elastos-server/src/publish.rs | 2 + .../elastos-server/src/runtime_control.rs | 92 +- elastos/crates/elastos-server/src/setup.rs | 293 ++- model-catalog.json | 1 + scripts/assistant-shell-smoke.mjs | 28 +- .../assistant-studio-preservation.test.mjs | 3 +- scripts/home-agent-cancellation.test.mjs | 2 +- scripts/home-agent-shell-smoke.mjs | 68 +- .../marketplace-product-behavior-smoke.mjs | 3 + scripts/model-package-handoff-test.py | 496 ++++ scripts/model-package-handoff.py | 555 +++++ scripts/model-preparation-failure-smoke.mjs | 24 +- scripts/model-selection-browser-smoke.mjs | 2 +- scripts/model-selection.test.mjs | 12 + scripts/pinned-model-consumer-proof.sh | 436 ++++ scripts/setup-source-home.sh | 47 + 48 files changed, 6990 insertions(+), 724 deletions(-) create mode 100644 docs/MODEL_PACKAGE_HANDOFF.md rename elastos/crates/elastos-server/src/api/capsule_inventory/preparation/{qwen_process_proof.rs => pinned_model_process_proof.rs} (72%) create mode 100644 model-catalog.json create mode 100755 scripts/model-package-handoff-test.py create mode 100755 scripts/model-package-handoff.py create mode 100755 scripts/pinned-model-consumer-proof.sh diff --git a/capsules/_shared/model-management.css b/capsules/_shared/model-management.css index 62a16ef7e..709773a38 100644 --- a/capsules/_shared/model-management.css +++ b/capsules/_shared/model-management.css @@ -7,3 +7,5 @@ .model-keep { display: inline-flex; align-items: center; gap: 8px; } .model-keep input { accent-color: var(--accent); } .model-row progress { width: 100%; max-width: 440px; accent-color: var(--accent); } +.model-list { list-style: none; margin: 12px 0; padding: 0; display: grid; gap: 12px; } +.model-list li { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; } diff --git a/capsules/_shared/model-management.js b/capsules/_shared/model-management.js index ce6378009..a770cc212 100644 --- a/capsules/_shared/model-management.js +++ b/capsules/_shared/model-management.js @@ -28,6 +28,38 @@ } else check(!r.admitted && !r.kept && !r.dispatch_ready); return r; } + function catalogEntries(catalog) { + check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); + check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); + if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); + const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); + check(entries.length <= 8); + if (!entries.length) return []; + check(catalog.model_catalog_state === "verified"); + const seen = new Set(); + for (const item of entries) { + check(typeof item.cid === "string" && !seen.has(item.cid)); + seen.add(item.cid); + check(item.role === "content" && item.installed === false && item.launchable === false + && CID.test(item.cid) && text(item.title) && text(item.publisher_did, 256) + && item.publisher_did.startsWith("did:") && item.signature_state === "catalog-signature-verified" + && Number.isSafeInteger(item.content_size_bytes) && item.content_size_bytes > 0); + parseRuntime(item.model_runtime, item.cid); + } + return entries; + } + function selectCatalogEntry(entries, selectedCid) { + if (selectedCid) { + const matches = entries.filter(item => item.cid === selectedCid); + return matches.length === 1 ? matches[0] : null; + } + return entries.length === 1 ? entries[0] : null; + } + function formatBytes(bytes) { + const units = ["B", "KB", "MB", "GB", "TB"]; + const rank = bytes ? Math.min(4, Math.floor(Math.log(bytes) / Math.log(1000))) : 0; + return `${(bytes / 1000 ** rank).toLocaleString(undefined, { maximumFractionDigits: rank ? 2 : 0 })} ${units[rank]}`; + } function element(tag, value, className) { const node = document.createElement(tag); if (value) node.textContent = value; @@ -43,9 +75,9 @@ dispatch_ready: output.dispatch_ready, offer_id: output.offer_id, preparation }, cid); } window.ElastosModelManagement = { - create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary" }) { + create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: requiredCid = null, compact = false, onReadyOpen = null }) { let visible = false, closed = false, generation = 0, busy = false, timer, reader; - let model = null, methods = new Map(), message = "", loading = false, polls = 0; + let selectedCid = requiredCid, choices = [], model = null, methods = new Map(), message = "", loading = false, polls = 0; let unresolvedUse = null, reconcileRequired = false; let pendingFocus = null; root.classList.add("model-management"); @@ -77,19 +109,10 @@ return { signal: value.signal, abort: () => value.abort(), done: () => clearTimeout(timeout) }; } function parseCatalog(catalog) { - check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); - check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); - if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); - const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); - check(entries.length <= 1 && (!entries.length || catalog.model_catalog_state === "verified")); + const entries = catalogEntries(catalog); + choices = entries; if (!entries.length) return null; - const entry = entries[0]; - check(entry.role === "content" && entry.installed === false && entry.launchable === false - && CID.test(entry.cid) && text(entry.title) && text(entry.publisher_did, 256) - && entry.publisher_did.startsWith("did:") && entry.signature_state === "catalog-signature-verified" - && Number.isSafeInteger(entry.content_size_bytes) && entry.content_size_bytes > 0); - parseRuntime(entry.model_runtime, entry.cid); - return entry; + return selectCatalogEntry(entries, requiredCid || selectedCid); } function bindMethods(registry) { check(Array.isArray(registry.interfaces) && registry.interfaces.length <= 1024); @@ -123,7 +146,7 @@ if (model?.cid !== unresolvedUse?.cid || model?.model_runtime.preparation) unresolvedUse = null; } catch { if (epoch !== generation || !show()) return; - model = null; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; + model = null; choices = []; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; } finally { io.done(); if (epoch === generation) { loading = false; render(); schedule(); } @@ -166,11 +189,14 @@ if (epoch !== generation || !show() || model?.cid !== cid) return; check(result.cid === cid); if (operation === "retention") { - check(result.kept === keep && result.admitted === true); - // Keep is confirmed only by the current caller's Runtime response. - const candidate = operationRuntime(result, cid, model.model_runtime.preparation); - check(candidate.kept === keep); - model.model_runtime = candidate; + check(result.kept === keep && typeof result.admitted === "boolean"); + // Admission can finish while Keep is in flight. Read a fresh catalog + // instead of combining new readiness with an older preparation. + const catalog = await request("/api/capsules/catalog", null, io.signal); + if (epoch !== generation || !show() || model?.cid !== cid) return; + const candidate = parseCatalog(catalog); + check(candidate?.cid === cid); + model = candidate; } else { const r = operationRuntime(result, cid); check(r.preparation?.operation_id === result.operation_id && (operation !== "cancel" || result.operation_id === id)); @@ -207,32 +233,66 @@ const status = element("p", loading ? "Loading models…" : message); status.setAttribute("role", "status"); root.append(status); if (!model) { + if (!loading && !message && !requiredCid && choices.length > 1) { + root.append(element("p", "Select a verified model. Keep and removal apply to the model you select.")); + const list = element("ul", "", "model-list"); + for (const item of choices) { + const row = element("li"); + const pick = button(item.title, () => { selectedCid = item.cid; void load(); }); + pick.dataset.modelControl = `select-${item.cid}`; + row.append(pick, element("span", formatBytes(item.content_size_bytes), "model-size")); + list.append(row); + } + root.append(list); + restoreFocus(); + return; + } if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); restoreFocus(); return; } const r = model.model_runtime, p = r.preparation; const row = element("article", "", "model-row"); - row.append(element("h2", model.title), element("p", `Verified publisher: ${model.publisher_did}`), - element("p", `Content ID: ${model.cid}`, "model-identity"), element("p", `${model.content_size_bytes.toLocaleString()} bytes`)); - const label = message ? "Current status unavailable" : r.dispatch_ready ? "Ready to use" : r.admitted ? "Prepared. The model offer is unavailable." - : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : "Preparing…") - : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Not prepared"; + if (!compact) { + row.append(element("h2", model.title), element("p", formatBytes(model.content_size_bytes), "model-size")); + const identity = element("details", "", "model-details"); + identity.append(element("summary", "Content details"), element("p", `Verified publisher: ${model.publisher_did}`), + element("p", `Content ID: ${model.cid}`, "model-identity")); + row.append(identity); + } + const label = message ? "Current status unavailable" : r.dispatch_ready ? "Available on this device" : r.admitted ? "Available on this device. Model service unavailable." + : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : p.state === "verifying" ? "Checking model files…" : "Preparing local files…") + : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Ready to prepare"; row.append(element("p", label)); if (active(p)) { const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; - progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${p.completed_bytes.toLocaleString()} of ${p.total_bytes.toLocaleString()} bytes`)); + progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${formatBytes(p.completed_bytes)} of ${formatBytes(p.total_bytes)} prepared`)); } const controls = element("div", "", "model-controls"); const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); use.dataset.modelControl = "use"; if (!active(p) && !r.dispatch_ready) controls.append(use); + if (r.dispatch_ready && typeof onReadyOpen === "function") { + const open = button("Open in Assistant", () => onReadyOpen({ cid: model.cid, offer_id: r.offer_id })); + open.dataset.modelControl = "open-assistant"; + controls.append(open); + } if (active(p)) { const cancel = button("Cancel preparation", () => void act("cancel"), p.cancel_requested || reconcileRequired); cancel.dataset.modelControl = "cancel"; controls.append(cancel); } const labelNode = element("label", "", "model-keep"), toggle = element("input"); - toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !r.admitted || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; + toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !(r.admitted || active(p)) || p?.cancel_requested || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; toggle.addEventListener("change", () => void act("retention", toggle.checked)); labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); - row.append(controls); root.append(row); + row.append(controls); + row.append(element("p", r.kept + ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") + : "Prepared files can be removed during cache cleanup.", "model-hint")); + row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + root.append(row); + if (!requiredCid && choices.length > 1) { + const back = button("Choose another model", () => { selectedCid = null; model = null; void load(); }); + back.dataset.modelControl = "choose-another"; + root.append(back); + } restoreFocus(); } function setVisible(value) { @@ -240,9 +300,15 @@ visible = value; root.hidden = !value; ++generation; stopRead(); if (show()) void load(); } - document.addEventListener("visibilitychange", () => { ++generation; stopRead(); if (show()) void load(); }); - window.addEventListener("pagehide", () => { closed = true; ++generation; stopRead(); }); - return { setVisible, refresh: load }; + const onVisibility = () => { ++generation; stopRead(); if (show()) void load(); }; + function destroy() { + closed = true; ++generation; stopRead(); + document.removeEventListener("visibilitychange", onVisibility); + window.removeEventListener("pagehide", destroy); + } + document.addEventListener("visibilitychange", onVisibility); + window.addEventListener("pagehide", destroy); + return { setVisible, refresh: load, destroy }; }, }; })(); diff --git a/capsules/_shared/model-selection.js b/capsules/_shared/model-selection.js index 607f3ed74..945f4fb08 100644 --- a/capsules/_shared/model-selection.js +++ b/capsules/_shared/model-selection.js @@ -6,7 +6,12 @@ export function catalogModels(catalog) { if (catalog?.schema !== "elastos.capsules.catalog/v1" || !Array.isArray(catalog.capsules) || catalog.capsules.length > 1024 || !["verified", "unavailable", "unconfigured"].includes(catalog.model_catalog_state)) throw new Error("Model catalog unavailable"); const rows = catalog.capsules.filter(row => row?.source === "signed-model-catalog"); - if (rows.length > 1) throw new Error("Model catalog unavailable"); + if (rows.length > 8) throw new Error("Model catalog unavailable"); + const seen = new Set(); + for (const row of rows) { + if (typeof row.cid !== "string" || seen.has(row.cid)) throw new Error("Model catalog unavailable"); + seen.add(row.cid); + } if (catalog.model_catalog_state !== "verified") return []; return rows.map(row => { const facts = row.model_runtime; diff --git a/capsules/assistant/browser/assistant.js b/capsules/assistant/browser/assistant.js index a46c05016..42beaaaa0 100644 --- a/capsules/assistant/browser/assistant.js +++ b/capsules/assistant/browser/assistant.js @@ -4,6 +4,7 @@ import { } from "/apps/home/home-clipboard-client.js?v=home-20260726a"; import { renderToString as renderMathToString } from "./vendor/katex/katex.mjs"; import { catalogModels, selectedModelOffer, readyModelChoices } from "./model-selection.js"; +import { applyRunEventsPage, parseCursor } from "./model-contract.js"; const MODEL_TEXT_INPUT_SCHEMA = "elastos.model.input.text/v1"; const MODEL_IMAGE_INPUT_SCHEMA = "elastos.model.input.image/v1"; @@ -190,10 +191,6 @@ function parseRunEventsPage(payload) { return page && typeof page === "object" && Array.isArray(page.events) ? page : null; } -function parseCursorValue(value) { - const cursor = Number(value); - return Number.isInteger(cursor) && cursor >= 0 ? cursor : null; -} function studioInputSchema(operation) { if (operation === "image.generate") { @@ -235,25 +232,6 @@ function parseStudioOutput(output) { return null; } -function parseStudioProgress(data) { - const phase = - typeof data?.phase === "string" && data.phase.trim() === data.phase - ? data.phase - : ""; - const completed = Number(data?.completed); - const total = Number(data?.total); - if ( - !phase || - !Number.isInteger(completed) || - !Number.isInteger(total) || - completed < 0 || - total < 0 || - completed > total - ) { - return null; - } - return { phase, completed, total }; -} function createSessionRecord({ id, mode, title = "", pinned = false } = {}) { const nextMode = mode === MODE_BUILD ? MODE_BUILD : MODE_CHAT; @@ -1041,69 +1019,28 @@ export function createAssistantApp({ }); } - function applyRunEvents(events) { + function applyDecodedEvents(applied) { const run = state.activeRun; if (!run) { return false; } - let nextText = run.outputText; - let terminal = null; - for (const event of events) { - if (!event || typeof event !== "object") { - continue; + if (run.mode === MODE_STUDIO) { + if (applied.studioProgress) { + state.studioProgress = applied.studioProgress; } - if (run.mode !== MODE_STUDIO && event.kind === "text_delta") { - nextText = boundedText( - nextText + String(event.data?.text ?? ""), - MAX_MESSAGE_CONTENT_BYTES, - ); - } else if (run.mode === MODE_STUDIO && event.kind === "progress") { - const progress = parseStudioProgress(event.data); - if (!progress) { - stopPollingUnavailable(run, "Model provider unavailable."); - return false; - } - state.studioProgress = progress; - } else if (event.kind === "output") { - terminal = { - status: "completed", - output: event.data ?? null, - error: null, - }; - } else if (event.kind === "completed") { - terminal = { - status: "completed", - output: event.data?.output_retained === false ? null : event.data ?? null, - error: null, - outputRetained: event.data?.output_retained !== false, - }; - } else if (event.kind === "failed") { - terminal = { - status: "failed", - output: null, - error: event.data ?? null, - }; - } else if (event.kind === "cancelled") { - terminal = { - status: "cancelled", - output: null, - error: event.data ?? null, - }; - } else if (event.kind === "settlement_unknown") { - terminal = { - status: "settlement_unknown", - output: null, - error: event.data ?? null, - }; + } else if (applied.textDeltas.length) { + let nextText = run.outputText; + for (const delta of applied.textDeltas) { + nextText = boundedText(nextText + delta, MAX_MESSAGE_CONTENT_BYTES); + } + if (nextText !== run.outputText) { + run.outputText = nextText; + updateStreamingMessage(run.sessionId, run.runId, nextText); + markWorkspaceDirty(); } } - if (run.mode !== MODE_STUDIO && nextText !== run.outputText) { - run.outputText = nextText; - updateStreamingMessage(run.sessionId, run.runId, nextText); - markWorkspaceDirty(); - } - if (terminal) { - return settleTerminal(run.runId, terminal); + if (applied.terminal) { + return settleTerminal(run.runId, applied.terminal); } notify(); return false; @@ -1175,30 +1112,16 @@ export function createAssistantApp({ stopPollingUnavailable(run, "Model provider unavailable."); return; } - const previousCursor = run.afterSequence; - const nextCursor = parseCursorValue(page.next_cursor); - if (nextCursor === null || nextCursor < previousCursor) { - stopPollingUnavailable(run, "Model provider unavailable."); - return; - } - const events = []; - let lastSequence = previousCursor; - for (const event of page.events) { - const sequence = parseCursorValue(event?.sequence); - if (sequence === null || sequence <= lastSequence) { - stopPollingUnavailable(run, "Model provider unavailable."); - return; - } - events.push(event); - lastSequence = sequence; - } - if (nextCursor < lastSequence) { + let applied; + try { + applied = applyRunEventsPage(page, run.afterSequence); + } catch { stopPollingUnavailable(run, "Model provider unavailable."); return; } run.pollErrorCount = 0; - run.afterSequence = nextCursor; - const settled = applyRunEvents(events); + run.afterSequence = applied.nextCursor; + const settled = applyDecodedEvents(applied); if (settled || run.terminal) { clearPollTimer(); return; @@ -1305,7 +1228,7 @@ export function createAssistantApp({ sessionId: isStudio ? studioSessionId : session?.id || "", mode: isStudio ? MODE_STUDIO : session.mode, mediaLabel: offer.mediaLabel || "", - afterSequence: parseCursorValue(runView.sequence_cursor) ?? 0, + afterSequence: parseCursor(runView.sequence_cursor) ?? 0, outputText: "", terminal: false, cancelRequested: false, @@ -1417,7 +1340,9 @@ export function createAssistantApp({ if (!["http:", "https:"].includes(origin.protocol) || origin.origin !== homeOrigin) return false; } catch { return false; } if (!homeToken || !targetWindow || targetWindow === sourceWindow) return false; - targetWindow.postMessage({ type: "home:open-target", homeToken, target: "system", query: { settings: "models" } }, homeOrigin); + const query = { category: "models" }; + if (state.selectedModelCid) query.model_cid = state.selectedModelCid; + targetWindow.postMessage({ type: "home:open-target", homeToken, target: "marketplace", query }, homeOrigin); return true; }, snapshot, diff --git a/capsules/assistant/browser/harness-host.js b/capsules/assistant/browser/harness-host.js index a4901e5e9..ae6136fa1 100644 --- a/capsules/assistant/browser/harness-host.js +++ b/capsules/assistant/browser/harness-host.js @@ -251,7 +251,7 @@ export function openModelsFromAgent() { if (!homeToken || window.top === window) return false; // The top Home registers this exact nested app source before accepting its intent. window.top.postMessage({ type: "home:app-ready", homeToken }, origin); - window.top.postMessage({ type: "home:open-target", target: "system", query: { settings: "models" }, homeToken }, origin); + window.top.postMessage({ type: "home:open-target", target: "marketplace", query: { category: "models" }, homeToken }, origin); return true; } diff --git a/capsules/assistant/browser/model-contract.js b/capsules/assistant/browser/model-contract.js index 4bdcd0a3c..46019790a 100644 --- a/capsules/assistant/browser/model-contract.js +++ b/capsules/assistant/browser/model-contract.js @@ -95,11 +95,35 @@ export function parseCursor(value) { return Number.isInteger(cursor) && cursor >= 0 ? cursor : null; } +/** Studio progress on a `progress` event. Null when the payload is malformed. */ +export function parseStudioProgress(data) { + const phase = + typeof data?.phase === "string" && data.phase.trim() === data.phase ? data.phase : ""; + const completed = Number(data?.completed); + const total = Number(data?.total); + if ( + !phase || + !Number.isInteger(completed) || + !Number.isInteger(total) || + completed < 0 || + total < 0 || + completed > total + ) { + return null; + } + return { phase, completed, total }; +} + /** * Validate a runs_events page against the cursor we hold and reduce it to - * what the stream needs. Throws on a page the provider should never send. + * what Chat and Studio need. The provider may replay events at or below the + * saved cursor; this function skips those. A new sequence must rise from + * the last newly applied event. An unseen earlier sequence is rejected. A + * cursor must not move backwards. Throws on a page the provider should + * never send. * @returns {{ nextCursor: number, hasMore: boolean, textDeltas: string[], - * terminal: null | { status: string, output: unknown, error: unknown } }} + * terminal: null | { status: string, output: unknown, error: unknown }, + * studioProgress: null | { phase: string, completed: number, total: number } }} */ export function applyRunEventsPage(page, afterSequence) { if (!page || typeof page !== "object" || !Array.isArray(page.events)) { @@ -111,12 +135,22 @@ export function applyRunEventsPage(page, afterSequence) { } const textDeltas = []; let terminal = null; + let studioProgress = null; let lastSequence = afterSequence; for (const event of page.events) { const sequence = parseCursor(event?.sequence); - if (sequence === null || sequence <= lastSequence) { + if (sequence === null) { throw contractError("bad_sequence", "run events are not strictly increasing"); } + if (sequence <= afterSequence) { + continue; + } + if (sequence < lastSequence) { + throw contractError("bad_sequence", "run events arrived out of order"); + } + if (sequence === lastSequence) { + continue; + } lastSequence = sequence; const kind = typeof event.kind === "string" ? event.kind : ""; if (kind === "text_delta") { @@ -124,6 +158,12 @@ export function applyRunEventsPage(page, afterSequence) { if (typeof text === "string" && text) { textDeltas.push(text); } + } else if (kind === "progress") { + const progress = parseStudioProgress(event.data); + if (!progress) { + throw contractError("bad_progress", "studio progress event is malformed"); + } + studioProgress = progress; } else if (kind === "output") { terminal = { status: "completed", output: event.data ?? null, error: null }; } else if (kind === "completed") { @@ -148,7 +188,13 @@ export function applyRunEventsPage(page, afterSequence) { if (nextCursor < lastSequence) { throw contractError("bad_cursor", "run events cursor behind last sequence"); } - return { nextCursor, hasMore: page.has_more === true, textDeltas, terminal }; + return { + nextCursor, + hasMore: page.has_more === true, + textDeltas, + terminal, + studioProgress, + }; } /** Final text the provider settles with, when the output is typed text. */ diff --git a/capsules/assistant/browser/model-selection.js b/capsules/assistant/browser/model-selection.js index 607f3ed74..945f4fb08 100644 --- a/capsules/assistant/browser/model-selection.js +++ b/capsules/assistant/browser/model-selection.js @@ -6,7 +6,12 @@ export function catalogModels(catalog) { if (catalog?.schema !== "elastos.capsules.catalog/v1" || !Array.isArray(catalog.capsules) || catalog.capsules.length > 1024 || !["verified", "unavailable", "unconfigured"].includes(catalog.model_catalog_state)) throw new Error("Model catalog unavailable"); const rows = catalog.capsules.filter(row => row?.source === "signed-model-catalog"); - if (rows.length > 1) throw new Error("Model catalog unavailable"); + if (rows.length > 8) throw new Error("Model catalog unavailable"); + const seen = new Set(); + for (const row of rows) { + if (typeof row.cid !== "string" || seen.has(row.cid)) throw new Error("Model catalog unavailable"); + seen.add(row.cid); + } if (catalog.model_catalog_state !== "verified") return []; return rows.map(row => { const facts = row.model_runtime; diff --git a/capsules/home-gui/browser/shell-windows.js b/capsules/home-gui/browser/shell-windows.js index 468a53166..91c3c158b 100644 --- a/capsules/home-gui/browser/shell-windows.js +++ b/capsules/home-gui/browser/shell-windows.js @@ -1306,6 +1306,19 @@ function requestDocumentsWindowClose(entry) { return promise; } +function postMarketplaceNavigate(entry, query) { + const frame = entry.node.querySelector(".window-frame"); + const homeToken = browserLaunchAuthority(frame?.dataset.route)?.homeToken; + if (!homeToken || !frame?.contentWindow) { + return; + } + frame.contentWindow.postMessage({ + type: "elastos.marketplace.navigate/v1", + homeToken, + query, + }, "*"); +} + function browserLaunchAuthority(route) { try { const url = new URL(route, window.location.href); @@ -1726,6 +1739,9 @@ async function launchBrowserTargetWindow(targetId, options = {}) { const ok = await requestSystemWindow(entry, "navigate", options.query); if (!ok) throw new Error("System could not open the requested settings"); } + if (targetId === "marketplace" && Object.keys(options.query || {}).length > 0) { + postMarketplaceNavigate(entry, options.query); + } if ((targetId === "inbox" || targetId === "wallet") && Object.keys(options.query || {}).length > 0) { const ok = await requestSelectionWindow(entry, options.query); if (!ok) throw Object.assign(new Error("The app could not select this request."), { selectionOnly: true }); @@ -2347,6 +2363,9 @@ export async function restoreShellSession() { const restoredEntries = []; const restoredSingleSessionTargets = new Set(); for (const restoredWindow of restoredWindows) { + if (restoredWindow.hidden) { + continue; + } if (isSingleWindowTarget(restoredWindow.target)) { if (restoredSingleSessionTargets.has(restoredWindow.target)) { continue; diff --git a/capsules/home/browser/home-shell-host.js b/capsules/home/browser/home-shell-host.js index 5df89e3e9..5be633cc6 100644 --- a/capsules/home/browser/home-shell-host.js +++ b/capsules/home/browser/home-shell-host.js @@ -70,8 +70,8 @@ const WALLET_CONNECTOR_TARGETS = new Set( Object.keys(WALLET_CONNECTOR_TARGET_TITLES), ); const SHELL_MESSAGE_OPEN_TARGET_SOURCES = Object.freeze({ - assistant: new Set(["system"]), - "home-agent": new Set(["system"]), + assistant: new Set(["marketplace"]), + "home-agent": new Set(["marketplace"]), "archive-manager": new Set(["library"]), browser: new Set(["library"]), "chat-room": new Set(["library"]), @@ -1822,7 +1822,11 @@ window.addEventListener("message", (event) => { } if (["assistant", "home-agent"].includes(context.targetId) && (!hasExactMessageKeys(data, ["type", "target", "query", "homeToken"]) || - !hasExactMessageKeys(data.query, ["settings"]) || data.query.settings !== "models")) return; + data.target !== "marketplace" || + !assistantModelsMarketplaceQuery(data.query))) return; + if (context.targetId === "marketplace" && target === "assistant" && + (!hasExactMessageKeys(data, ["type", "target", "query", "homeToken"]) || + !marketplaceAssistantHandoffQuery(data.query))) return; if ( context.kind === "app-frame" && context.targetId === "wallet" && @@ -1959,6 +1963,37 @@ function browserInstanceFromRoute(route) { } } +const MODEL_CONTENT_CID = /^bafybei[a-z2-7]{51}[aeimquy4]$/; +const MODEL_OFFER_ID = /^model:[0-9a-f]{64}$/; + +function assistantModelsMarketplaceQuery(query) { + if (!query || typeof query !== "object" || Array.isArray(query) || query.category !== "models") { + return false; + } + const keys = Object.keys(query); + if (keys.length === 1) { + return keys[0] === "category"; + } + return keys.length === 2 + && Object.prototype.hasOwnProperty.call(query, "category") + && Object.prototype.hasOwnProperty.call(query, "model_cid") + && MODEL_CONTENT_CID.test(query.model_cid); +} + +function marketplaceAssistantHandoffQuery(query) { + if (!query || typeof query !== "object" || Array.isArray(query) + || !MODEL_CONTENT_CID.test(query.model_cid || "")) { + return false; + } + const keys = Object.keys(query); + if (keys.length === 1) { + return keys[0] === "model_cid"; + } + return keys.length === 2 + && Object.prototype.hasOwnProperty.call(query, "offer_id") + && MODEL_OFFER_ID.test(query.offer_id); +} + function hasExactMessageKeys(message, expectedKeys) { const actual = Object.keys(message).sort(); const expected = [...expectedKeys].sort(); diff --git a/capsules/ipfs-provider/src/main.rs b/capsules/ipfs-provider/src/main.rs index 3d7e9f0a2..52271dd50 100644 --- a/capsules/ipfs-provider/src/main.rs +++ b/capsules/ipfs-provider/src/main.rs @@ -916,9 +916,17 @@ impl IpfsProvider { .redirects(0) .try_proxy_from_env(false) .build(); + // A bounded read is the local-repo probe before Carrier + // availability. Searching DHT here would spend the whole + // deadline on a miss and then still fail over. Kubo path cat + // of a missing DAG also keeps the HTTP call open with + // offline=true; a 1ms API timeout fails that miss immediately. + // A local hit still returns the bounded range. let mut request = agent .post(&format!("{}/api/v0/cat", self.api_url())) .query("arg", &arg) + .query("offline", "true") + .query("timeout", "1ms") .set("Accept-Encoding", "identity") .timeout(BOUNDED_READ_TIMEOUT); if let Some(range) = &range { @@ -2027,6 +2035,8 @@ mod tests { assert_eq!(provider.state, KuboState::Ready); assert!(version_headers.starts_with("POST /api/v0/version")); assert!(headers.starts_with("POST /api/v0/cat?")); + assert!(headers.contains("offline=true")); + assert!(headers.contains("timeout=1ms")); assert!(headers.contains("offset=8") && headers.contains("length=4")); let Response::Ok { data: Some(data) } = result else { panic!("bounded read failed"); @@ -2373,6 +2383,8 @@ mod tests { ); for (headers, offset) in [(first, "offset=8"), (second, "offset=12")] { assert!(headers.starts_with("POST /api/v0/cat?")); + assert!(headers.contains("offline=true")); + assert!(headers.contains("timeout=1ms")); assert!(headers.contains(offset) && headers.contains("length=4")); } let Response::Ok { data: Some(data) } = next else { @@ -2422,6 +2434,8 @@ mod tests { let requests = backend.join().unwrap(); assert_eq!(requests.len(), 1, "bounded failure must not follow, retry or prefetch"); assert!(requests[0].starts_with("POST /api/v0/cat?")); + assert!(requests[0].contains("offline=true")); + assert!(requests[0].contains("timeout=1ms")); assert!(matches!(result, Response::Error { .. })); } } @@ -2554,6 +2568,8 @@ mod tests { "bounded Cat must not pin, prefetch or retry" ); assert!(requests[0].0.starts_with("POST /api/v0/cat?")); + assert!(requests[0].0.contains("offline=true")); + assert!(requests[0].0.contains("timeout=1ms")); assert!(requests[0].1, "bounded Cat must send offset=8 and length=4"); if ignore_range { assert!( diff --git a/capsules/marketplace/browser/marketplace.js b/capsules/marketplace/browser/marketplace.js index 3d8eb942f..7f9b383ad 100644 --- a/capsules/marketplace/browser/marketplace.js +++ b/capsules/marketplace/browser/marketplace.js @@ -29,6 +29,9 @@ const pendingMediaBuys = new Set(); let detailPreviousFocus = null; let detailModels = null; + let pendingModelLaunch = null; + const MODEL_CONTENT_CID = /^bafybei[a-z2-7]{51}[aeimquy4]$/; + const MODEL_OFFER_ID = /^model:[0-9a-f]{64}$/; let homeChromeReady = false; let lastHomeMenuManifestSignature = ""; @@ -76,6 +79,7 @@ announceHomeChrome(); await loadData(); render(); + applyModelsLaunch(modelsQueryFromLocation()); } function announceHomeChrome() { @@ -180,12 +184,88 @@ return; } const data = event.data; + if (data?.type === "elastos.marketplace.navigate/v1") { + const keys = data && typeof data === "object" ? Object.keys(data).sort() : []; + if (data.homeToken !== homeToken || keys.length !== 3 + || keys[0] !== "homeToken" || keys[1] !== "query" || keys[2] !== "type") { + return; + } + applyModelsLaunch(data.query); + return; + } if (data?.type !== "elastos:menu-command" || typeof data.cmd !== "string") { return; } handleHomeMenuCommand(data.cmd); } + function modelsQueryFromLocation() { + const query = {}; + const category = params.get("category"); + const modelCid = params.get("model_cid"); + if (category) query.category = category; + if (modelCid) query.model_cid = modelCid; + return query; + } + + function modelsLaunchQuery(query) { + if (!query || typeof query !== "object" || Array.isArray(query) || query.category !== "models") { + return null; + } + const keys = Object.keys(query); + if (keys.length === 1 && keys[0] === "category") { + return { category: "models" }; + } + if (keys.length === 2 && Object.prototype.hasOwnProperty.call(query, "model_cid") + && MODEL_CONTENT_CID.test(query.model_cid)) { + return { category: "models", model_cid: query.model_cid }; + } + return null; + } + + function applyModelsLaunch(query) { + const launch = modelsLaunchQuery(query); + if (!launch) { + return; + } + pendingModelLaunch = launch.model_cid ? launch : null; + selectDestination("models"); + openPendingModelDetail(); + } + + function openPendingModelDetail() { + const launch = pendingModelLaunch; + if (!launch?.model_cid) { + return; + } + const app = state.apps.find((candidate) => candidate.modelCid === launch.model_cid); + if (!app) { + return; + } + pendingModelLaunch = null; + showAppDetail(app.id); + } + + function openReadyModelInAssistant({ cid, offer_id }) { + if (!MODEL_CONTENT_CID.test(cid || "")) { + return; + } + if (window.top === window || !homeParentOrigin) { + showToast("Open Apps from Home to use this model.", true); + return; + } + const query = { model_cid: cid }; + if (MODEL_OFFER_ID.test(offer_id || "")) { + query.offer_id = offer_id; + } + window.top.postMessage({ + type: "home:open-target", + target: "assistant", + query, + homeToken, + }, homeParentOrigin); + } + function handleHomeMenuCommand(command) { if (command === "refresh") { loadData() @@ -466,6 +546,7 @@ renderSections(); } syncHomeMenuManifest(); + openPendingModelDetail(); } function normalizeDestination(id) { @@ -909,6 +990,7 @@ detailModels = window.ElastosModelManagement.create({ root: els.detailContent.querySelector("[data-model-management]"), capsule: "marketplace", token: homeToken, buttonClass: "modal-btn secondary", cid: app.modelCid, compact: true, + onReadyOpen: openReadyModelInAssistant, }); detailModels.setVisible(true); } diff --git a/capsules/marketplace/browser/model-management.css b/capsules/marketplace/browser/model-management.css index d87a1de9f..709773a38 100644 --- a/capsules/marketplace/browser/model-management.css +++ b/capsules/marketplace/browser/model-management.css @@ -1,14 +1,11 @@ -/* Shared tokens follow the active System or Marketplace theme. */ .model-management[hidden] { display: none; } -.model-management { min-width: 0; color: var(--el-text); font-size: 13px; } -.model-management > p:empty { display: none; } -.model-row { padding: 16px; margin-top: 12px; border: 1px solid var(--el-hairline); border-radius: 8px; background: var(--el-surface-raised); min-width: 0; } -.model-row h2 { font-size: 16px; margin: 0 0 8px; } +.model-management { min-width: 0; color: var(--ink); } +.model-row { padding: 16px 0; border-top: 1px solid var(--line); min-width: 0; } +.model-row h2 { font-size: 18px; margin: 0 0 12px; } .model-row p { overflow-wrap: anywhere; line-height: 1.5; margin: 8px 0; } -.model-row .model-size, .model-row .model-hint, .model-details { color: var(--el-muted); font-size: 12px; } -.model-details summary { cursor: pointer; } -.model-details p { font-size: 11px; } -.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; margin: 12px 0; } -.model-keep { display: inline-flex; align-items: center; gap: 8px; cursor: pointer; } -.model-keep input { accent-color: var(--el-accent); width: 16px; height: 16px; margin: 0; } -.model-row progress { width: 100%; accent-color: var(--el-accent); } +.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 16px; } +.model-keep { display: inline-flex; align-items: center; gap: 8px; } +.model-keep input { accent-color: var(--accent); } +.model-row progress { width: 100%; max-width: 440px; accent-color: var(--accent); } +.model-list { list-style: none; margin: 12px 0; padding: 0; display: grid; gap: 12px; } +.model-list li { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; } diff --git a/capsules/marketplace/browser/model-management.js b/capsules/marketplace/browser/model-management.js index 2ba57fcb3..a770cc212 100644 --- a/capsules/marketplace/browser/model-management.js +++ b/capsules/marketplace/browser/model-management.js @@ -28,6 +28,33 @@ } else check(!r.admitted && !r.kept && !r.dispatch_ready); return r; } + function catalogEntries(catalog) { + check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); + check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); + if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); + const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); + check(entries.length <= 8); + if (!entries.length) return []; + check(catalog.model_catalog_state === "verified"); + const seen = new Set(); + for (const item of entries) { + check(typeof item.cid === "string" && !seen.has(item.cid)); + seen.add(item.cid); + check(item.role === "content" && item.installed === false && item.launchable === false + && CID.test(item.cid) && text(item.title) && text(item.publisher_did, 256) + && item.publisher_did.startsWith("did:") && item.signature_state === "catalog-signature-verified" + && Number.isSafeInteger(item.content_size_bytes) && item.content_size_bytes > 0); + parseRuntime(item.model_runtime, item.cid); + } + return entries; + } + function selectCatalogEntry(entries, selectedCid) { + if (selectedCid) { + const matches = entries.filter(item => item.cid === selectedCid); + return matches.length === 1 ? matches[0] : null; + } + return entries.length === 1 ? entries[0] : null; + } function formatBytes(bytes) { const units = ["B", "KB", "MB", "GB", "TB"]; const rank = bytes ? Math.min(4, Math.floor(Math.log(bytes) / Math.log(1000))) : 0; @@ -48,9 +75,9 @@ dispatch_ready: output.dispatch_ready, offer_id: output.offer_id, preparation }, cid); } window.ElastosModelManagement = { - create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: selectedCid = null, compact = false }) { + create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: requiredCid = null, compact = false, onReadyOpen = null }) { let visible = false, closed = false, generation = 0, busy = false, timer, reader; - let model = null, methods = new Map(), message = "", loading = false, polls = 0; + let selectedCid = requiredCid, choices = [], model = null, methods = new Map(), message = "", loading = false, polls = 0; let unresolvedUse = null, reconcileRequired = false; let pendingFocus = null; root.classList.add("model-management"); @@ -82,20 +109,10 @@ return { signal: value.signal, abort: () => value.abort(), done: () => clearTimeout(timeout) }; } function parseCatalog(catalog) { - check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); - check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); - if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); - const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); - check(entries.length <= 1 && (!entries.length || catalog.model_catalog_state === "verified")); + const entries = catalogEntries(catalog); + choices = entries; if (!entries.length) return null; - const entry = entries[0]; - if (selectedCid && entry.cid !== selectedCid) return null; - check(entry.role === "content" && entry.installed === false && entry.launchable === false - && CID.test(entry.cid) && text(entry.title) && text(entry.publisher_did, 256) - && entry.publisher_did.startsWith("did:") && entry.signature_state === "catalog-signature-verified" - && Number.isSafeInteger(entry.content_size_bytes) && entry.content_size_bytes > 0); - parseRuntime(entry.model_runtime, entry.cid); - return entry; + return selectCatalogEntry(entries, requiredCid || selectedCid); } function bindMethods(registry) { check(Array.isArray(registry.interfaces) && registry.interfaces.length <= 1024); @@ -129,7 +146,7 @@ if (model?.cid !== unresolvedUse?.cid || model?.model_runtime.preparation) unresolvedUse = null; } catch { if (epoch !== generation || !show()) return; - model = null; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; + model = null; choices = []; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; } finally { io.done(); if (epoch === generation) { loading = false; render(); schedule(); } @@ -216,6 +233,20 @@ const status = element("p", loading ? "Loading models…" : message); status.setAttribute("role", "status"); root.append(status); if (!model) { + if (!loading && !message && !requiredCid && choices.length > 1) { + root.append(element("p", "Select a verified model. Keep and removal apply to the model you select.")); + const list = element("ul", "", "model-list"); + for (const item of choices) { + const row = element("li"); + const pick = button(item.title, () => { selectedCid = item.cid; void load(); }); + pick.dataset.modelControl = `select-${item.cid}`; + row.append(pick, element("span", formatBytes(item.content_size_bytes), "model-size")); + list.append(row); + } + root.append(list); + restoreFocus(); + return; + } if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); restoreFocus(); return; @@ -241,6 +272,11 @@ const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); use.dataset.modelControl = "use"; if (!active(p) && !r.dispatch_ready) controls.append(use); + if (r.dispatch_ready && typeof onReadyOpen === "function") { + const open = button("Open in Assistant", () => onReadyOpen({ cid: model.cid, offer_id: r.offer_id })); + open.dataset.modelControl = "open-assistant"; + controls.append(open); + } if (active(p)) { const cancel = button("Cancel preparation", () => void act("cancel"), p.cancel_requested || reconcileRequired); cancel.dataset.modelControl = "cancel"; controls.append(cancel); } const labelNode = element("label", "", "model-keep"), toggle = element("input"); toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !(r.admitted || active(p)) || p?.cancel_requested || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; @@ -252,6 +288,11 @@ : "Prepared files can be removed during cache cleanup.", "model-hint")); row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); root.append(row); + if (!requiredCid && choices.length > 1) { + const back = button("Choose another model", () => { selectedCid = null; model = null; void load(); }); + back.dataset.modelControl = "choose-another"; + root.append(back); + } restoreFocus(); } function setVisible(value) { diff --git a/capsules/system/browser/model-management.css b/capsules/system/browser/model-management.css index d87a1de9f..709773a38 100644 --- a/capsules/system/browser/model-management.css +++ b/capsules/system/browser/model-management.css @@ -1,14 +1,11 @@ -/* Shared tokens follow the active System or Marketplace theme. */ .model-management[hidden] { display: none; } -.model-management { min-width: 0; color: var(--el-text); font-size: 13px; } -.model-management > p:empty { display: none; } -.model-row { padding: 16px; margin-top: 12px; border: 1px solid var(--el-hairline); border-radius: 8px; background: var(--el-surface-raised); min-width: 0; } -.model-row h2 { font-size: 16px; margin: 0 0 8px; } +.model-management { min-width: 0; color: var(--ink); } +.model-row { padding: 16px 0; border-top: 1px solid var(--line); min-width: 0; } +.model-row h2 { font-size: 18px; margin: 0 0 12px; } .model-row p { overflow-wrap: anywhere; line-height: 1.5; margin: 8px 0; } -.model-row .model-size, .model-row .model-hint, .model-details { color: var(--el-muted); font-size: 12px; } -.model-details summary { cursor: pointer; } -.model-details p { font-size: 11px; } -.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; margin: 12px 0; } -.model-keep { display: inline-flex; align-items: center; gap: 8px; cursor: pointer; } -.model-keep input { accent-color: var(--el-accent); width: 16px; height: 16px; margin: 0; } -.model-row progress { width: 100%; accent-color: var(--el-accent); } +.model-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 16px; } +.model-keep { display: inline-flex; align-items: center; gap: 8px; } +.model-keep input { accent-color: var(--accent); } +.model-row progress { width: 100%; max-width: 440px; accent-color: var(--accent); } +.model-list { list-style: none; margin: 12px 0; padding: 0; display: grid; gap: 12px; } +.model-list li { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; } diff --git a/capsules/system/browser/model-management.js b/capsules/system/browser/model-management.js index 2ba57fcb3..a770cc212 100644 --- a/capsules/system/browser/model-management.js +++ b/capsules/system/browser/model-management.js @@ -28,6 +28,33 @@ } else check(!r.admitted && !r.kept && !r.dispatch_ready); return r; } + function catalogEntries(catalog) { + check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); + check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); + if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); + const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); + check(entries.length <= 8); + if (!entries.length) return []; + check(catalog.model_catalog_state === "verified"); + const seen = new Set(); + for (const item of entries) { + check(typeof item.cid === "string" && !seen.has(item.cid)); + seen.add(item.cid); + check(item.role === "content" && item.installed === false && item.launchable === false + && CID.test(item.cid) && text(item.title) && text(item.publisher_did, 256) + && item.publisher_did.startsWith("did:") && item.signature_state === "catalog-signature-verified" + && Number.isSafeInteger(item.content_size_bytes) && item.content_size_bytes > 0); + parseRuntime(item.model_runtime, item.cid); + } + return entries; + } + function selectCatalogEntry(entries, selectedCid) { + if (selectedCid) { + const matches = entries.filter(item => item.cid === selectedCid); + return matches.length === 1 ? matches[0] : null; + } + return entries.length === 1 ? entries[0] : null; + } function formatBytes(bytes) { const units = ["B", "KB", "MB", "GB", "TB"]; const rank = bytes ? Math.min(4, Math.floor(Math.log(bytes) / Math.log(1000))) : 0; @@ -48,9 +75,9 @@ dispatch_ready: output.dispatch_ready, offer_id: output.offer_id, preparation }, cid); } window.ElastosModelManagement = { - create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: selectedCid = null, compact = false }) { + create({ root, capsule, token, buttonClass = "pc2-btn pc2-btn-secondary", cid: requiredCid = null, compact = false, onReadyOpen = null }) { let visible = false, closed = false, generation = 0, busy = false, timer, reader; - let model = null, methods = new Map(), message = "", loading = false, polls = 0; + let selectedCid = requiredCid, choices = [], model = null, methods = new Map(), message = "", loading = false, polls = 0; let unresolvedUse = null, reconcileRequired = false; let pendingFocus = null; root.classList.add("model-management"); @@ -82,20 +109,10 @@ return { signal: value.signal, abort: () => value.abort(), done: () => clearTimeout(timeout) }; } function parseCatalog(catalog) { - check(catalog.schema === "elastos.capsules.catalog/v1" && Array.isArray(catalog.capsules) && catalog.capsules.length <= 1024); - check(["verified", "unconfigured", "unavailable"].includes(catalog.model_catalog_state)); - if (catalog.model_catalog_state === "unavailable") throw new Error("Model trust unavailable"); - const entries = catalog.capsules.filter(e => e.source === "signed-model-catalog"); - check(entries.length <= 1 && (!entries.length || catalog.model_catalog_state === "verified")); + const entries = catalogEntries(catalog); + choices = entries; if (!entries.length) return null; - const entry = entries[0]; - if (selectedCid && entry.cid !== selectedCid) return null; - check(entry.role === "content" && entry.installed === false && entry.launchable === false - && CID.test(entry.cid) && text(entry.title) && text(entry.publisher_did, 256) - && entry.publisher_did.startsWith("did:") && entry.signature_state === "catalog-signature-verified" - && Number.isSafeInteger(entry.content_size_bytes) && entry.content_size_bytes > 0); - parseRuntime(entry.model_runtime, entry.cid); - return entry; + return selectCatalogEntry(entries, requiredCid || selectedCid); } function bindMethods(registry) { check(Array.isArray(registry.interfaces) && registry.interfaces.length <= 1024); @@ -129,7 +146,7 @@ if (model?.cid !== unresolvedUse?.cid || model?.model_runtime.preparation) unresolvedUse = null; } catch { if (epoch !== generation || !show()) return; - model = null; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; + model = null; choices = []; message = "Models are unavailable. Check your connection and trusted catalog, then retry."; } finally { io.done(); if (epoch === generation) { loading = false; render(); schedule(); } @@ -216,6 +233,20 @@ const status = element("p", loading ? "Loading models…" : message); status.setAttribute("role", "status"); root.append(status); if (!model) { + if (!loading && !message && !requiredCid && choices.length > 1) { + root.append(element("p", "Select a verified model. Keep and removal apply to the model you select.")); + const list = element("ul", "", "model-list"); + for (const item of choices) { + const row = element("li"); + const pick = button(item.title, () => { selectedCid = item.cid; void load(); }); + pick.dataset.modelControl = `select-${item.cid}`; + row.append(pick, element("span", formatBytes(item.content_size_bytes), "model-size")); + list.append(row); + } + root.append(list); + restoreFocus(); + return; + } if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); restoreFocus(); return; @@ -241,6 +272,11 @@ const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); use.dataset.modelControl = "use"; if (!active(p) && !r.dispatch_ready) controls.append(use); + if (r.dispatch_ready && typeof onReadyOpen === "function") { + const open = button("Open in Assistant", () => onReadyOpen({ cid: model.cid, offer_id: r.offer_id })); + open.dataset.modelControl = "open-assistant"; + controls.append(open); + } if (active(p)) { const cancel = button("Cancel preparation", () => void act("cancel"), p.cancel_requested || reconcileRequired); cancel.dataset.modelControl = "cancel"; controls.append(cancel); } const labelNode = element("label", "", "model-keep"), toggle = element("input"); toggle.type = "checkbox"; toggle.checked = r.kept; toggle.disabled = !(r.admitted || active(p)) || p?.cancel_requested || busy || loading || reconcileRequired; toggle.dataset.modelControl = "keep"; @@ -252,6 +288,11 @@ : "Prepared files can be removed during cache cleanup.", "model-hint")); row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); root.append(row); + if (!requiredCid && choices.length > 1) { + const back = button("Choose another model", () => { selectedCid = null; model = null; void load(); }); + back.dataset.modelControl = "choose-another"; + root.append(back); + } restoreFocus(); } function setVisible(value) { diff --git a/components.json b/components.json index 8c4474e0d..da60ef85b 100644 --- a/components.json +++ b/components.json @@ -1448,8 +1448,9 @@ "linux-amd64": { "url": "https://github.com/ggml-org/llama.cpp/releases/download/b10516/llama-b10516-bin-ubuntu-x64.tar.gz", "checksum": "sha256:f263a91280471b4c33c4999d7c76259c0f3a0a53a0b3e692b2c0b84380137a35", - "extract_path": "llama-b10516/llama-server", - "install_path": "bin/llama-server" + "extract_path": "llama-b10516", + "install_path": "libexec/llama.cpp/b10516/linux-amd64", + "binary_path": "llama-server" }, "linux-arm64": { "strategy": "source-build", @@ -1558,7 +1559,9 @@ "marketplace", "archive-manager", "inbox", - "assistant" + "assistant", + "model-provider", + "llama-server" ] }, "demo": { @@ -1609,7 +1612,9 @@ "marketplace", "archive-manager", "inbox", - "assistant" + "assistant", + "model-provider", + "llama-server" ] }, "operator": { @@ -1798,5 +1803,15 @@ "model-qwen3.5-9b" ] } + }, + "model_catalog": { + "head_cid": "bafkreigidvbvotnjdljhqmao3jmtgdh56h6wveso6lv7mnsyvmhcmhjpfq", + "publisher_dids": [ + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA" + ], + "local_use": { + "max_cache_bytes": 21474836480, + "max_model_memory_bytes": 8589934592 + } } } diff --git a/docs/CAPSULE_MODEL.md b/docs/CAPSULE_MODEL.md index 7e471fcbe..6be4b0d4a 100644 --- a/docs/CAPSULE_MODEL.md +++ b/docs/CAPSULE_MODEL.md @@ -48,6 +48,27 @@ an inference offer after Runtime admits the model. The full Get, bootstrap, and external-gateway contract is in [Content capsule distribution](CONTENT_CAPSULE_DISTRIBUTION.md). +## Capability placement + +An app composes typed capabilities whose execution and state can have different +placements. Runtime acquires verified capsules for local execution or binds +approved remote services. It checks interfaces, host support, resources and +authority before effects, preserves explicit selections and explains missing +requirements. Package availability, execution grants and state access remain +separate decisions. Capsule code uses the same contract across placements. + +A local Browser UI can use a remote Chromium VM Engine, a local Exit and local +durable state. A local Assistant or Builder can use remote inference, a local +sandbox and local durable state. The inference result proposes local actions; +Runtime authorises each tool operation independently. Each advertised combination +needs compatible providers and installed proof. A provider manifest or a model +reply alone does not establish working sandbox support. + +The [shared state contract](STORAGE_AND_ACCESS.md#shared-application-state) +defines checkpoints and handoff when execution and durable storage are on +different nodes. Current accepted placements remain in state.md and the release +plan; broad placement support is a target, not a claim of hardware qualification. + ## Isolation boundary Runtime admits an executable artifact for a session and binds the instance to diff --git a/docs/CONTENT_CAPSULE_DISTRIBUTION.md b/docs/CONTENT_CAPSULE_DISTRIBUTION.md index 5e1d3e812..06ffb2ed6 100644 --- a/docs/CONTENT_CAPSULE_DISTRIBUTION.md +++ b/docs/CONTENT_CAPSULE_DISTRIBUTION.md @@ -113,8 +113,9 @@ workflow, file picker or editable model path. It uses this sequence: records. They show availability, Preparing with progress, Ready, or an actionable failed, offline or incompatible state. The intended Open handoff selects that model in Agent/chat and composes preparation with use; inference - waits for admission and provider readiness. Current Open Models instead - targets System and still requires the bounded adaptation below. + waits for admission and provider readiness. Open Models opens Marketplace + Models. A ready model can open Assistant with that exact CID. Inference + still waits for admission, provider readiness, and Send. A pin is retention, not evidence of trust, license acceptance or inference readiness. Runtime keeps paths and backend routes private. Selection preserves drafts and existing runs and never silently substitutes another model. @@ -226,6 +227,20 @@ declared CID/size and model metadata, with installed and launchable both false. The caller's admission and exact-offer dispatch readiness determine whether the model state is unprepared, admitted or ready. A same-name local directory does not establish admission. + +A timed catalog carries `expires_at` and stops verifying exactly at that +second. An explicitly pinned snapshot that omits `expires_at` is a permanent +publisher statement; canonical publisher output omits the key, and a JSON +`null` decodes the same way. Runtime readers older than this change require +the field and reject a permanent snapshot, so those Runtimes update before an +operator pins one. Re-pinning `model_catalog.head_cid` binds only new +preparation records. An admitted package keeps its bytes and its original +receipt, while readiness follows the current head: startup composition and +dispatch readiness cover only records bound to the current head, so the +package reports admitted and not ready until one explicit authorized same-CID +Use under the new pin creates an alias record that reuses the admitted bytes +without another payload fetch. + Absent model configuration preserves ordinary installed inventory; invalid model configuration marks only the model catalog unavailable. Snapshot verification provides publisher metadata. Transfer, atomic admission and @@ -244,9 +259,33 @@ startup, pin, retry or fallback for this mode. Runtime validates the private applied-range receipt against the exact CID, relative path, range and byte count. It consumes the range once and removes -the receipt from Bytes and Stream output. Missing or conflicting receipts fail; -bounded Content failures do not enter ordinary availability fallback. Remote -bounded calls are outside this local contract. +the receipt from Bytes and Stream output. Missing or conflicting receipts fail +the local read. A bounded local miss then continues to the availability plane +with the same CID, path and bound inside the request, and Runtime's outer +invocation carries no range of its own, so the bytes are sliced exactly once. +Carrier availability tries announced holders one at a time in the reputation +order the replication path already uses, and records each holder's outcome, so +a holder that failed one piece drops behind one that served the next. The +verified announcement signer is the peer the hop must reach: the ticket's +endpoint has to belong to that key, a mismatch is refused before any connect +and counts as that signer's failure, so only the authenticated holder earns +credit for bytes it served. For the bounded Content fetch, and only for it, +the route timeout also bounds the holder's answer; connect and answer are +separate budgets of that size, so a connected holder that stays silent or +trickles costs the caller at most one answer budget on top of its connect +before the next holder is tried, and a preparation that was cancelled or +expired while one such read waited settles on its next check once that read +returns. Other Carrier operations keep their open-ended answer, because +cutting one could discard the receipt of an effect that is still running on +the peer. Each hop forwards Content's bounded fetch to the +holder's Runtime, which applies the bound against its own local backend under +the same receipt checks, and the consumer accepts only a payload whose length +is the requested range exactly, or one to `max_bytes` for metadata; a bounded +Carrier response line above 128 KiB is refused before it is buffered. +`local_only` reads stay local. A consumer with an empty cache therefore moves +a package as 64 KiB pieces without any Runtime holding a whole file, and the +caller still compares the metadata it receives against the signed catalog +before trusting it. Complete metadata uses a maximum of 64 KiB and requests at most that cap plus one byte from Kubo. Success requires EOF within the cap. Runtime checks the @@ -307,7 +346,7 @@ content CID and the consumer's typed model-run authority. | --- | --- | | `gateway_capsule_catalog/read_model.rs` already merges signed passive model entries into the capsule catalog. | Make Marketplace the primary content discovery path; retain System management and one catalog. Review the exact-one-entry validation and both helpers' at-most-one limit before broader choices. | | Generic `CapsuleManifest.viewer`, viewer compatibility audit and Home open-target handoff describe a content consumer. | The model profile rejects `viewer`; the current audit also requires an installed viewer-role target with a content interface. Agent/Assistant use their own roles and model interfaces. Map compatible selection explicitly rather than removing checks or relabeling execution as passive content. | -| Library's protected-video handoff and Home launch context keep identity separate from authority. | Define the smallest exact-CID selection handoff into Agent/chat, preserving drafts, current launch binding and deliberate run intent. Current ready-only choices and Open Models to System do not provide this flow. | +| Library's protected-video handoff and Home launch context keep identity separate from authority. | Define the smallest exact-CID selection handoff into Agent/chat, preserving drafts, current launch binding and deliberate run intent. Open Models now opens Marketplace Models, and a ready result can select that exact CID in Assistant. | | Runtime preparation inventory, private artifact descriptors and the existing model provider own admission, retention and execution. | Compose selection with preparation while retaining fail-closed admission, exact offer binding and cancellation/outcome ownership. The content manifest supplies no execution capability. | | Content/availability own delivery; Carrier is the private off-box transport. | Adapt bounded local preparation reads to that delivery contract with finite byte/time/cancellation limits. Both the ordinary whole-file materializer and `viewer_gateway::viewer_content`, which reads the entrypoint into browser-delivered bytes, are unsuitable for model weights. Reuse handoff identity, not those byte paths. | @@ -407,8 +446,9 @@ Implemented source boundaries and remaining acceptance: Assistant/Home Agent keep exact offer/CID intent in existing workspaces and use current unique ready mappings. Missing choices and failed refreshes preserve drafts and accepted runs; new dispatch requires current readiness - and deliberate Send. Current Open Models uses the Home handoff to System; - the content-to-Agent review above covers the intended discovery/use flow. + and deliberate Send. Open Models uses the Home handoff to Marketplace Models. + A ready result can select that exact CID in Assistant; + the content-to-Agent review above covers the remaining discovery/use flow. Source fixtures cover the current boundaries. Verify the combined installed views, offline/incompatible states and busy-safe eviction with the exact model. Keep ordinary app catalog behavior and hosted configuration unchanged. diff --git a/docs/MODEL_PACKAGE_HANDOFF.md b/docs/MODEL_PACKAGE_HANDOFF.md new file mode 100644 index 000000000..5d6f232bd --- /dev/null +++ b/docs/MODEL_PACKAGE_HANDOFF.md @@ -0,0 +1,242 @@ +# Model package handoff + +This document tells a sender operator and a receiver how to move one signed +model content package from the sender's Runtime-managed Kubo into the +receiver's Runtime-managed Kubo, and how the receiver pins the sender's signed +permanent catalog snapshot so their Runtime can run normal local Use. The +helper is `scripts/model-package-handoff.py` and needs only Python 3. Catalog +and package rules live in +[CONTENT_CAPSULE_DISTRIBUTION.md](CONTENT_CAPSULE_DISTRIBUTION.md). + +## Purpose and boundaries + +The package travels as a CAR file. A CAR is a portable copy of the exact block +closure under the package root CID, so the receiver's Kubo ends up with the +same blocks and the same root the sender holds. CAR transfer is package +portability between two operators. Cold Content or Carrier Get for models +remains a separate, unaccepted flow; this handoff does not stand in for it. + +Two pins exist and stay separate. The Content retention pin is a Kubo recursive +pin of the package root CID inside the receiver's Kubo; it keeps the bytes. The +catalog trust pin is `components.json.model_catalog.head_cid`, the raw CIDv1 +SHA-256 of the exact bytes of the signed `model-catalog.json` snapshot; it tells +the Runtime which publisher statement to trust. The helper writes each pin with +its own subcommand, removing one leaves the other untouched, and neither pin +implies the other. + +The Runtime verifies the catalog signature when it reads the snapshot. The +helper checks the head CID, the schema, the entry, the signer DID and the +permanent-snapshot rule. It does not verify the signature, and its receipt +says `"signature_verification": "pending_runtime_read"` until the Runtime has +read and verified the snapshot. A permanent snapshot omits `expires_at`. +Runtime readers older than the permanent-snapshot change require that field +and reject a permanent snapshot, so the receiver's Runtime includes that +change before the receiver pins one. + +Local Use also needs the platform llama.cpp engine. The Runtime derives +dispatch readiness from the current signed entry, the admission, the verified +`llama-server` engine receipt and one matching offer from the local model +provider, and its local model startup profile is accepted on `darwin-arm64` +only. A receiver on another host admits the package and stays not ready. + +The helper streams every transfer in 1 MiB chunks and hashes as it goes, so a +6 GB package stays on disk and travels as raw bytes end to end. + +## Sender steps + +The sender holds the package in their Runtime's Kubo. The example package root +is `bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi` with a payload +of 6169366387 bytes; the CAR is a little larger than the payload because it +carries block headers and the DAG-PB directory nodes. + +1. Build the permanent catalog payload. It carries the schema, `published_at`, + and one to eight entries. Each entry `cid` is a package root, its + `capsule_manifest` is that package's `capsule.json` object, and its + `object_manifest` is its `_elastos_object.json` object. Package CIDs and + capsule names stay unique. Leave `expires_at` out. Serialize it with sorted + keys and no whitespace, which is what the Runtime signs over. The example + below shows one entry: + + ```bash + PAYLOAD=$(jq -cS -n \ + --arg cid bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi \ + --argjson capsule "$(cat package/capsule.json)" \ + --argjson object "$(cat package/_elastos_object.json)" \ + '{schema: "elastos.model.catalog/v1", + published_at: (now | floor), + entries: [{cid: $cid, capsule_manifest: $capsule, + object_manifest: $object}]}') + ``` + +2. Sign the payload with the publisher key and assemble the envelope. The + Runtime accepts the envelope keys `payload`, `signature` and `signer_did` + and nothing else: + + ```bash + SIGNED=$(printf '%s' "$PAYLOAD" | \ + elastos sign-payload --domain elastos.model.catalog.v1 --key "$KEY") + jq -cS -n --argjson payload "$PAYLOAD" \ + --arg signature "$(jq -r .signature <<<"$SIGNED")" \ + --arg signer_did "$(jq -r .signer_did <<<"$SIGNED")" \ + '{payload: $payload, signature: $signature, signer_did: $signer_did}' \ + > model-catalog.json + ``` + +3. Compute the head CID of the exact file bytes. The receiver pins this value, + so record it as `HEAD_CID` and hand it over as its own item: + + ```bash + python3 scripts/model-package-handoff.py head-cid \ + --catalog model-catalog.json + ``` + +4. Export the CAR from the Runtime's Kubo. The helper reads the Kubo API port + from `ipfs-coords.json` in the Runtime data dir, checks that Kubo is 0.40.1 + with the canonical import profile, streams the closure to + `qwen-package.car.partial`, renames it on success and writes the receipt: + + ```bash + python3 scripts/model-package-handoff.py export \ + --data-dir "${XDG_DATA_HOME:-$HOME/.local/share}/elastos" \ + --cid bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi \ + --output qwen-package.car + ``` + + The receipt `qwen-package.car.receipt.json` records the package CID, the + CAR SHA-256, the CAR size, the Kubo version and the export time. + +5. Hand over five things: `qwen-package.car`, `qwen-package.car.receipt.json`, + `model-catalog.json`, the exact `HEAD_CID` from step 3, and the publisher + DID printed by `sign-payload`. Send the head CID and the publisher DID in + text the receiver can compare against, not only inside the files. Any + durable file transport works, for example a USB drive, rsync or an object + store. The CAR is about 6.2 GB, so plan for a resumable transport; this path + moves the file as raw bytes and has no base64 or whole-file upload API. + +## Receiver steps + +The receiver runs these on the Home that will keep the package, with the +Runtime's Kubo running. `DATA_DIR` is the Runtime data dir, by default +`~/.local/share/elastos` on Linux and `~/Library/Application Support/elastos` +on macOS. + +1. Confirm the Runtime's Kubo is the pinned version with the canonical import + profile, and install the platform llama.cpp engine if the Runtime does not + already hold its verified receipt: + + ```bash + python3 scripts/model-package-handoff.py verify-kubo --data-dir "$DATA_DIR" + elastos setup --with llama-server + ``` + +2. Check the CAR against its receipt before Kubo sees it. This recomputes the + size and SHA-256 by streaming the file and refuses a truncated or altered + CAR or a receipt for a different package: + + ```bash + python3 scripts/model-package-handoff.py verify-car \ + --car qwen-package.car --receipt qwen-package.car.receipt.json \ + --cid bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi + ``` + +3. Import and pin. The command repeats the CAR check, verifies Kubo, reads the + Kubo repo path and refuses unless the free space left after the import stays + at or above the floor you name. The floor is your budget and the command + requires it. Kubo pins the root recursively during the import and the helper + confirms the pin afterwards. Kubo answers only after the whole CAR has + arrived and the root is pinned, so give the 6 GB import a timeout that + covers that wait: + + ```bash + python3 scripts/model-package-handoff.py import \ + --data-dir "$DATA_DIR" \ + --car qwen-package.car --receipt qwen-package.car.receipt.json \ + --cid bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi \ + --free-space-floor-bytes 21474836480 \ + --timeout-seconds 1800 + ``` + + The receipt reports `"pinned": true` with the block count and block bytes + Kubo stored. Running the same command again succeeds and leaves the pin in + place. + +4. Pin the catalog. First compare the handed-over `HEAD_CID` with the head of + the bytes you received; `pin-catalog` refuses a mismatch as well: + + ```bash + python3 scripts/model-package-handoff.py head-cid --catalog model-catalog.json + ``` + + The helper then checks the envelope, refuses a timed catalog, installs + `model-catalog.json` into `DATA_DIR` at mode 0600 and replaces only the + `model_catalog` key of `components.json`, leaving every other key as it + was. The local Use budgets are yours to set and the Runtime rejects zero. + Two floors apply to this package. The Qwen manifest declares + `minimum_memory_mb` 8192, so `--max-model-memory-bytes` is at least + 8589934592. The Runtime charges the cache three times the payload plus + 8 MiB and 192 KiB of index allowance, which is 18516684377 bytes for the + 6169366387-byte payload, so `--max-cache-bytes` is at least that; the + example uses 20 GiB: + + ```bash + python3 scripts/model-package-handoff.py pin-catalog \ + --data-dir "$DATA_DIR" \ + --catalog model-catalog.json \ + --head-cid "$HEAD_CID" \ + --publisher-did "$PUBLISHER_DID" \ + --package-cid bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi \ + --max-cache-bytes 21474836480 \ + --max-model-memory-bytes 8589934592 + ``` + + `DATA_DIR` has to be owned by you and closed to group and other writes, + which is how the Runtime reads it as well. + +5. Restart the Runtime with the platform restart helper from + [HOME_LOCAL_SETUP.md](HOME_LOCAL_SETUP.md), then check readiness. + `elastos content status --cid ` reports the package, and the + System view in Home shows the model catalog as verified with the sender as + publisher. Run local Use from Home as usual. That first successful Runtime + read is the signature verification the `pin-catalog` receipt left pending. + +If this Home already admitted the same package under a previous catalog head, +the admitted bytes and the original receipt stay in place, and readiness +follows the current head. Until you act, the Runtime reports the package as +admitted and not ready, and startup composition offers nothing for it. One +explicit same-CID Use under the new pin creates an alias record that reuses +the admitted bytes, so that Use completes without a second download and the +package becomes ready again. + +## Failure handling + +A truncated or corrupt CAR fails at `verify-car`, and `import` runs that same +check first, so Kubo receives no request for a bad file. If a transfer was +interrupted, copy the CAR again and rerun the same commands; every step +converges to the same end state, and a repeated `import` or `pin-catalog` is +safe. + +If `import` refuses on the free-space floor, free space in the Kubo repo's +filesystem or choose a smaller floor and rerun. If Kubo reports an import error +mid-stream, the helper prints Kubo's message and the root stays unpinned. +Blocks that Kubo already wrote before the failure stay in its blockstore as +unpinned garbage until its next collection; a rerun with a complete CAR +deduplicates them, pins the root, and converges to the same end state. On the +synthetic closure a truncated import committed nothing; the helper does not +rely on that for larger packages. + +The two pins come off independently. `kubo pin rm ` (with +`IPFS_PATH` set to the Runtime's Kubo repo) releases the retention pin and +leaves `components.json` alone. Removing the `model_catalog` key from +`components.json` and restarting the Runtime withdraws the catalog trust pin +and leaves the Kubo pin alone. + +## Limits + +The helper and this document are verified against a synthetic closure and a +test-only signing key on two offline Kubo 0.40.1 daemons +(`scripts/model-package-handoff-test.py`). The Runtime side is proved by unit +tests: permanent-snapshot verification and same-CID alias reuse run on Linux, +and the readiness and startup-composer test for a permanent re-pin is gated to +`darwin-arm64` and waits for a Mac run. The real Qwen package, the production +signing key, installation on a target Home and a second-Home reply proof are +operator work and are recorded as evidence when they happen. diff --git a/docs/STORAGE_AND_ACCESS.md b/docs/STORAGE_AND_ACCESS.md index 0042e6a87..74d2f3312 100644 --- a/docs/STORAGE_AND_ACCESS.md +++ b/docs/STORAGE_AND_ACCESS.md @@ -69,6 +69,57 @@ Identity resolution and key updates remain behind existing identity contracts. Private keys, custody shares, bearer access secrets, and provider routes belong inside their private boundaries rather than public manifests. +## Shared application state + +Browser profiles, game saves, Assistant conversations, Builder work and logs use +the same object ownership, protection, revision, availability and recovery +mechanics. Their mutable state stays separate from executable capsules, model +packages and service offers. Updating a capsule preserves the identity and +permissions of the user's state. + +Each payload keeps its own format and save boundary. Chromium needs a consistent +profile checkpoint; a game supplies save bytes; conversations preserve drafts +and message history; run journals preserve ordered events and durable operation +identities. Reuse existing protected storage, content and authority mechanisms +with narrow payload adapters. A common contract does not require one file layout +or a new storage service. + +A successful save is acknowledged by the selected durable store after it commits +a usable revision. Interrupted writes, exhausted +storage and failed migration preserve the previous committed revision and report +unsaved work. Local commit and confirmed remote retention are separate outcomes. +Copying an actively mounted Browser disk is not a consistent checkpoint. One +Runtime holds writable ownership of a Browser profile; transfer fences that +writer before another begins. Other payloads declare merge or conflict behavior. + +Durable storage placement is separate from execution placement. With a remote +Browser Engine and local storage, the consumer's store retains the canonical +checkpoint and the Engine receives an authorised working copy. Checkpoints +return through the common protected-state and content path. A remote working +copy alone cannot satisfy a local-save claim. Host failure preserves the last +acknowledged checkpoint; pending edits and unresolved writer ownership remain +explicit. Availability peers retain only the objects selected by the owner's +storage and access policy. + +Recovery restores the selected revision and usable keys under fresh authority on +a compatible Runtime. Restored conversations and logs retain operation identity +without replaying inference, tools, payments or approvals. Active authentication +sessions and stale grants are not recovered as authority. A storage replica holds +protected bytes; an execution provider separately needs permission to use +readable state within an accepted execution boundary. + +Logs share storage and protection mechanics while retaining their own semantics. +User history, provider settlement journals and Runtime audit records each have +an explicit owner, integrity rule and bounded retention policy. Secret redaction, +pruned-history reporting and pending replica cleanup are part of that policy. +User-facing log controls confer only the rights granted over those records. + +The release plan's Required SA1-SA6 criteria qualify this contract for each +shipped payload. Reuse common mechanism tests, then prove actual payload saves, +restore and failure handling. Existing game-save checks cover the shipped GBA +surface; this contract adds no separate emulator product. Current implementation +and acceptance limits remain in state.md. + ## The everyday object journey | Action | Required behavior | diff --git a/elastos/crates/elastos-common/src/manifest.rs b/elastos/crates/elastos-common/src/manifest.rs index e206eced3..0adb836ca 100644 --- a/elastos/crates/elastos-common/src/manifest.rs +++ b/elastos/crates/elastos-common/src/manifest.rs @@ -189,7 +189,7 @@ impl ModelContentMetadata { ); } if self.format != "gguf" - || self.quantization != "Q4_K_M" + || !matches!(self.quantization.as_str(), "Q4_K_M" | "Q8_0") || self.engine != "llama.cpp" || self.consumer_interface != "elastos.provider.model" || self.consumer_interface_version != "0.1.0" @@ -1171,6 +1171,12 @@ mod tests { serde_json::to_value(manifest).unwrap()["model_content"], value["model_content"] ); + let mut q8 = value.clone(); + q8["model_content"]["quantization"] = serde_json::json!("Q8_0"); + serde_json::from_value::(q8) + .unwrap() + .validate() + .unwrap(); } #[test] diff --git a/elastos/crates/elastos-runtime/src/provider/registry.rs b/elastos/crates/elastos-runtime/src/provider/registry.rs index 78c959a37..c433c3547 100755 --- a/elastos/crates/elastos-runtime/src/provider/registry.rs +++ b/elastos/crates/elastos-runtime/src/provider/registry.rs @@ -1692,12 +1692,29 @@ fn apply_provider_transfer_response( const MAX_BOUNDED_PROVIDER_READ_BYTES: u64 = 64 * 1024; +/// Content and the availability plane carry a bounded read to the backend or +/// holder that applies it once; Runtime does not slice their responses again. +/// Content may make that hop over Carrier to a remote holder, whose own +/// Runtime applies the bound locally. Only local IPFS Cat applies a range. +fn bounded_read_nested_owner(invocation: &ProviderInvocation) -> bool { + match (invocation.target.as_str(), invocation.op.as_str()) { + ("content", "fetch") => matches!( + invocation.transport, + ProviderInvocationTransport::Local | ProviderInvocationTransport::Carrier(_) + ), + ("availability", "fetch") => { + matches!(invocation.transport, ProviderInvocationTransport::Local) + } + _ => false, + } +} + fn bounded_provider_metadata_max( invocation: &ProviderInvocation, ) -> Result, ProviderError> { if !matches!( (invocation.target.as_str(), invocation.op.as_str()), - ("ipfs", "cat") | ("content", "fetch") + ("ipfs", "cat") | ("content", "fetch") | ("availability", "fetch") ) { return Ok(None); } @@ -1722,14 +1739,17 @@ fn bounded_provider_metadata_max( .as_ref() .and_then(|p| p.expected_bytes) .is_some() - || !matches!(invocation.transport, ProviderInvocationTransport::Local) + || !(matches!(invocation.transport, ProviderInvocationTransport::Local) + || bounded_read_nested_owner(invocation)) || invocation.transfer == ProviderTransfer::Json { return Err(invalid()); } match (invocation.target.as_str(), invocation.op.as_str()) { ("ipfs", "cat") => Ok(Some(max)), - ("content", "fetch") => Ok(None), // Content validates its own nested request/receipt. + // Content validates its own nested request/receipt; the availability + // plane forwards Content's bound to the holder. + ("content", "fetch") | ("availability", "fetch") => Ok(None), _ => Err(invalid()), } } @@ -1796,7 +1816,7 @@ fn bounded_provider_read_range( if invocation.request.get("max_bytes").is_some() && matches!( (invocation.target.as_str(), invocation.op.as_str()), - ("ipfs", "cat") | ("content", "fetch") + ("ipfs", "cat") | ("content", "fetch") | ("availability", "fetch") ) { bounded_provider_metadata_max(invocation)?; @@ -1811,13 +1831,7 @@ fn bounded_provider_read_range( )) } } - // Content owns its nested fetch range. Its outer call must not - // slice the already-normalized backend response again. - if invocation.target == "content" - && invocation.op == "fetch" - && invocation.range.is_none() - && matches!(invocation.transport, ProviderInvocationTransport::Local) - { + if invocation.range.is_none() && bounded_read_nested_owner(invocation) { return Ok(None); } if invocation.target != "ipfs" @@ -3484,23 +3498,30 @@ mod tests { } #[tokio::test] - async fn complete_metadata_rejects_carrier_before_dispatch() { + async fn complete_metadata_rejects_raw_carrier_backend_but_forwards_content() { let registry = ProviderRegistry::new(); let carrier = Arc::new(MockCarrierInvoker::default()); registry.set_carrier_invoker(carrier.clone()).await; - for (target, op) in [("ipfs", "cat"), ("content", "fetch")] { - let (mut invocation, _) = complete_metadata_fixture(ProviderTransfer::Bytes); - invocation.target = target.into(); - invocation.op = op.into(); - invocation.request["op"] = serde_json::json!(op); - invocation.transport = - ProviderInvocationTransport::Carrier(ProviderCarrierRoute::PeerDid { - peer_did: "did:key:zFixture".into(), - timeout_ms: Some(5000), - }); - assert!(registry.invoke_provider(invocation).await.is_err()); - } + let route = ProviderInvocationTransport::Carrier(ProviderCarrierRoute::PeerDid { + peer_did: "did:key:zFixture".into(), + timeout_ms: Some(5000), + }); + let (mut raw, _) = complete_metadata_fixture(ProviderTransfer::Bytes); + raw.transport = route.clone(); + assert!(registry.invoke_provider(raw).await.is_err()); assert!(carrier.requests.lock().await.is_empty()); + // Content carries the metadata bound to the holder's Runtime, which + // applies it against its own local backend. + let (mut content, _) = complete_metadata_fixture(ProviderTransfer::Bytes); + content.target = "content".into(); + content.op = "fetch".into(); + content.request["op"] = serde_json::json!("fetch"); + content.transport = route; + let _ = registry.invoke_provider(content).await; + let requests = carrier.requests.lock().await; + assert_eq!(requests.len(), 1); + assert_eq!(requests[0]["request"]["max_bytes"], 65536); + assert_eq!(requests[0]["request"]["bounded_read"], true); } #[test] @@ -3597,7 +3618,7 @@ mod tests { } #[tokio::test] - async fn bounded_read_rejects_remote_content_and_outer_range_before_dispatch() { + async fn bounded_read_forwards_remote_content_but_rejects_outer_range_and_raw_backend() { let registry = ProviderRegistry::new(); let carrier = Arc::new(MockCarrierInvoker::default()); registry.set_carrier_invoker(carrier.clone()).await; @@ -3607,12 +3628,38 @@ mod tests { invocation.request["op"] = serde_json::json!("fetch"); invocation.range = None; assert!(bounded_provider_read_range(&invocation).is_ok()); - let mut remote = invocation.clone(); - remote.transport = ProviderInvocationTransport::Carrier(ProviderCarrierRoute::PeerDid { + let route = ProviderInvocationTransport::Carrier(ProviderCarrierRoute::PeerDid { peer_did: "did:key:zFixture".into(), timeout_ms: Some(5000), }); - assert!(matches!(registry.invoke_provider(remote).await, + // Content carries the bound inside its request; the holder's Runtime + // applies it, so the hop reaches the Carrier invoker unchanged. + let mut remote = invocation.clone(); + remote.transport = route.clone(); + assert!(bounded_provider_read_range(&remote).is_ok()); + let _ = registry.invoke_provider(remote).await; + { + let requests = carrier.requests.lock().await; + assert_eq!(requests.len(), 1); + assert_eq!(requests[0]["request"]["bounded_read"], true); + assert!(requests[0]["request"]["_runtime_invocation"]["range"].is_null()); + } + let mut availability = invocation.clone(); + availability.target = "availability".into(); + assert!(bounded_provider_read_range(&availability).is_ok()); + availability.transport = route.clone(); + assert!(matches!(bounded_provider_read_range(&availability), + Err(ProviderError::Provider(message)) if message.contains("bounded read requires local IPFS"))); + let mut raw = invocation.clone(); + raw.target = "ipfs".into(); + raw.op = "cat".into(); + raw.request["op"] = serde_json::json!("cat"); + raw.range = Some(ProviderByteRange { + start: 8, + end: Some(11), + }); + raw.transport = route; + assert!(matches!(registry.invoke_provider(raw).await, Err(ProviderError::Provider(message)) if message.contains("bounded read requires local IPFS"))); invocation.range = Some(ProviderByteRange { start: 8, @@ -3620,7 +3667,7 @@ mod tests { }); assert!(matches!(registry.invoke_provider(invocation).await, Err(ProviderError::Provider(message)) if message.contains("bounded read requires local IPFS"))); - assert!(carrier.requests.lock().await.is_empty()); + assert_eq!(carrier.requests.lock().await.len(), 1); } #[tokio::test] diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory.rs b/elastos/crates/elastos-server/src/api/capsule_inventory.rs index 661224c90..ed786bfc9 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory.rs @@ -14,6 +14,7 @@ const DEV_CAPSULES_ROOT: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../../../c const MODEL_CATALOG_FILE: &str = "model-catalog.json"; const MODEL_CATALOG_DOMAIN: &str = "elastos.model.catalog.v1"; const MAX_MODEL_CATALOG_BYTES: usize = 128 * 1024; +const MAX_MODEL_CATALOG_ENTRIES: usize = 8; pub(crate) struct VerifiedModelCatalogEntry { pub cid: String, @@ -36,7 +37,8 @@ struct SignedModelCatalog { struct ModelCatalogPayload { schema: String, published_at: u64, - expires_at: u64, + /// `None` is a permanent publisher statement for the pinned snapshot. + expires_at: Option, entries: Vec, } @@ -151,13 +153,16 @@ fn verify_model_catalog( let payload = signed.payload; if payload.schema != "elastos.model.catalog/v1" || payload.published_at > now - || payload.expires_at <= now - || payload.expires_at <= payload.published_at - || payload.entries.len() != 1 + || payload + .expires_at + .is_some_and(|expires| expires <= now || expires <= payload.published_at) + || !(1..=MAX_MODEL_CATALOG_ENTRIES).contains(&payload.entries.len()) { - anyhow::bail!("model catalog requires one current signed entry"); + anyhow::bail!("model catalog requires 1 to 8 unique current signed entries"); } - let mut verified = Vec::with_capacity(1); + let mut verified = Vec::with_capacity(payload.entries.len()); + let mut seen_cids = BTreeSet::new(); + let mut seen_names = BTreeSet::new(); for entry in payload.entries { if entry.cid.len() > 128 { anyhow::bail!("model package CID exceeds its bound"); @@ -171,6 +176,9 @@ fn verify_model_catalog( { anyhow::bail!("model package requires a canonical DAG-PB SHA-256 closure CIDv1"); } + if !seen_cids.insert(entry.cid.clone()) { + anyhow::bail!("model catalog entries require unique package CIDs"); + } if entry .object_manifest .get("publisher_did") @@ -184,6 +192,9 @@ fn verify_model_catalog( &entry.capsule_manifest, &entry.object_manifest, )?; + if !seen_names.insert(manifest.name.clone()) { + anyhow::bail!("model catalog entries require unique capsule names"); + } verified.push(VerifiedModelCatalogEntry { cid: entry.cid, publisher_did: signed.signer_did.clone(), @@ -387,6 +398,28 @@ pub(crate) mod tests { cid::Cid::new_v1(0x55, hash).to_string() } + fn refresh_catalog_entry_object_manifest(entry: &mut Value) { + let capsule_bytes = serde_json::to_vec(&entry["capsule_manifest"]).unwrap(); + let files = entry["object_manifest"]["files"].as_array_mut().unwrap(); + for file in files.iter_mut() { + if file["path"] == "capsule.json" { + file["size"] = serde_json::json!(capsule_bytes.len() as u64); + file["sha256"] = serde_json::json!(format!("{:x}", Sha256::digest(&capsule_bytes))); + } + } + let mut digest = Sha256::new(); + for file in files { + digest.update(file["path"].as_str().unwrap().as_bytes()); + digest.update(b"\0"); + digest.update(file["sha256"].as_str().unwrap().as_bytes()); + digest.update(b"\0"); + digest.update(file["size"].to_string().as_bytes()); + digest.update(b"\0"); + } + entry["object_manifest"]["content_digest"] = + serde_json::json!(format!("sha256:{:x}", digest.finalize())); + } + fn sign_model_catalog(payload: &Value) -> (crate::setup::ModelCatalogConfig, Vec) { let key = elastos_runtime::signature::SigningKey::from_bytes(&[7; 32]); let (signature, signer_did) = crate::crypto::domain_separated_sign( @@ -432,6 +465,44 @@ pub(crate) mod tests { assert!(!root.path().join("capsules/model-fixture").exists()); } + #[test] + fn model_catalog_accepts_two_distinct_entries_and_rejects_duplicate_identity() { + let mut payload = model_catalog_fixture(); + let mut second = payload["entries"][0].clone(); + second["cid"] = + serde_json::json!("bafybeihgnsjhpoktqbyspaqv6moblyny3txs5nkjdxfx7wm346odxkhlrm"); + second["capsule_manifest"]["name"] = "model-fixture-b".into(); + refresh_catalog_entry_object_manifest(&mut second); + payload["entries"].as_array_mut().unwrap().push(second); + let (trust, bytes) = sign_model_catalog(&payload); + let entries = verify_model_catalog(&trust, &bytes, 2).unwrap(); + assert_eq!(entries.len(), 2); + assert_eq!(entries[0].manifest.name, "model-fixture"); + assert_eq!(entries[1].manifest.name, "model-fixture-b"); + + let mut duplicate_cid = model_catalog_fixture(); + let entry = duplicate_cid["entries"][0].clone(); + duplicate_cid["entries"].as_array_mut().unwrap().push(entry); + let (trust, bytes) = sign_model_catalog(&duplicate_cid); + assert!(verify_model_catalog(&trust, &bytes, 2).is_err()); + + let mut duplicate_name = model_catalog_fixture(); + let mut second = duplicate_name["entries"][0].clone(); + second["cid"] = + serde_json::json!("bafybeihgnsjhpoktqbyspaqv6moblyny3txs5nkjdxfx7wm346odxkhlrm"); + duplicate_name["entries"] + .as_array_mut() + .unwrap() + .push(second); + let (trust, bytes) = sign_model_catalog(&duplicate_name); + assert!(verify_model_catalog(&trust, &bytes, 2).is_err()); + + let mut empty = model_catalog_fixture(); + empty["entries"] = serde_json::json!([]); + let (trust, bytes) = sign_model_catalog(&empty); + assert!(verify_model_catalog(&trust, &bytes, 2).is_err()); + } + #[test] fn model_catalog_rejects_empty_untrusted_publishers_and_wrong_head() { let (trust, bytes) = sign_model_catalog(&model_catalog_fixture()); @@ -527,6 +598,109 @@ pub(crate) mod tests { assert!(verify_model_catalog(&trust, &bytes, 2).is_err()); } + fn permanent_model_catalog_fixture() -> Value { + let mut payload = model_catalog_fixture(); + payload.as_object_mut().unwrap().remove("expires_at"); + payload + } + + #[test] + fn model_catalog_permanent_snapshot_verifies_at_any_later_time() { + let (trust, bytes) = sign_model_catalog(&permanent_model_catalog_fixture()); + for now in [1, 2, 4_000_000_001, u64::MAX] { + let entries = verify_model_catalog(&trust, &bytes, now).unwrap(); + assert_eq!(entries.len(), 1, "now={now}"); + assert_eq!(entries[0].manifest.name, "model-fixture", "now={now}"); + } + assert!( + verify_model_catalog(&trust, &bytes, 0).is_err(), + "publication in the future stays rejected" + ); + } + + #[test] + fn model_catalog_null_expires_at_decodes_as_permanent() { + let mut payload = model_catalog_fixture(); + payload["expires_at"] = Value::Null; + let (trust, bytes) = sign_model_catalog(&payload); + let signed: SignedModelCatalog = serde_json::from_slice(&bytes).unwrap(); + assert!(signed.payload.expires_at.is_none()); + let entries = verify_model_catalog(&trust, &bytes, u64::MAX).unwrap(); + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].manifest.name, "model-fixture"); + } + + #[test] + fn model_catalog_timed_snapshot_rejects_exactly_at_expiry() { + let mut payload = model_catalog_fixture(); + payload["expires_at"] = serde_json::json!(10); + let (trust, bytes) = sign_model_catalog(&payload); + let entries = verify_model_catalog(&trust, &bytes, 9).unwrap(); + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].manifest.name, "model-fixture"); + assert!(verify_model_catalog(&trust, &bytes, 10).is_err()); + assert!(verify_model_catalog(&trust, &bytes, 11).is_err()); + for expires_at in [1, 0] { + let mut payload = model_catalog_fixture(); + payload["expires_at"] = serde_json::json!(expires_at); + let (trust, bytes) = sign_model_catalog(&payload); + assert!( + verify_model_catalog(&trust, &bytes, 1).is_err(), + "expires_at={expires_at} never postdates published_at=1" + ); + } + } + + #[test] + fn model_catalog_permanent_snapshot_keeps_pin_signature_and_publisher_checks() { + let (trust, bytes) = sign_model_catalog(&permanent_model_catalog_fixture()); + let entries = verify_model_catalog(&trust, &bytes, u64::MAX).unwrap(); + assert_eq!(entries[0].manifest.name, "model-fixture"); + let untrusted = "did:key:untrusted"; + let mut changed = trust.clone(); + changed.head_cid = head_cid(b"different"); + assert!(verify_model_catalog(&changed, &bytes, u64::MAX).is_err()); + for publishers in [vec![untrusted.to_owned()], Vec::new()] { + let mut changed = trust.clone(); + changed.publisher_dids = publishers; + assert!(verify_model_catalog(&changed, &bytes, u64::MAX).is_err()); + } + let mut envelope: Value = serde_json::from_slice(&bytes).unwrap(); + envelope["payload"]["entries"][0]["capsule_manifest"]["name"] = "tampered".into(); + let tampered = serde_json::to_vec(&envelope).unwrap(); + let mut repinned = trust.clone(); + repinned.head_cid = head_cid(&tampered); + assert!( + verify_model_catalog(&repinned, &tampered, u64::MAX).is_err(), + "a repinned permanent payload still needs the configured signature" + ); + let mut envelope: Value = serde_json::from_slice(&bytes).unwrap(); + envelope["signer_did"] = untrusted.into(); + let reassigned = serde_json::to_vec(&envelope).unwrap(); + repinned.head_cid = head_cid(&reassigned); + assert!(verify_model_catalog(&repinned, &reassigned, u64::MAX).is_err()); + let mut payload = permanent_model_catalog_fixture(); + payload["renewal"] = true.into(); + let (trust, bytes) = sign_model_catalog(&payload); + assert!( + verify_model_catalog(&trust, &bytes, u64::MAX).is_err(), + "unknown payload fields stay rejected for permanent snapshots" + ); + } + + #[test] + fn model_catalog_permanent_snapshot_verifies_after_reread() { + let root = tempfile::tempdir().unwrap(); + write_model_catalog_fixture(root.path(), &permanent_model_catalog_fixture()); + let first = model_catalog_entries(root.path()).unwrap().unwrap(); + let second = model_catalog_entries(root.path()).unwrap().unwrap(); + for entries in [&first, &second] { + assert_eq!(entries.len(), 1); + assert_eq!(entries[0].manifest.name, "model-fixture"); + } + assert_eq!(first[0].cid, second[0].cid); + } + #[cfg(unix)] #[test] fn model_catalog_rejects_symlink_hardlink_and_writable_snapshot() { @@ -656,4 +830,36 @@ pub(crate) mod tests { BTreeSet::from(["object-provider".to_string()]) ); } + + #[test] + fn checkout_permanent_catalog_matches_components_pin() { + let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../../"); + let components: crate::setup::ComponentsManifest = + serde_json::from_slice(&std::fs::read(root.join("components.json")).unwrap()).unwrap(); + let trust = components + .model_catalog + .expect("matching install pins a signed model catalog"); + let bytes = std::fs::read(root.join(MODEL_CATALOG_FILE)).unwrap(); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let entries = verify_model_catalog(&trust, &bytes, now).unwrap(); + assert_eq!(entries.len(), 2); + assert_eq!( + entries[0].cid, + "bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi" + ); + assert_eq!(entries[0].manifest.name, "qwen3-5-9b-q4-k-m-local"); + assert_eq!( + entries[1].cid, + "bafybeidy5kfvqwg6g6pfgdfwslmhijosbeskt5b2duqdqxnc7e6fwmr72y" + ); + assert_eq!(entries[1].manifest.name, "smollm2-135m-instruct-q8-0-local"); + assert_eq!( + entries[0].publisher_did, + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA" + ); + assert_eq!(entries[1].publisher_did, entries[0].publisher_did); + } } diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs index 4f888bbe2..09d462406 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs @@ -1932,15 +1932,19 @@ async fn verify_package_identity( } fn local_model_startup_profile(platform: &str) -> anyhow::Result { - ensure!( - platform == "darwin-arm64", - "admitted model host profile is unavailable" - ); - // Runtime-owned profile from the verified local Qwen/engine proof. Catalog + // Runtime-owned profiles from verified local engine proofs. Catalog // metadata cannot tune execution. Other hosts require their own proof. + let (threads, gpu_layers) = match platform { + // Apple silicon offloads every layer to Metal (local Qwen proof). + "darwin-arm64" => (8, 99), + // Linux x86-64 runs on a bounded CPU thread budget without GPU offload + // (pinned SmolLM2 proof on the b10516 Ubuntu bundle). + "linux-amd64" => (4, 0), + _ => anyhow::bail!("admitted model host profile is unavailable"), + }; Ok(serde_json::json!({ - "context_size":4096, "parallel":1, "threads":8, "batch_threads":8, - "gpu_layers":99, "health_timeout_ms":120000, + "context_size":4096, "parallel":1, "threads":threads, "batch_threads":threads, + "gpu_layers":gpu_layers, "health_timeout_ms":120000, "shutdown_timeout_ms":5000, "enable_thinking":false })) } @@ -2056,10 +2060,12 @@ impl ModelActivation { expected.as_object_mut().unwrap().remove("stream_output"); expected["policy"].as_object_mut().unwrap().remove("schema"); expected["enabled"] = serde_json::json!(true); + // A saved descriptor is bound to the host that composed it; its engine + // receipt already carries that platform, so the same host profile applies. expected["adapter"] = serde_json::json!({"kind":"local_llama_cpp_text", "engine":{"path":engine_path,"sha256":engine_sha}, "model":{"path":model_path,"sha256":model_sha}, - "settings":local_model_startup_profile("darwin-arm64")?}); + "settings":local_model_startup_profile(&crate::setup::detect_platform())?}); ensure!( self.offer == expected, "activation descriptor binding changed" @@ -2381,6 +2387,7 @@ async fn append_admitted_model_offers_locked( .context("model startup offers unavailable")? .clone(); let mut admitted = Vec::new(); + let mut pending = Vec::new(); for entry in entries { // Aliases authorize reuse of one artifact; they do not create duplicate // offers or transfer ownership between preparation request records. @@ -2448,15 +2455,11 @@ async fn append_admitted_model_offers_locked( entrypoint: weights.path.clone(), engine_receipt_sha256: engine.receipt_sha256.clone(), }; - let inventory = Inventory::open(data_dir, false)?; - let mut durable = inventory.load()?; - ensure!( - durable.records == snapshot.records, - "model admission changed before Init" - ); - let owner = durable + // The selected record may be a reuse alias; the saved descriptor lives + // on the admission owner, so the retry comparison reads that record. + let owner = snapshot .records - .iter_mut() + .iter() .find(|r| r.operation_id == record.admission_id) .context("activation owner unavailable")?; activation.check_root(data_dir, owner)?; @@ -2466,12 +2469,36 @@ async fn append_admitted_model_offers_locked( "activation descriptor changed on retry" ); } else { - owner.activation = Some(activation); - inventory.save(&durable)?; + pending.push((owner.operation_id.clone(), activation)); } admitted.push(serde_json::json!({"offer_id":offer["id"]})); offers.push(offer); } + // Every entry was verified against the same inventory snapshot. The new + // descriptors are saved together after that verification, so a later entry + // never mistakes this composer's own write for a concurrent admission + // change, and the snapshot comparison stays strict for real changes. + if !pending.is_empty() { + let inventory = Inventory::open(data_dir, false)?; + let mut durable = inventory.load()?; + ensure!( + durable.records == snapshot.records, + "model admission changed before Init" + ); + for (owner_id, activation) in pending { + let owner = durable + .records + .iter_mut() + .find(|r| r.operation_id == owner_id) + .context("activation owner unavailable")?; + ensure!( + owner.activation.is_none(), + "activation descriptor changed before Init" + ); + owner.activation = Some(activation); + } + inventory.save(&durable)?; + } let mut extra = config.extra.clone(); extra["offers"] = serde_json::Value::Array(offers); extra["runtime_admitted_offers"] = serde_json::Value::Array(admitted); @@ -2578,6 +2605,10 @@ mod tests { struct PreparationBackend { files: std::collections::BTreeMap>, cid: Mutex, + /// Further signed packages this backend serves, keyed by CID. `files` + /// and `cid` remain the primary package that single-entry tests use. + extra_packages: + std::collections::BTreeMap>>, calls: Mutex>, capacity_requests: Mutex>, capacity_volume: u64, @@ -2589,6 +2620,7 @@ mod tests { cold_backend: AtomicBool, hold_read: AtomicBool, hold_hash: AtomicBool, + missing_cache: AtomicBool, read_fault: Mutex>, native: Option>, entered: tokio::sync::Notify, @@ -2631,7 +2663,12 @@ mod tests { match op { "cat" => { use base64::Engine as _; + if self.missing_cache.load(Ordering::Acquire) { + return Ok(serde_json::json!({"status":"error","code":"not_found", + "message":"consumer cache is empty"})); + } assert_eq!(request["bounded_read"], true); + let files = self.package_files(request["cid"].as_str().unwrap_or_default()); let path = request["path"].as_str().unwrap(); if path == "weights.gguf" && self.hold_read.swap(false, Ordering::AcqRel) { self.entered.notify_one(); @@ -2641,7 +2678,7 @@ mod tests { assert_eq!(request["max_bytes"], 65536); serde_json::json!({"_runtime_complete_metadata":{ "schema":"elastos.provider.complete-metadata/v1", "cid":request["cid"], - "path":path, "max_bytes":65536, "actual_bytes":self.files[path].len(), "completed":true + "path":path, "max_bytes":65536, "actual_bytes":files[path].len(), "completed":true }}) } else { let range = &request["_runtime_invocation"]["range"]; @@ -2651,13 +2688,13 @@ mod tests { }}) }; let mut bytes = if path == "_elastos_object.json" { - self.files[path].clone() + files[path].clone() } else { let range = &request["_runtime_invocation"]["range"]; let start = range["start"].as_u64().unwrap() as usize; let end = range["end"].as_u64().unwrap() as usize; assert!(end >= start && end - start < 65536); - self.files[path][start..=end].to_vec() + files[path][start..=end].to_vec() }; let fault = *self.read_fault.lock().unwrap(); match (path, fault) { @@ -2693,7 +2730,7 @@ mod tests { .lock() .unwrap() .push(if path == "_elastos_object.json" { - self.files[path].len() + files[path].len() } else { let range = &request["_runtime_invocation"]["range"]; (range["end"].as_u64().unwrap() - range["start"].as_u64().unwrap() + 1) @@ -2745,19 +2782,30 @@ mod tests { } let directory = &request["directory"]; let root = Path::new(directory["root"].as_str().unwrap()); - assert_eq!( - directory["files"].as_array().unwrap().len(), - self.files.len() - ); - for file in directory["files"].as_array().unwrap() { - let path = file["path"].as_str().unwrap(); - let bytes = std::fs::read(root.join(path)).unwrap(); - assert_eq!(file["size"].as_u64().unwrap(), bytes.len() as u64); - assert_eq!(bytes, self.files[path]); - } - Ok( - serde_json::json!({"status":"ok","data":{"cid":self.cid.lock().unwrap().clone()}}), - ) + let listed = directory["files"].as_array().unwrap(); + let staged: std::collections::BTreeMap> = listed + .iter() + .map(|file| { + let path = file["path"].as_str().unwrap(); + let bytes = std::fs::read(root.join(path)).unwrap(); + assert_eq!(file["size"].as_u64().unwrap(), bytes.len() as u64); + (path.to_owned(), bytes) + }) + .collect(); + // Extra packages are matched by exact content; the primary + // package keeps its strict assertion for single-entry tests. + let cid = match self + .extra_packages + .iter() + .find(|(_, files)| **files == staged) + { + Some((cid, _)) => cid.clone(), + None => { + assert_eq!(staged, self.files); + self.cid.lock().unwrap().clone() + } + }; + Ok(serde_json::json!({"status":"ok","data":{"cid":cid}})) } _ => panic!("reconciliation/reuse dispatched unexpected operation {op}"), } @@ -2779,8 +2827,30 @@ mod tests { ) -> ( serde_json::Value, std::collections::BTreeMap>, + ) { + let payload = super::super::tests::model_catalog_fixture(); + let name = payload["entries"][0]["capsule_manifest"]["name"] + .as_str() + .unwrap() + .to_owned(); + let cid = payload["entries"][0]["cid"].as_str().unwrap().to_owned(); + package_fixture_named(weights, provenance, &name, &cid) + } + + /// One signed catalog entry for a distinct package identity, so a catalog + /// fixture can carry several models. + fn package_fixture_named( + weights: Vec, + provenance: &[u8], + name: &str, + cid: &str, + ) -> ( + serde_json::Value, + std::collections::BTreeMap>, ) { let mut payload = super::super::tests::model_catalog_fixture(); + payload["entries"][0]["cid"] = serde_json::json!(cid); + payload["entries"][0]["capsule_manifest"]["name"] = serde_json::json!(name); let mut files: std::collections::BTreeMap> = std::collections::BTreeMap::from([ ("LICENSE".into(), b"fixture license".to_vec()), @@ -2893,12 +2963,36 @@ mod tests { "shutdown_timeout_ms":5000, "enable_thinking":false }) ); - for platform in ["linux-arm64", "linux-amd64", "darwin-amd64", "*"] { - assert!(local_model_startup_profile(platform).is_err()); + // The Linux profile is CPU-only with a bounded thread budget; the + // remaining startup policy is shared by every proved host. + assert_eq!( + local_model_startup_profile("linux-amd64").unwrap(), + serde_json::json!({ + "context_size":4096, "parallel":1, "threads":4, "batch_threads":4, + "gpu_layers":0, "health_timeout_ms":120000, + "shutdown_timeout_ms":5000, "enable_thinking":false + }) + ); + for platform in [ + "linux-arm64", + "darwin-amd64", + "linux-x86_64", + "Linux-amd64", + "*", + "", + ] { + assert!( + local_model_startup_profile(platform).is_err(), + "{platform:?} has no proved host profile" + ); } } - #[cfg(all(target_os = "macos", target_arch = "aarch64"))] + // Startup binding runs wherever a Runtime-owned host profile exists. + #[cfg(any( + all(target_os = "macos", target_arch = "aarch64"), + all(target_os = "linux", target_arch = "x86_64") + ))] mod startup_binding { use super::*; use std::os::unix::fs::PermissionsExt as _; @@ -4858,7 +4952,7 @@ mod tests { assert_eq!(offer["adapter"]["kind"], "local_llama_cpp_text"); assert_eq!( offer["adapter"]["settings"], - local_model_startup_profile("darwin-arm64").unwrap() + local_model_startup_profile(&crate::setup::detect_platform()).unwrap() ); assert_eq!( offer["adapter"]["model"]["sha256"], @@ -5048,6 +5142,396 @@ mod tests { assert!(backend.calls.lock().unwrap().is_empty()); } + #[tokio::test] + async fn model_startup_permanent_repin_blocks_readiness_until_authorized_alias() { + let root = tempfile::tempdir().unwrap(); + let (payload, files) = package_fixture(b"GGUF\x03\0\0\0fixture".to_vec()); + write_preparation_catalog(root.path(), &payload); + let cid = payload["entries"][0]["cid"].as_str().unwrap().to_owned(); + let backend = Arc::new(PreparationBackend::new(files, cid.clone())); + let registry = Arc::new(elastos_runtime::provider::ProviderRegistry::new()); + registry + .register_sub_provider("ipfs", backend.clone()) + .await + .unwrap(); + register_content(®istry, root.path()).await; + let _engine = install_engine(root.path()); + let model = Arc::new(ModelActivationFixture::default()); + registry + .register_sub_provider("model", model.clone()) + .await + .unwrap(); + let input = serde_json::json!({"cid": cid}); + let cats = || { + backend + .calls + .lock() + .unwrap() + .iter() + .filter(|op| *op == "cat") + .count() + }; + + let owner = PreparationOwner::default(); + let first = owner + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("use")), + "permanent-first", + &input, + Arc::new(|| Ok(())), + ) + .unwrap(); + join_worker(&owner).await; + let record = + load_operation(root.path(), first["operation_id"].as_str().unwrap()).unwrap(); + assert_eq!(record.state, PreparationState::Admitted); + let ready = model_runtime_projection( + root.path(), + Some(®istry), + &context(), + &cid, + Some(record.operation_id.as_str()), + ) + .await; + assert_eq!(ready["dispatch_ready"], true, "{ready}"); + let reads = cats(); + + // Re-pin the same package under a permanent snapshot; keep the engine + // component that write_preparation_catalog's fresh manifest drops. + let manifest: serde_json::Value = serde_json::from_slice( + &std::fs::read(root.path().join("components.json")).unwrap(), + ) + .unwrap(); + let engine_component = manifest["external"]["llama-server"].clone(); + let mut permanent = payload.clone(); + permanent.as_object_mut().unwrap().remove("expires_at"); + write_preparation_catalog(root.path(), &permanent); + change_config(root.path(), |config| { + config["external"]["llama-server"] = engine_component.clone() + }); + let head_b = serde_json::from_slice::( + &std::fs::read(root.path().join("components.json")).unwrap(), + ) + .unwrap() + .model_catalog + .unwrap() + .head_cid; + assert_ne!(head_b, record.catalog_head_cid); + + let mut composed = config(root.path()); + let operator_only = composed.extra.clone(); + let _ = append_admitted_model_startup_offers(root.path(), ®istry, &mut composed) + .await + .unwrap(); + assert_eq!( + composed.extra["offers"], operator_only["offers"], + "an admission bound to the previous head is not composed" + ); + assert_eq!( + composed.extra["runtime_admitted_offers"], + serde_json::json!([]), + "an admission bound to the previous head is not composed" + ); + let blocked = model_runtime_projection( + root.path(), + Some(®istry), + &context(), + &cid, + Some(record.operation_id.as_str()), + ) + .await; + assert_eq!(blocked["admitted"], true, "{blocked}"); + assert_eq!(blocked["dispatch_ready"], false, "{blocked}"); + assert_eq!(cats(), reads, "re-pinning dispatches nothing"); + assert_eq!( + load_operation(root.path(), &record.operation_id) + .unwrap() + .catalog_head_cid, + record.catalog_head_cid + ); + + let reopened = PreparationOwner::default(); + let reused = reopened + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("use")), + "permanent-migrated", + &input, + Arc::new(|| Ok(())), + ) + .unwrap(); + join_worker(&reopened).await; + let alias = + load_operation(root.path(), reused["operation_id"].as_str().unwrap()).unwrap(); + assert_eq!(alias.admission_id, record.operation_id); + assert_eq!(alias.reserved_bytes, 0); + assert_eq!(alias.catalog_head_cid, head_b); + assert_eq!(alias.state, PreparationState::Admitted); + assert_eq!(cats(), reads, "the alias reuses the admitted bytes"); + let ready_again = model_runtime_projection( + root.path(), + Some(®istry), + &context(), + &cid, + Some(alias.operation_id.as_str()), + ) + .await; + assert_eq!(ready_again["dispatch_ready"], true, "{ready_again}"); + + let mut restarted = config(root.path()); + let _ = append_admitted_model_startup_offers(root.path(), ®istry, &mut restarted) + .await + .unwrap(); + assert_eq!(restarted.extra["offers"].as_array().unwrap().len(), 2); + assert_eq!( + restarted.extra["offers"][1]["adapter"]["model"]["sha256"], + format!("sha256:{:x}", Sha256::digest(b"GGUF\x03\0\0\0fixture")) + ); + let (_, guard) = crate::api::model_provider_config(root.path(), ®istry) + .await + .unwrap(); + drop(guard); + let after_restart = model_runtime_projection( + root.path(), + Some(®istry), + &context(), + &cid, + Some(alias.operation_id.as_str()), + ) + .await; + assert_eq!(after_restart["dispatch_ready"], true, "{after_restart}"); + assert_eq!(cats(), reads); + + // The alias is the record selected under head B; its saved + // descriptor lives on the admission owner. Equal reuse leaves the + // owner's descriptor and the inventory bytes unchanged. + let state_path = root.path().join("model-preparation/state.json"); + let owner_descriptor = |root: &Path| { + load_operation(root, &record.operation_id) + .unwrap() + .activation + .expect("owner keeps its saved descriptor") + }; + let saved = owner_descriptor(root.path()); + assert!(load_operation(root.path(), &alias.operation_id) + .unwrap() + .activation + .is_none()); + let inventory_bytes = std::fs::read(&state_path).unwrap(); + let mut reused_config = config(root.path()); + let _ = + append_admitted_model_startup_offers(root.path(), ®istry, &mut reused_config) + .await + .unwrap(); + assert_eq!(reused_config.extra, restarted.extra); + assert_eq!(std::fs::read(&state_path).unwrap(), inventory_bytes); + assert_eq!(owner_descriptor(root.path()), saved); + + // A structurally valid but different saved owner descriptor is a + // changed retry, never a silent replacement through the alias. + { + // Runtime writers refuse descriptor changes, so the changed + // descriptor arrives the only way it can: written from outside. + let mut durable: serde_json::Value = + serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); + let owner = durable["records"] + .as_array_mut() + .unwrap() + .iter_mut() + .find(|r| r["operation_id"] == record.operation_id) + .unwrap(); + owner["activation"]["offer"]["title"] = + serde_json::json!("renamed by another writer"); + std::fs::write(&state_path, serde_json::to_vec(&durable).unwrap()).unwrap(); + } + let changed = owner_descriptor(root.path()); + assert_ne!(changed, saved); + changed.check_root(root.path(), &record).unwrap(); + let changed_bytes = std::fs::read(&state_path).unwrap(); + let mut rejected = config(root.path()); + let before = serde_json::to_vec(&rejected).unwrap(); + let error = append_admitted_model_startup_offers(root.path(), ®istry, &mut rejected) + .await + .err() + .expect("a changed owner descriptor is rejected"); + assert!( + error + .to_string() + .contains("activation descriptor changed on retry"), + "{error}" + ); + assert_eq!(serde_json::to_vec(&rejected).unwrap(), before); + assert_eq!(std::fs::read(&state_path).unwrap(), changed_bytes); + assert_eq!(owner_descriptor(root.path()), changed); + } + + // Two signed catalog entries, acquired in either order, compose on the + // first startup pass: the pass after the second admission saves both + // descriptors and reports both offers ready without a retry, and a + // Runtime restart recomposes the identical config from those saved + // descriptors. + #[tokio::test] + async fn model_startup_composes_every_admitted_entry_on_first_attempt_in_both_orders() { + use elastos_runtime::provider::ProviderRegistry; + let second_cid = cid::Cid::new_v1( + 0x70, + cid::multihash::Multihash::<64>::wrap( + 0x12, + &Sha256::digest(b"second fixture package"), + ) + .unwrap(), + ) + .to_string(); + for order in [[0usize, 1], [1, 0]] { + let root = tempfile::tempdir().unwrap(); + let (mut payload, primary) = package_fixture(b"GGUF\x03\0\0\0fixture".to_vec()); + let (second, second_files) = package_fixture_named( + b"GGUF\x03\0\0\0second".to_vec(), + b"second provenance", + "second-model-fixture", + &second_cid, + ); + payload["entries"] + .as_array_mut() + .unwrap() + .push(second["entries"][0].clone()); + write_preparation_catalog(root.path(), &payload); + let cids: Vec = payload["entries"] + .as_array() + .unwrap() + .iter() + .map(|entry| entry["cid"].as_str().unwrap().to_owned()) + .collect(); + let mut backend = PreparationBackend::new(primary, cids[0].clone()); + backend.extra_packages.insert(cids[1].clone(), second_files); + let backend = Arc::new(backend); + let registry = Arc::new(ProviderRegistry::new()); + registry + .register_sub_provider("ipfs", backend.clone()) + .await + .unwrap(); + register_content(®istry, root.path()).await; + let _engine = install_engine(root.path()); + let model = Arc::new(ModelActivationFixture::default()); + registry + .register_sub_provider("model", model.clone()) + .await + .unwrap(); + + let mut admitted = Vec::new(); + for index in order { + let owner = PreparationOwner::default(); + let response = owner + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("use")), + &format!("use-{index}"), + &serde_json::json!({"cid": cids[index]}), + Arc::new(|| Ok(())), + ) + .unwrap(); + join_worker(&owner).await; + let record = + load_operation(root.path(), response["operation_id"].as_str().unwrap()) + .unwrap(); + assert_eq!(record.state, PreparationState::Admitted, "order {order:?}"); + admitted.push((index, record)); + } + let hashes_after_use = backend + .calls + .lock() + .unwrap() + .iter() + .filter(|op| *op == "runtime_hash_staged_directory") + .count(); + + // First attempt: both descriptors exist and both offers are ready. + let state = Inventory::open(root.path(), false).unwrap().load().unwrap(); + for (index, record) in &admitted { + let owner = state + .records + .iter() + .find(|r| r.operation_id == record.operation_id) + .unwrap(); + assert!( + owner.activation.is_some(), + "order {order:?}: entry {index} has no saved descriptor after its Use" + ); + let ready = model_runtime_projection( + root.path(), + Some(®istry), + &context(), + &cids[*index], + Some(record.operation_id.as_str()), + ) + .await; + assert_eq!( + ready["dispatch_ready"], true, + "order {order:?}: entry {index} is not ready on the first attempt: {ready}" + ); + } + + // Restart: one fresh composer pass admits both entries at once and a + // second pass reproduces it exactly from the saved descriptors. + let mut restarted = config(root.path()); + let worker = + append_admitted_model_startup_offers(root.path(), ®istry, &mut restarted) + .await + .unwrap_or_else(|err| { + panic!("order {order:?}: restart composition failed: {err}") + }); + drop(worker); + let offer_ids: Vec<_> = restarted.extra["runtime_admitted_offers"] + .as_array() + .unwrap() + .iter() + .map(|entry| entry["offer_id"].as_str().unwrap().to_owned()) + .collect(); + assert_eq!(offer_ids.len(), 2, "order {order:?}"); + assert_ne!(offer_ids[0], offer_ids[1]); + assert_eq!( + restarted.extra["offers"].as_array().unwrap().len(), + config(root.path()).extra["offers"] + .as_array() + .unwrap() + .len() + + 2 + ); + let mut again = config(root.path()); + let worker = + append_admitted_model_startup_offers(root.path(), ®istry, &mut again) + .await + .unwrap(); + drop(worker); + assert_eq!(again.extra, restarted.extra, "order {order:?}"); + assert_eq!( + Inventory::open(root.path(), false) + .unwrap() + .load() + .unwrap() + .records, + state.records, + "order {order:?}: restart keeps the saved descriptors" + ); + // Each restart pass verifies every admitted package identity once. + assert_eq!( + backend + .calls + .lock() + .unwrap() + .iter() + .filter(|op| *op == "runtime_hash_staged_directory") + .count(), + hashes_after_use + 4, + "order {order:?}" + ); + } + } + #[tokio::test] async fn model_startup_binding_rejects_tampering_and_unavailable_verifier_without_config_effects( ) { @@ -5477,10 +5961,15 @@ server.serve_forever() } impl PreparationBackend { + fn package_files(&self, cid: &str) -> &std::collections::BTreeMap> { + self.extra_packages.get(cid).unwrap_or(&self.files) + } + fn new(files: std::collections::BTreeMap>, cid: String) -> Self { Self { files, cid: Mutex::new(cid), + extra_packages: std::collections::BTreeMap::new(), calls: Mutex::new(vec![]), capacity_requests: Mutex::new(vec![]), capacity_volume: 7, @@ -5492,6 +5981,7 @@ server.serve_forever() cold_backend: AtomicBool::new(false), hold_read: AtomicBool::new(false), hold_hash: AtomicBool::new(false), + missing_cache: AtomicBool::new(false), read_fault: Mutex::new(None), native: None, entered: tokio::sync::Notify::new(), @@ -6137,7 +6627,7 @@ server.serve_forever() use std::process::{Child, Command, Stdio}; use std::time::{Duration, Instant}; - include!("preparation/qwen_process_proof.rs"); + include!("preparation/pinned_model_process_proof.rs"); struct OwnedChild(Child); impl Drop for OwnedChild { @@ -6363,10 +6853,10 @@ server.serve_forever() } async fn preparation_process(cold: bool) { - preparation_process_with_qwen(cold, None).await; + preparation_process_with_model(cold, None).await; } - async fn preparation_process_with_qwen(cold: bool, qwen: Option) { + async fn preparation_process_with_model(cold: bool, pinned: Option) { use elastos_runtime::provider::{ BridgeProviderConfig, CapsuleProvider, ProviderBridge, }; @@ -6392,8 +6882,8 @@ server.serve_forever() let setup_started = Instant::now(); // Fixed synthetic sizes only. This whole-buffer fixture is not a // large-model publisher or a product-memory measurement. - let weights_size = if qwen.is_some() { - QWEN_BYTES as usize + let weights_size = if let Some(pinned) = &pinned { + pinned.model.bytes as usize } else if cold { 64 * 1024 * 1024 - 65536 } else { @@ -6401,7 +6891,7 @@ server.serve_forever() }; let package_bound = weights_size as u64 + 65536; let layout_charge = preparation_charge(package_bound).unwrap() - + (if qwen.is_some() { 2 } else { 3 }) * package_bound + + (if pinned.is_some() { 2 } else { 3 }) * package_bound + 32 * 1024 * 1024; let dir = File::open(&root_path).unwrap(); let (volume_capacity, initial_free) = volume_bytes(&dir); @@ -6419,8 +6909,8 @@ server.serve_forever() // The native provider resolves the explicit fixture tool locally; // only this isolated directory contains the test link. std::os::unix::fs::symlink(&kubo, data.join("bin/kubo")).unwrap(); - let deadline = - Instant::now() + Duration::from_secs(if qwen.is_some() { 3500 } else { 90 }); + let deadline = Instant::now() + + Duration::from_secs(pinned.as_ref().map_or(90, |p| p.model.deadline_secs)); assert_eq!( run( command( @@ -6528,8 +7018,8 @@ server.serve_forever() } else { None }; - let (mut payload, files) = if let Some(qwen) = &qwen { - qwen.package() + let (mut payload, files) = if let Some(pinned) = &pinned { + pinned.package() } else { let mut weights = vec![0; weights_size]; let mut random = 0x7b16_984d_3c20_a5e1u64; @@ -6551,14 +7041,14 @@ server.serve_forever() args.extend(files.keys().map(String::as_str)); // Import the borrowed real weights through stdin, never a seed copy, // symlink or whole-file JSON/base64 request. Metadata remains small. - let cid = if let Some(qwen) = &qwen { + let cid = if let Some(pinned) = &pinned { import_streamed_package( &kubo, &root_path, seed_repo, &seed, &args, - &qwen.weights, + &pinned.weights, deadline, ) .await @@ -6573,8 +7063,8 @@ server.serve_forever() assert!(canonical_cid(&cid, 0x70)); payload["entries"][0]["cid"] = serde_json::json!(cid); write_preparation_catalog(&data, &payload); - let engine_cleanup = if let Some(qwen) = &qwen { - Some(qwen.install_engine(&data, &root_path, deadline).await) + let engine_cleanup = if let Some(pinned) = &pinned { + Some(pinned.install_engine(&data, &root_path, deadline).await) } else { None }; @@ -6665,7 +7155,7 @@ server.serve_forever() require_peer(&kubo, &root_path, &publisher_repo, &peer_ids[0], deadline).await; } let setup_ms = setup_started.elapsed().as_millis(); - let publisher_before = if cold && qwen.is_none() { + let publisher_before = if cold && pinned.is_none() { disk_bytes(&publisher_repo) } else { (0, 0) @@ -6704,8 +7194,8 @@ server.serve_forever() let test_backend = backend.clone(); let test_volume = dir.try_clone().unwrap(); let test_publisher_repo = publisher_repo.clone(); - let real_qwen = qwen.is_some(); - let qwen_provider = qwen.as_ref().map(|q| q.provider.clone()); + let real_model = pinned.is_some(); + let pinned_model = pinned.as_ref().map(|p| (p.model, p.provider.clone())); let model_children = Arc::new(Mutex::new(Vec::new())); let test_model_children = model_children.clone(); let mut logs = vec![stdout.try_clone().unwrap(), stderr.try_clone().unwrap()]; @@ -6743,19 +7233,19 @@ server.serve_forever() .unwrap() .is_finished() { - // A Qwen backend has tens of thousands of blocks. Sample + // A real model backend has thousands of blocks. Sample // volume space and owned-process RSS, not recursive trees. - let stage = if real_qwen { + let stage = if real_model { (0, 0) } else { disk_bytes(&test_data.join("model-preparation/stage")) }; - let backend = if real_qwen { + let backend = if real_model { (0, 0) } else { disk_bytes(&test_repo) }; - let publisher = if cold && !real_qwen { + let publisher = if cold && !real_model { disk_bytes(&test_publisher_repo) } else { (0, 0) @@ -6778,10 +7268,10 @@ server.serve_forever() max_sample_gap_ms = max_sample_gap_ms.max(last_sample.elapsed().as_millis()); last_sample = Instant::now(); samples += 1; - if real_qwen { + if real_model { rss_peak_kib = rss_peak_kib.max(proof_rss_kib()); } - tokio::time::sleep(Duration::from_millis(if real_qwen { 1000 } else { 10 })) + tokio::time::sleep(Duration::from_millis(if real_model { 1000 } else { 10 })) .await; } join_worker(&test_owner).await; @@ -6807,15 +7297,15 @@ server.serve_forever() assert_eq!(fs::read(admitted.path.join(path)).unwrap(), *bytes); } let admitted_disk = disk_bytes(&admitted.path); - if real_qwen { - assert_qwen_file(&admitted.path.join("weights.gguf")); + if let Some((model, _)) = &pinned_model { + model.assert_weights(&admitted.path.join("weights.gguf")); } - let backend_after = if real_qwen { + let backend_after = if real_model { (0, 0) } else { disk_bytes(&test_repo) }; - let publisher_after = if cold && !real_qwen { + let publisher_after = if cold && !real_model { disk_bytes(&test_publisher_repo) } else { (0, 0) @@ -6828,7 +7318,7 @@ server.serve_forever() publisher_peak.0.max(publisher_after.0), publisher_peak.1.max(publisher_after.1), ); - if cold && !real_qwen { + if cold && !real_model { assert!( backend_after.0 >= backend_before.0 + weights_size as u64, "cold transfer must grow consumer backend logical bytes" @@ -6844,12 +7334,11 @@ server.serve_forever() storage::require_space_floor(volume_capacity, minimum_free, 0).unwrap(); let requests = test_backend.calls.lock().unwrap().clone(); let reads = requests.iter().filter(|op| *op == "cat").count(); - let expected_reads = - 1 + if real_qwen { - QWEN_BYTES.div_ceil(65536) as usize - } else { - 0 - } + test_backend + let expected_reads = 1 + + pinned_model + .as_ref() + .map_or(0, |(model, _)| model.bytes.div_ceil(65536) as usize) + + test_backend .files .iter() .filter(|(path, _)| *path != "_elastos_object.json") @@ -6858,7 +7347,7 @@ server.serve_forever() assert_eq!(reads, expected_reads); // The real engine enables activation composition after admission; // both independently verify the exact stored package CID. - let admission_and_activation_hashes = if real_qwen { 2 } else { 1 }; + let admission_and_activation_hashes = if real_model { 2 } else { 1 }; assert_eq!( requests .iter() @@ -6867,7 +7356,7 @@ server.serve_forever() admission_and_activation_hashes ); let reopened = PreparationOwner::default(); - if real_qwen { + if real_model { retention_intent(&test_data, &context(), &cid, true).unwrap(); } let reuse_started = Instant::now(); @@ -6881,7 +7370,7 @@ server.serve_forever() Arc::new(|| Ok(())), ) .unwrap(); - if real_qwen { + if real_model { // Reuse rehashes the full closure twice; use the existing // whole-proof deadline, not the tiny-fixture five-second join. let worker = reopened.worker.lock().unwrap().take().unwrap(); @@ -6920,11 +7409,12 @@ server.serve_forever() 2 * admission_and_activation_hashes ); let reuse_ms = reuse_started.elapsed().as_millis(); - let inference = if let Some(binary) = qwen_provider { - qwen_reply_and_restart( + let inference = if let Some((model, binary)) = &pinned_model { + pinned_reply_and_restart( + model, &test_data, &test_registry, - &binary, + binary, &cid, deadline, &test_model_children, @@ -6951,7 +7441,7 @@ server.serve_forever() .iter() .filter(|op| *op == "runtime_hash_staged_directory") .count(), - 2 * admission_and_activation_hashes + if real_qwen { 2 } else { 0 } + 2 * admission_and_activation_hashes + if real_model { 2 } else { 0 } ); assert!( logs.iter() @@ -6961,7 +7451,7 @@ server.serve_forever() serde_json::json!({"cold":cold,"setup_ms":setup_ms,"elapsed_preparation_ms":elapsed_ms, "reuse_ms":reuse_ms,"content_reads":reads, "layout_preflight_bytes":layout_charge,"minimum_sampled_free_bytes":minimum_free, - "sample_interval_ms":if real_qwen {1000} else {10},"samples":samples,"max_sample_gap_ms":max_sample_gap_ms, + "sample_interval_ms":if real_model {1000} else {10},"samples":samples,"max_sample_gap_ms":max_sample_gap_ms, "test_process_sampled_rss_peak_kib":rss_peak_kib,"inference":inference, "publisher_before":publisher_before,"publisher_sampled_peak":publisher_peak, "publisher_after":publisher_after, @@ -6970,7 +7460,7 @@ server.serve_forever() "index_bytes":record.index_bytes,"reserved_bytes":record.reserved_bytes, "seed_disk":seed_disk,"backend_before":backend_before,"backend_sampled_peak":backend_peak, "staging_sampled_peak":staged_peak,"admitted_disk":admitted_disk,"backend_after":backend_after, - "reuse_content_reads":0,"inference_executed":real_qwen}) + "reuse_content_reads":0,"inference_executed":real_model}) }); let outcome = tokio::time::timeout_at(tokio::time::Instant::from_std(deadline), &mut work).await; @@ -7059,7 +7549,7 @@ server.serve_forever() receipt["kubo_children_reaped"] = serde_json::json!(if cold { 2 } else { 1 }); receipt["consumer_package_absent_before_connect"] = serde_json::json!(cold); receipt["consumer_init_block"] = serde_json::json!(consumer_baseline); - if qwen.is_some() { + if let Some(pinned) = &pinned { // Volume samples replace recursive large-backend scans. Do not // represent the skipped measurements as zero allocations. for key in [ @@ -7077,17 +7567,19 @@ server.serve_forever() receipt["rss_scope"] = serde_json::json!( "sampled test-process high-water RSS only; full process-tree peak remains open" ); - receipt["source_weights_sha256"] = serde_json::json!(QWEN_SHA); + receipt["source_weights_sha256"] = serde_json::json!(pinned.model.sha); + receipt["pinned_model"] = serde_json::json!(pinned.model.capsule_name); + receipt["declared_quantization"] = serde_json::json!(pinned.model.quantization); receipt["publisher_authority"] = serde_json::json!( "isolated test signing key; local operator attestation, not upstream signature" ); } - receipt["proof_limit"] = serde_json::json!(if qwen.is_some() { - "Exact Qwen, isolated publisher-attested catalog, cold loopback Content admission and native model reply/restart. Volume/RSS are samples, not continuous peaks; GUI, installed acceptance and eviction remain open." + receipt["proof_limit"] = serde_json::json!(if pinned.is_some() { + "Exact pinned model, isolated publisher-attested catalog, cold loopback Content admission and native model reply/restart. Volume/RSS are samples, not continuous peaks; GUI, installed acceptance and eviction remain open." } else if cold { - "64 MiB-capped signed synthetic cold loopback delivery through Content/native Use/admission/reuse. Allocations are sampled peaks; whole-buffer fixture memory is harness-only. Public-network, exact Qwen and inference remain open." + "64 MiB-capped signed synthetic cold loopback delivery through Content/native Use/admission/reuse. Allocations are sampled peaks; whole-buffer fixture memory is harness-only. Public-network, exact pinned models and inference remain open." } else { - "8 MiB signed synthetic seeded offline Kubo proof through Content/native Use/admission/reuse. Allocations are sampled peaks; whole-buffer fixture memory is harness-only. Cold-network, exact Qwen and inference remain open." + "8 MiB signed synthetic seeded offline Kubo proof through Content/native Use/admission/reuse. Allocations are sampled peaks; whole-buffer fixture memory is harness-only. Cold-network, exact pinned models and inference remain open." }); println!("{receipt}"); } @@ -7384,6 +7876,282 @@ server.serve_forever() ); } + #[tokio::test] + async fn model_head_migration_reuses_admitted_bytes_without_refetch_or_implicit_dispatch() { + let (root, record, backend, registry) = staged_fixture(now().unwrap(), true).await; + finish_retention_fixture(root.path(), &record, registry.clone()).await; + let admitted = load_operation(root.path(), &record.operation_id).unwrap(); + let cats = || { + backend + .calls + .lock() + .unwrap() + .iter() + .filter(|op| *op == "cat") + .count() + }; + let baseline_cats = cats(); + let baseline_calls = backend.calls.lock().unwrap().len(); + + let (mut permanent, _) = package_fixture(b"GGUF\x03\0\0\0fixture".to_vec()); + permanent.as_object_mut().unwrap().remove("expires_at"); + write_preparation_catalog(root.path(), &permanent); + let manifest: crate::setup::ComponentsManifest = + serde_json::from_slice(&std::fs::read(root.path().join("components.json")).unwrap()) + .unwrap(); + let head_b = manifest.model_catalog.unwrap().head_cid; + assert_ne!(head_b, admitted.catalog_head_cid); + assert!( + current_entry(root.path(), &admitted).is_err(), + "the original admission stays bound to the previous head" + ); + assert_eq!( + load_operation(root.path(), &record.operation_id).unwrap(), + admitted + ); + assert_eq!(backend.calls.lock().unwrap().len(), baseline_calls); + + let owner = PreparationOwner::default(); + let reply = owner + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("use")), + "migrated-use", + &serde_json::json!({"cid": record.package_cid}), + Arc::new(|| Ok(())), + ) + .unwrap(); + join_worker(&owner).await; + let alias = load_operation(root.path(), reply["operation_id"].as_str().unwrap()).unwrap(); + assert_ne!(alias.operation_id, record.operation_id); + assert_eq!(alias.admission_id, record.operation_id); + assert_eq!(alias.reserved_bytes, 0); + assert_eq!(alias.catalog_head_cid, head_b); + assert_eq!(alias.state, PreparationState::Admitted); + assert_eq!(cats(), baseline_cats, "migration reuses the admitted bytes"); + let original = load_operation(root.path(), &record.operation_id).unwrap(); + assert_eq!(original.state, PreparationState::Admitted); + assert_eq!(original.catalog_head_cid, admitted.catalog_head_cid); + assert_eq!(original.admission_id, admitted.admission_id); + assert_eq!(original.request_binding, admitted.request_binding); + assert_eq!( + ( + original.total_bytes, + original.completed_bytes, + original.index_bytes + ), + ( + admitted.total_bytes, + admitted.completed_bytes, + admitted.index_bytes + ) + ); + assert!(current_entry(root.path(), &alias).is_ok()); + + let reopened = Inventory::open(root.path(), false).unwrap().load().unwrap(); + assert_eq!(reopened.records.len(), 2); + assert_eq!( + reopened + .records + .iter() + .find(|r| r.operation_id == alias.operation_id) + .unwrap() + .state, + PreparationState::Admitted + ); + let projection = + model_runtime_projection(root.path(), None, &context(), &record.package_cid, None) + .await; + assert_eq!(projection["admitted"], true); + } + + #[tokio::test] + async fn model_preparation_cancels_and_fails_over_within_deadline_against_silent_holder() { + use crate::carrier::tests as carrier_fixture; + let root = tempfile::tempdir().unwrap(); + let (payload, files) = package_fixture(b"GGUF\x03\0\0\0fixture".to_vec()); + write_preparation_catalog(root.path(), &payload); + let cid = payload["entries"][0]["cid"].as_str().unwrap().to_owned(); + // The consumer's own backend holds nothing; every read must come over Carrier. + let backend = Arc::new(PreparationBackend::new(files.clone(), cid.clone())); + backend.missing_cache.store(true, Ordering::Release); + let registry = Arc::new(elastos_runtime::provider::ProviderRegistry::new()); + registry + .register_sub_provider("ipfs", backend.clone()) + .await + .unwrap(); + register_content(®istry, root.path()).await; + let (consumer_sk, consumer_did) = elastos_identity::derive_did(&[90u8; 32]); + let consumer = crate::carrier::start_isolated_carrier_node_with_registry( + &consumer_sk, + &consumer_did, + root.path().join("carrier"), + Some(Arc::downgrade(®istry)), + ) + .await + .unwrap(); + registry + .set_carrier_invoker(Arc::new( + crate::carrier::CarrierProviderInvoker::with_carrier_endpoint_and_registry( + consumer.endpoint.clone(), + Arc::downgrade(®istry), + ), + )) + .await; + registry + .register(Arc::new( + crate::carrier::CarrierAvailabilityProvider::with_provider_registry( + consumer.gossip_state.clone(), + Arc::downgrade(®istry), + ), + )) + .await; + let silent_requests = Arc::new(std::sync::Mutex::new(Vec::new())); + let silent = carrier_fixture::start_holder_runtime( + 91, + Arc::new(carrier_fixture::SilentContentProvider { + requests: silent_requests.clone(), + }), + None, + ) + .await; + let holder_requests = Arc::new(std::sync::Mutex::new(Vec::new())); + let holder = carrier_fixture::start_content_holder_runtime( + 92, + files.clone(), + holder_requests.clone(), + ) + .await; + consumer + .memory_lookup + .add_endpoint_info(holder.addr.clone()); + let input = serde_json::json!({"cid": cid}); + let join_within = |owner: &PreparationOwner, secs: u64| { + let task = owner + .worker + .lock() + .unwrap() + .take() + .expect("worker was started"); + async move { + tokio::time::timeout(std::time::Duration::from_secs(secs), task) + .await + .expect("worker settles within its bound") + .unwrap(); + } + }; + + // Only a connected silent holder is announced. The operator cancels while + // the index read is waiting on it; the bounded read's answer budget + // returns control and the preparation settles as Cancelled instead of + // hanging. With more holders announced, settlement waits for the read + // to exhaust them, one answer budget each. + let now = now().unwrap(); + carrier_fixture::seed_content_availability_announcements( + &consumer, + &cid, + &[(silent.ticket.clone(), [91u8; 32], silent.did.clone(), now)], + ) + .await; + let owner = PreparationOwner::default(); + let started = std::time::Instant::now(); + let reply = owner + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("use")), + "silent-cancel", + &input, + Arc::new(|| Ok(())), + ) + .unwrap(); + let operation_id = reply["operation_id"].as_str().unwrap().to_owned(); + for _ in 0..100 { + if !silent_requests.lock().unwrap().is_empty() { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + assert_eq!( + silent_requests.lock().unwrap().len(), + 1, + "the read is waiting on the connected silent holder" + ); + let cancelled = owner + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("cancel")), + "silent-cancel-request", + &serde_json::json!({"operation_id": operation_id}), + Arc::new(|| Ok(())), + ) + .unwrap(); + assert_eq!(cancelled["cancel_requested"], true); + join_within(&owner, 45).await; + let settled = started.elapsed(); + let record = load_operation(root.path(), &operation_id).unwrap(); + assert_eq!(record.state, PreparationState::Cancelled, "{record:?}"); + assert_eq!(record.reserved_bytes, 0); + assert!( + settled < std::time::Duration::from_secs(35), + "cancellation settles within the 30s Carrier availability wait: {settled:?}" + ); + assert!(holder_requests.lock().unwrap().is_empty()); + assert!(!root.path().join("model-preparation/stage").exists()); + + // The silent holder announces newest, but its recorded failure already + // ranks it behind the real holder, so admission pays no further deadline. + carrier_fixture::seed_content_availability_announcements( + &consumer, + &cid, + &[ + ( + holder.ticket.clone(), + [92u8; 32], + holder.did.clone(), + now - 20, + ), + (silent.ticket.clone(), [91u8; 32], silent.did.clone(), now), + ], + ) + .await; + let owner = PreparationOwner::default(); + let started = std::time::Instant::now(); + let reply = owner + .invoke( + root.path(), + Some(registry.clone()), + caller(&context(), &method("use")), + "silent-then-real", + &input, + Arc::new(|| Ok(())), + ) + .unwrap(); + join_within(&owner, 60).await; + let admitted = started.elapsed(); + let record = load_operation(root.path(), reply["operation_id"].as_str().unwrap()).unwrap(); + assert_eq!(record.state, PreparationState::Admitted, "{record:?}"); + assert_eq!(record.completed_bytes, record.total_bytes); + let served = holder_requests.lock().unwrap().clone(); + assert!(!served.is_empty()); + assert!(served.iter().all(|request| request["bounded_read"] == true)); + assert!( + admitted < std::time::Duration::from_secs(30), + "the demoted silent holder costs no further answer budget here: {admitted:?}" + ); + eprintln!( + "preparation: cancelled against a silent holder in {settled:?}; admitted after failover in {admitted:?} with {} bounded holder reads and {} silent attempts", + served.len(), + silent_requests.lock().unwrap().len() + ); + + carrier_fixture::shutdown_test_carrier_node(silent.node).await; + carrier_fixture::shutdown_test_carrier_node(holder.node).await; + carrier_fixture::shutdown_test_carrier_node(consumer).await; + } + #[test] fn model_retention_pending_cancel_expire_and_fresh_retry_clear_intent() { for terminal in ["cancel", "expire", "capacity-cancel", "capacity-expire"] { diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/qwen_process_proof.rs b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/pinned_model_process_proof.rs similarity index 72% rename from elastos/crates/elastos-server/src/api/capsule_inventory/preparation/qwen_process_proof.rs rename to elastos/crates/elastos-server/src/api/capsule_inventory/preparation/pinned_model_process_proof.rs index 287c19e35..07c348dd3 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/qwen_process_proof.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/pinned_model_process_proof.rs @@ -1,9 +1,91 @@ -// Opt-in proof only. The caller supplies read-only operator artifacts; the +// Opt-in proofs only. The caller supplies read-only operator artifacts; the // existing disposable process proof owns every import, journal and child. -const QWEN_BYTES: u64 = 6_169_341_984; -const QWEN_SHA: &str = "d784ce9eda1a5a7b51e8f705a9e6310844bf4f173654d115823c775fdea56d43"; -const QWEN_LICENSE_SHA: &str = "bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a"; -const QWEN_ENGINE_SHA: &str = "d0878274b8d6bd3c8ea26a78eb66cd1ffd943d007c62b9dff31c8aa99922d713"; +// +// Model bytes are platform-independent, so one pinned model runs on every +// proved host profile. The engine executable has its own identity per +// platform: the same llama.cpp release ships a different bundle per host. +struct PinnedModel { + /// `ELASTOS_TEST__PATH`, `_LICENSE_PATH` and `_ENGINE_DATA` name + /// the caller-owned read-only inputs. + env_prefix: &'static str, + capsule_name: &'static str, + bytes: u64, + sha: &'static str, + license_bytes: u64, + license_sha: &'static str, + quantization: &'static str, + minimum_memory_mb: u64, + base_repository: &'static str, + base_revision: &'static str, + quantized_repository: &'static str, + quantized_revision: &'static str, + provenance: &'static str, + /// Whole-proof budget: import, loopback transfer, admission, reply/restart. + deadline_secs: u64, + /// Accepts the completed text of the first reply. + reply_accepts: fn(&str) -> bool, + /// Keeps this model generating long enough for an active cancellation + /// to be observed while text deltas stream. + long_prompt: &'static str, +} + +const QWEN: PinnedModel = PinnedModel { + env_prefix: "QWEN", + capsule_name: "qwen-isolated-proof", + bytes: 6_169_341_984, + sha: "d784ce9eda1a5a7b51e8f705a9e6310844bf4f173654d115823c775fdea56d43", + license_bytes: 11544, + license_sha: "bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a", + quantization: "Q4_K_M", + minimum_memory_mb: 8192, + base_repository: "Qwen/Qwen3.5-9B", + base_revision: "c202236235762e1c871ad0ccb60c8ee5ba337b9a", + quantized_repository: "bartowski/Qwen_Qwen3.5-9B-GGUF", + quantized_revision: "bc658f7a1ae222ee901a565aafd2f5b8c839e176", + provenance: "Isolated operator/test publisher attestation, not an upstream signature. Weights match the pinned bartowski quantization. Its README names Qwen/Qwen3.5-9B and llama.cpp b9222 but does not attest a base conversion revision or include a standalone license. The base_revision is an independently checked upstream reference with Apache-2.0 license, not an asserted conversion input. LICENSE and LICENSE.base reproduce that checked license for this isolated proof.\n", + deadline_secs: 3500, + // A 9B instruct model follows the one-word instruction exactly. + reply_accepts: |text| { + text.trim() + .trim_end_matches(['.', '!']) + .eq_ignore_ascii_case("ready") + }, + long_prompt: "Write a continuous numbered list of detailed, distinct sentences about ordinary garden plants. Produce at least 12000 characters of text. Continue the list without a summary or closing remarks.", +}; + +const SMOLLM2: PinnedModel = PinnedModel { + env_prefix: "SMOLLM2", + capsule_name: "smollm2-isolated-proof", + bytes: 144_811_072, + sha: "c4a3dd037301b6ecea31d6da37f5cd793ead920dd5ddfe6d589294628d6ce66a", + // Canonical Apache License 2.0 text (apache.org/licenses/LICENSE-2.0.txt). + license_bytes: 11358, + license_sha: "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30", + quantization: "Q8_0", + minimum_memory_mb: 1024, + base_repository: "HuggingFaceTB/SmolLM2-135M-Instruct", + base_revision: "12fd25f77366fa6b3b4b768ec3050bf629380bac", + quantized_repository: "unsloth/SmolLM2-135M-Instruct-GGUF", + quantized_revision: "9e6855bc4be717fca1ef21360a1db4b29d5c559a", + provenance: "Isolated operator/test publisher attestation, not an upstream signature. Weights match the pinned unsloth Q8_0 quantization, whose README names HuggingFaceTB/SmolLM2-135M-Instruct as base_model under license apache-2.0. Neither repository ships a standalone LICENSE file; LICENSE and LICENSE.base reproduce the canonical Apache License 2.0 text for this isolated proof. The base_revision is the independently checked upstream head, not an asserted conversion input.\n", + deadline_secs: 900, + // A 135M model answers freely under the provider's default sampling; the + // proof binds a completed, non-empty streamed reply, not its wording. + reply_accepts: |text| !text.trim().is_empty(), + // A small model ignores length instructions but keeps narrating a story + // for well over a thousand tokens. + long_prompt: "Write a very long story about a lighthouse keeper. Keep writing new chapters and never stop.", +}; + +/// SHA-256 of the verified `llama-server` executable inside the pinned +/// b10516 bundle for each proved host profile. +fn pinned_engine_sha(platform: &str) -> &'static str { + match platform { + "darwin-arm64" => "d0878274b8d6bd3c8ea26a78eb66cd1ffd943d007c62b9dff31c8aa99922d713", + "linux-amd64" => "fa24fc90877d1edc68990af5f4f8e476256959357d7f58cf59910e5657f7403f", + other => panic!("no pinned b10516 engine identity for {other}"), + } +} const PACKAGE_ADD_ARGS: &[&str] = &[ "add", @@ -262,7 +344,8 @@ async fn model_preparation_streamed_bootstrap_matches_directory_cid() { ); } -struct QwenProof { +struct PinnedModelProof { + model: &'static PinnedModel, weights: File, license: Vec, engine_data: PathBuf, @@ -321,30 +404,42 @@ fn streamed_sha(file: &mut File) -> String { hex::encode(digest.finalize()) } -fn assert_qwen_file(path: &Path) { - let mut file = proof_file(path, QWEN_BYTES); - assert_eq!(file.metadata().unwrap().len(), QWEN_BYTES); - assert_eq!(streamed_sha(&mut file), QWEN_SHA); +impl PinnedModel { + fn assert_weights(&self, path: &Path) { + let mut file = proof_file(path, self.bytes); + assert_eq!(file.metadata().unwrap().len(), self.bytes); + assert_eq!(streamed_sha(&mut file), self.sha); + } + + fn env_path(&self, suffix: &str) -> PathBuf { + let name = format!("ELASTOS_TEST_{}_{suffix}", self.env_prefix); + PathBuf::from( + std::env::var_os(&name) + .unwrap_or_else(|| panic!("explicit {name} proof prerequisite")), + ) + } } -impl QwenProof { - fn from_env() -> Self { - let path = - |name| PathBuf::from(std::env::var_os(name).expect("explicit Qwen proof prerequisite")); - let mut weights = proof_file(&path("ELASTOS_TEST_QWEN_PATH"), QWEN_BYTES); - assert_eq!(weights.metadata().unwrap().len(), QWEN_BYTES); - assert_eq!(streamed_sha(&mut weights), QWEN_SHA); +impl PinnedModelProof { + fn from_env(model: &'static PinnedModel) -> Self { + let mut weights = proof_file(&model.env_path("PATH"), model.bytes); + assert_eq!(weights.metadata().unwrap().len(), model.bytes); + assert_eq!(streamed_sha(&mut weights), model.sha); let mut license = Vec::new(); - proof_file(&path("ELASTOS_TEST_QWEN_LICENSE_PATH"), 11544) + proof_file(&model.env_path("LICENSE_PATH"), model.license_bytes) .read_to_end(&mut license) .unwrap(); - assert_eq!(license.len(), 11544); - assert_eq!(hex::encode(Sha256::digest(&license)), QWEN_LICENSE_SHA); + assert_eq!(license.len() as u64, model.license_bytes); + assert_eq!(hex::encode(Sha256::digest(&license)), model.license_sha); Self { + model, weights, license, - engine_data: path("ELASTOS_TEST_QWEN_ENGINE_DATA"), - provider: path("ELASTOS_TEST_MODEL_PROVIDER_PATH"), + engine_data: model.env_path("ENGINE_DATA"), + provider: PathBuf::from( + std::env::var_os("ELASTOS_TEST_MODEL_PROVIDER_PATH") + .expect("explicit ELASTOS_TEST_MODEL_PROVIDER_PATH proof prerequisite"), + ), } } @@ -354,18 +449,20 @@ impl QwenProof { serde_json::Value, std::collections::BTreeMap>, ) { + let model = self.model; let mut payload = super::super::super::tests::model_catalog_fixture(); let capsule = &mut payload["entries"][0]["capsule_manifest"]; - capsule["name"] = serde_json::json!("qwen-isolated-proof"); + capsule["name"] = serde_json::json!(model.capsule_name); + capsule["model_content"]["quantization"] = serde_json::json!(model.quantization); + capsule["model_content"]["minimum_memory_mb"] = serde_json::json!(model.minimum_memory_mb); let provenance = &mut capsule["model_content"]["provenance"]; - provenance["base_repository"] = serde_json::json!("Qwen/Qwen3.5-9B"); - provenance["base_revision"] = serde_json::json!("c202236235762e1c871ad0ccb60c8ee5ba337b9a"); - provenance["quantized_repository"] = serde_json::json!("bartowski/Qwen_Qwen3.5-9B-GGUF"); - provenance["quantized_revision"] = - serde_json::json!("bc658f7a1ae222ee901a565aafd2f5b8c839e176"); + provenance["base_repository"] = serde_json::json!(model.base_repository); + provenance["base_revision"] = serde_json::json!(model.base_revision); + provenance["quantized_repository"] = serde_json::json!(model.quantized_repository); + provenance["quantized_revision"] = serde_json::json!(model.quantized_revision); let mut files = std::collections::BTreeMap::from([ ("LICENSE".into(), self.license.clone()), ("LICENSE.base".into(), self.license.clone()), - ("PROVENANCE.md".into(), b"Isolated operator/test publisher attestation, not an upstream signature. Weights match the pinned bartowski quantization. Its README names Qwen/Qwen3.5-9B and llama.cpp b9222 but does not attest a base conversion revision or include a standalone license. The base_revision is an independently checked upstream reference with Apache-2.0 license, not an asserted conversion input. LICENSE and LICENSE.base reproduce that checked license for this isolated proof.\n".to_vec()), + ("PROVENANCE.md".into(), model.provenance.as_bytes().to_vec()), ("capsule.json".into(), serde_json::to_vec(capsule).unwrap()), ]); let object = &mut payload["entries"][0]["object_manifest"]; @@ -373,7 +470,7 @@ impl QwenProof { for file in object["files"].as_array_mut().unwrap() { let path = file["path"].as_str().unwrap().to_owned(); let (size, sha) = if path == "weights.gguf" { - (QWEN_BYTES, QWEN_SHA.to_owned()) + (model.bytes, model.sha.to_owned()) } else { ( files[&path].len() as u64, @@ -413,7 +510,7 @@ impl QwenProof { crate::setup::verified_local_model_engine(&self.engine_data, &manifest).unwrap(); assert_eq!( identity.sha256.trim_start_matches("sha256:"), - QWEN_ENGINE_SHA + pinned_engine_sha(&crate::setup::detect_platform()) ); let source = identity.path.parent().unwrap(); let receipt: serde_json::Value = @@ -422,6 +519,8 @@ impl QwenProof { assert_eq!(receipt["version"], "b10516"); // This pinned bundle has a root executable. Bound copies from its // verified receipt, including directories and relative library links. + // The Linux bundle carries one CPU backend library per + // microarchitecture, so its bound is wider than the Metal bundle. let entries = receipt["entries"].as_array().unwrap(); assert!(entries.len() <= 128); let total = entries @@ -433,7 +532,7 @@ impl QwenProof { .len() }) .sum::(); - assert!(total + 256 * 1024 <= 28 * 1024 * 1024); + assert!(total + 256 * 1024 <= 96 * 1024 * 1024); let bundle = data.join("proof-engine"); let mut copy = Command::new("/bin/cp"); copy.args(["-pR"]) @@ -482,22 +581,36 @@ fn proof_rss_kib() -> u64 { unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) }, 0 ); - // macOS reports bytes; this opt-in profile is Darwin-arm64 only. - (unsafe { usage.assume_init() }.ru_maxrss as u64) / 1024 + let maxrss = unsafe { usage.assume_init() }.ru_maxrss as u64; + // macOS reports bytes; Linux already reports KiB. + if cfg!(target_os = "macos") { + maxrss / 1024 + } else { + maxrss + } } #[tokio::test] #[ignore = "requires exact read-only Qwen weights, verified engine and native provider"] async fn model_native_qwen_exact_profile_lifecycle() { + native_exact_profile_lifecycle(&QWEN).await; +} + +#[tokio::test] +#[ignore = "requires exact read-only SmolLM2 weights, verified engine and native provider"] +async fn model_native_smollm2_exact_profile_lifecycle() { + native_exact_profile_lifecycle(&SMOLLM2).await; +} + +async fn native_exact_profile_lifecycle(model: &'static PinnedModel) { use elastos_runtime::provider::{bridge::ProviderConfig, ProviderBridge}; - let path = - |name| PathBuf::from(std::env::var_os(name).expect("explicit native proof prerequisite")); - let weights = path("ELASTOS_TEST_QWEN_PATH").canonicalize().unwrap(); + let platform = crate::setup::detect_platform(); + let weights = model.env_path("PATH").canonicalize().unwrap(); assert_eq!( - proof_file(&weights, QWEN_BYTES).metadata().unwrap().len(), - QWEN_BYTES + proof_file(&weights, model.bytes).metadata().unwrap().len(), + model.bytes ); - let engine_data = path("ELASTOS_TEST_QWEN_ENGINE_DATA"); + let engine_data = model.env_path("ENGINE_DATA"); let mut manifest_bytes = Vec::new(); proof_file(&engine_data.join("components.json"), 4 * 1024 * 1024) .read_to_end(&mut manifest_bytes) @@ -505,7 +618,10 @@ async fn model_native_qwen_exact_profile_lifecycle() { let manifest: crate::setup::ComponentsManifest = serde_json::from_slice(&manifest_bytes).unwrap(); let engine = crate::setup::verified_local_model_engine(&engine_data, &manifest).unwrap(); - assert_eq!(engine.sha256.trim_start_matches("sha256:"), QWEN_ENGINE_SHA); + assert_eq!( + engine.sha256.trim_start_matches("sha256:"), + pinned_engine_sha(&platform) + ); let engine_path = engine.path.canonicalize().unwrap(); let base = weights .ancestors() @@ -517,13 +633,14 @@ async fn model_native_qwen_exact_profile_lifecycle() { ); // This is native startup proof, not Content delivery/admission. Reuse the // exact Runtime policy while granting read-only access to just two files. + let name = model.capsule_name; let mut offer = bound_model_offer( "native-only-fixture", "weights.gguf", - QWEN_SHA, + model.sha, &engine.receipt_sha256, &engine.sha256, - "Qwen native proof", + &format!("{name} native"), ) .unwrap(); offer.as_object_mut().unwrap().remove("stream_output"); @@ -532,15 +649,15 @@ async fn model_native_qwen_exact_profile_lifecycle() { offer["adapter"] = serde_json::json!({ "kind":"local_llama_cpp_text", "engine":{"path":engine_path,"sha256":engine.sha256}, - "model":{"path":weights,"sha256":format!("sha256:{QWEN_SHA}")}, - "settings":local_model_startup_profile(&crate::setup::detect_platform()).unwrap() + "model":{"path":weights,"sha256":format!("sha256:{}", model.sha)}, + "settings":local_model_startup_profile(&platform).unwrap() }); assert_eq!(offer["policy"]["runtime_ms_limit"], 120000); let id = offer["id"].as_str().unwrap().to_owned(); let root = tempfile::tempdir().unwrap(); fs::set_permissions(root.path(), fs::Permissions::from_mode(0o700)).unwrap(); let cleanup_path = root.path().to_owned(); - eprintln!("Qwen native proof journal: {}", cleanup_path.display()); + eprintln!("{name} native proof journal: {}", cleanup_path.display()); let config = ProviderConfig { base_path: base.to_string_lossy().into_owned(), allowed_paths: [&weights, &engine_path] @@ -553,21 +670,25 @@ async fn model_native_qwen_exact_profile_lifecycle() { ..Default::default() }; let started = Instant::now(); - let binary = path("ELASTOS_TEST_MODEL_PROVIDER_PATH"); + let binary = PathBuf::from( + std::env::var_os("ELASTOS_TEST_MODEL_PROVIDER_PATH") + .expect("explicit ELASTOS_TEST_MODEL_PROVIDER_PATH proof prerequisite"), + ); let bridge = Arc::new( ProviderBridge::spawn(&binary, config.clone()) .await .unwrap(), ); eprintln!( - "Qwen native Init elapsed_ms={}", + "{name} native Init elapsed_ms={}", started.elapsed().as_millis() ); let children = Arc::new(Mutex::new(vec![bridge.clone()])); let task_children = children.clone(); // Two independently bounded 120 s runs, plus startup/replay/cleanup margin. let mut task = tokio::spawn(async move { - let proof = qwen_runs_and_active_cancel( + let proof = pinned_runs_and_active_cancel( + model, bridge.clone(), id, Instant::now() + Duration::from_secs(300), @@ -579,7 +700,7 @@ async fn model_native_qwen_exact_profile_lifecycle() { .expect("native model provider reaped"); let restarted = Arc::new(ProviderBridge::spawn(&binary, config).await.unwrap()); task_children.lock().unwrap().push(restarted.clone()); - assert_qwen_replay(&restarted, &proof).await; + assert_pinned_replay(&restarted, &proof).await; proof.active_delta_bytes }); let result = tokio::time::timeout(Duration::from_secs(300), &mut task).await; @@ -588,7 +709,7 @@ async fn model_native_qwen_exact_profile_lifecycle() { let _ = task.await; } eprintln!( - "Qwen native terminal elapsed_ms={}", + "{name} native terminal elapsed_ms={}", started.elapsed().as_millis() ); let mut shutdowns = Vec::new(); @@ -602,15 +723,15 @@ async fn model_native_qwen_exact_profile_lifecycle() { } cleanup.expect("native proof journal cleanup"); assert!(!cleanup_path.exists()); - eprintln!("Qwen native proof provider reaped and journal removed"); + eprintln!("{name} native proof provider reaped and journal removed"); let delta_bytes = result .expect("native proof waiter deadline") .expect("native proof task"); assert!(delta_bytes > 0); - eprintln!("Qwen native active_delta_bytes={delta_bytes}; exact restart replay passed"); + eprintln!("{name} native active_delta_bytes={delta_bytes}; exact restart replay passed"); } -async fn qwen_first_reply( +async fn pinned_first_reply( bridge: Arc, offer: String, deadline: Instant, @@ -627,7 +748,7 @@ async fn qwen_first_reply( session_id: ctx.session_id.clone(), capsule_id: "assistant".into(), grant_id: ctx.grant_id.clone(), - request_id: "qwen-proof-reply".into(), + request_id: "pinned-proof-reply".into(), offer_id: offer.clone(), operation: "text.generate".into(), input_hash: model_input_hash(&input).unwrap(), @@ -658,11 +779,11 @@ async fn qwen_first_reply( } tokio::time::sleep(Duration::from_millis(100)).await; }; - report_qwen_terminal(&terminal); + report_terminal(&terminal); (get, terminal) } -fn report_qwen_terminal(terminal: &serde_json::Value) { +fn report_terminal(terminal: &serde_json::Value) { // Preserve bounded public failure facts before cleanup; never dump paths, // prompts, provider config or the complete private response. let bounded = |value: &serde_json::Value| { @@ -674,7 +795,7 @@ fn report_qwen_terminal(terminal: &serde_json::Value) { .collect::() }; eprintln!( - "Qwen terminal {}", + "pinned model terminal {}", serde_json::json!({ "status": bounded(&terminal["data"]["status"]), "error_class": bounded(&terminal["data"]["terminal"]["error"]["class"]), @@ -684,43 +805,49 @@ fn report_qwen_terminal(terminal: &serde_json::Value) { ); } -fn assert_qwen_reply(terminal: &serde_json::Value) { +fn assert_pinned_reply(model: &PinnedModel, terminal: &serde_json::Value) { assert_eq!(terminal["data"]["status"], "completed"); let text = terminal["data"]["terminal"]["output"]["text"] .as_str() .unwrap() .trim(); - assert!(text - .trim_end_matches(['.', '!']) - .eq_ignore_ascii_case("ready")); + // The reply is the only model output this proof records; keep it short. + eprintln!( + "{} completed reply chars={} head={:?}", + model.capsule_name, + text.chars().count(), + text.chars().take(64).collect::() + ); + assert!((model.reply_accepts)(text), "reply rejected by pinned model check"); } -struct QwenRunProof { +struct PinnedRunProof { replay: [(serde_json::Value, serde_json::Value); 4], active_delta_bytes: usize, } -async fn qwen_runs_and_active_cancel( +async fn pinned_runs_and_active_cancel( + model: &PinnedModel, bridge: Arc, offer: String, deadline: Instant, -) -> QwenRunProof { +) -> PinnedRunProof { use elastos_model_contract::{ model_input_hash, RuntimeAccessBinding, RuntimeCreateBinding, RUNTIME_ACCESS_BINDING_SCHEMA, RUNTIME_CREATE_BINDING_SCHEMA, }; - let (get, terminal) = qwen_first_reply(bridge.clone(), offer.clone(), deadline).await; - assert_qwen_reply(&terminal); + let (get, terminal) = pinned_first_reply(bridge.clone(), offer.clone(), deadline).await; + assert_pinned_reply(model, &terminal); let ctx = context(); let input = serde_json::json!({"schema":"elastos.model.input.text/v1", - "prompt":"Write a continuous numbered list of detailed, distinct sentences about ordinary garden plants. Produce at least 12000 characters of text. Continue the list without a summary or closing remarks."}); + "prompt":model.long_prompt}); let binding = RuntimeCreateBinding { schema: RUNTIME_CREATE_BINDING_SCHEMA.into(), principal_id: ctx.principal_id.clone(), session_id: ctx.session_id.clone(), capsule_id: "assistant".into(), grant_id: ctx.grant_id.clone(), - request_id: "qwen-proof-active-cancel".into(), + request_id: "pinned-proof-active-cancel".into(), offer_id: offer.clone(), operation: "text.generate".into(), input_hash: model_input_hash(&input).unwrap(), @@ -747,7 +874,7 @@ async fn qwen_runs_and_active_cancel( let active_delta_bytes = loop { assert!( Instant::now() < deadline, - "active Qwen delta observation deadline" + "active pinned model delta observation deadline" ); let page = bridge .send_raw(&serde_json::json!({"op":"runs_events", @@ -773,14 +900,14 @@ async fn qwen_runs_and_active_cancel( current["data"]["status"].as_str(), Some("prepared" | "running") ) { - report_qwen_terminal(¤t); + report_terminal(¤t); } assert!( matches!( current["data"]["status"].as_str(), Some("prepared" | "running") ), - "active Qwen cancellation not observed before terminal status: {}", + "active cancellation not observed before terminal status: {}", current["data"]["status"] ); if delta_bytes > 0 && current["data"]["status"] == "running" { @@ -790,20 +917,20 @@ async fn qwen_runs_and_active_cancel( }; let cancel = serde_json::json!({"op":"runs_cancel","run_id":id,"runtime_binding":access}); let cancelled = bridge.send_raw(&cancel).await.unwrap(); - report_qwen_terminal(&cancelled); + report_terminal(&cancelled); assert_eq!(cancelled["status"], "ok"); assert!( matches!( cancelled["data"]["status"].as_str(), Some("reconciling" | "settlement_unknown") ), - "active Qwen cancellation raced terminal completion: {}", + "active cancellation raced terminal completion: {}", cancelled["data"]["status"] ); let cancelled_terminal = loop { assert!( Instant::now() < deadline, - "Qwen cancellation settlement deadline" + "cancellation settlement deadline" ); let result = bridge.send_raw(&cancelled_get).await.unwrap(); assert_eq!(result["status"], "ok"); @@ -815,7 +942,7 @@ async fn qwen_runs_and_active_cancel( } tokio::time::sleep(Duration::from_millis(100)).await; }; - report_qwen_terminal(&cancelled_terminal); + report_terminal(&cancelled_terminal); // Preserve the unresolved run outcome. HTTP disconnect does not prove that // backend execution stopped; later artifact removal needs physical close proof. assert_eq!(cancelled_terminal["data"]["status"], "settlement_unknown"); @@ -845,7 +972,7 @@ async fn qwen_runs_and_active_cancel( ); assert_eq!(bridge.send_raw(&events).await.unwrap(), settled_events); } - QwenRunProof { + PinnedRunProof { replay: [ (get, terminal), (cancelled_get, cancelled_terminal.clone()), @@ -856,9 +983,9 @@ async fn qwen_runs_and_active_cancel( } } -async fn assert_qwen_replay( +async fn assert_pinned_replay( bridge: &elastos_runtime::provider::ProviderBridge, - proof: &QwenRunProof, + proof: &PinnedRunProof, ) { let mut replies = Vec::new(); for (query, _) in &proof.replay { @@ -873,7 +1000,8 @@ async fn assert_qwen_replay( } } -async fn qwen_reply_and_restart( +async fn pinned_reply_and_restart( + model: &PinnedModel, data: &Path, registry: &elastos_runtime::provider::ProviderRegistry, binary: &Path, @@ -890,14 +1018,19 @@ async fn qwen_reply_and_restart( config.extra["runtime_admitted_offers"], serde_json::json!([{"offer_id":offer}]) ); + assert_eq!( + config.extra["offers"][0]["adapter"]["settings"], + local_model_startup_profile(&crate::setup::detect_platform()).unwrap(), + "startup offer carries this host's Runtime-owned profile" + ); let bridge = Arc::new(ProviderBridge::spawn(binary, config.clone()).await.unwrap()); children.lock().unwrap().push(bridge.clone()); drop(worker); - let proof = qwen_runs_and_active_cancel(bridge.clone(), offer.clone(), deadline).await; + let proof = pinned_runs_and_active_cancel(model, bridge.clone(), offer.clone(), deadline).await; bridge .shutdown() .await - .expect("Qwen provider shutdown/reap"); + .expect("pinned model provider shutdown/reap"); children.lock().unwrap().clear(); let (restarted_config, worker) = crate::api::model_provider_config(data, registry) .await @@ -910,14 +1043,14 @@ async fn qwen_reply_and_restart( ); children.lock().unwrap().push(restarted.clone()); drop(worker); - assert_qwen_replay(&restarted, &proof).await; + assert_pinned_replay(&restarted, &proof).await; children.lock().unwrap().clear(); assert!(Inventory::open(data, false) .unwrap() .load() .unwrap() .kept(&context().principal_id, cid)); - serde_json::json!({"offer_id":offer,"cid":cid,"reply":"ready","restart_replay_exact":true, + serde_json::json!({"offer_id":offer,"cid":cid,"reply_completed":true,"restart_replay_exact":true, "keep_persisted":true,"provider_shutdown_reap":true,"engine_process_group_absence":null, "active_text_delta_bytes":proof.active_delta_bytes,"active_cancel_unknown_replay":true, "per_run_backend_stop_confirmed":false}) @@ -927,5 +1060,13 @@ async fn qwen_reply_and_restart( #[ignore = "requires explicit read-only Qwen/license/verified engine roots and native binaries; review footprint before running"] async fn model_preparation_real_qwen_cold_reply_restart() { assert_eq!(crate::setup::detect_platform(), "darwin-arm64"); - preparation_process_with_qwen(true, Some(QwenProof::from_env())).await; + preparation_process_with_model(true, Some(PinnedModelProof::from_env(&QWEN))).await; +} + +#[tokio::test] +#[ignore = "requires explicit read-only SmolLM2/license/verified engine roots and native binaries; isolated small-model proof"] +async fn model_preparation_real_smollm2_cold_reply_restart() { + // Any host with a Runtime-owned startup profile runs this small model. + local_model_startup_profile(&crate::setup::detect_platform()).unwrap(); + preparation_process_with_model(true, Some(PinnedModelProof::from_env(&SMOLLM2))).await; } diff --git a/elastos/crates/elastos-server/src/api/gateway_capsule_catalog.rs b/elastos/crates/elastos-server/src/api/gateway_capsule_catalog.rs index 91b33afa6..9ee9b15de 100644 --- a/elastos/crates/elastos-server/src/api/gateway_capsule_catalog.rs +++ b/elastos/crates/elastos-server/src/api/gateway_capsule_catalog.rs @@ -58,34 +58,40 @@ async fn caller_capsule_catalog_summary( ) -> CapsuleCatalogResponse { #[cfg(unix)] { - // Signed model catalogs currently admit exactly one entry. Observe its - // selected slot once, not once per ordinary app in the catalog. - let selected = capsule_catalog_summary(&state.data_dir) - .capsules - .into_iter() - .find(|row| row.model_content.is_some()) - .and_then(|row| row.cid); - let projection = if let Some(cid) = selected.as_deref() { - Some( - crate::api::capsule_inventory::preparation::model_runtime_projection( - &state.data_dir, - state.provider_registry.as_deref(), - context, - cid, - None, - ) - .await, - ) - } else { - None - }; // Catalog trust may be revoked while the provider is answering. Rebuild // its signed rows while retaining ordinary installed app inventory. + // Each signed model CID keeps its own admission and offer projection. + let initial = capsule_catalog_summary(&state.data_dir); + let mut projections = Vec::new(); + for row in &initial.capsules { + if row.model_content.is_some() { + if let Some(cid) = row.cid.clone() { + projections.push(( + cid.clone(), + crate::api::capsule_inventory::preparation::model_runtime_projection( + &state.data_dir, + state.provider_registry.as_deref(), + context, + &cid, + None, + ) + .await, + )); + } + } + } let mut catalog = capsule_catalog_summary(&state.data_dir); for row in &mut catalog.capsules { if row.model_content.is_some() { - row.model_runtime = Some((if row.cid == selected { projection.clone() } else { None }) - .unwrap_or_else(crate::api::capsule_inventory::preparation::unavailable_model_runtime_projection)); + row.model_runtime = Some( + projections + .iter() + .find(|(cid, _)| row.cid.as_ref() == Some(cid)) + .map(|(_, projection)| projection.clone()) + .unwrap_or_else( + crate::api::capsule_inventory::preparation::unavailable_model_runtime_projection, + ), + ); let current = row.model_runtime.as_ref().unwrap(); if current["admitted"] == true { row.state = if current["dispatch_ready"] == true { diff --git a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs index 40e2d5751..f17fe1f12 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_remote.rs @@ -344,17 +344,9 @@ fn classify_carrier_error(message: &str) -> RemoteRouteError { RemoteRouteError::Transport(message.to_string()) } -fn run_id_of(result: &Value) -> Option { - result - .pointer("/data/run/id") - .or_else(|| result.pointer("/data/run_id")) - .or_else(|| result.pointer("/run/id")) - .or_else(|| result.get("run_id")) - .and_then(Value::as_str) - .map(str::to_string) -} - -use super::gateway_model_service::run_result_is_terminal as run_is_terminal; +use super::gateway_model_service::{ + normalize_remote_model_reply, run_result_is_terminal as run_is_terminal, +}; /// Hold one route slot for a create, or recognise a request that already has /// its route (a retry after a lost reply). Check and hold share one lock and @@ -426,23 +418,35 @@ fn release_route_slot(data_dir: &Path, now: u64, key: &str) { } } +fn rewrite_protocol_offer_id(value: &mut Value, grant_id: &str) { + if let Some(Value::String(offer_id)) = value.get_mut("offer_id") { + if parse_remote_offer_id(offer_id).is_none() { + *offer_id = remote_offer_id(grant_id, offer_id); + } + } +} + +fn rewrite_protocol_offer_list(offers: Option<&mut Value>, grant_id: &str) { + let Some(Value::Array(items)) = offers else { + return; + }; + for item in items { + rewrite_protocol_offer_id(item, grant_id); + } +} + +/// Prefix protocol offer identities for the consumer. User, event, and output +/// objects keep their own fields. fn rewrite_offer_ids(value: &mut Value, grant_id: &str) { - match value { - Value::Object(map) => { - if let Some(Value::String(offer_id)) = map.get_mut("offer_id") { - if parse_remote_offer_id(offer_id).is_none() { - *offer_id = remote_offer_id(grant_id, offer_id); - } - } - for child in map.values_mut() { - rewrite_offer_ids(child, grant_id); - } + rewrite_protocol_offer_id(value, grant_id); + if let Some(data) = value.get_mut("data") { + rewrite_protocol_offer_id(data, grant_id); + if let Some(run) = data.get_mut("run") { + rewrite_protocol_offer_id(run, grant_id); } - Value::Array(items) => items - .iter_mut() - .for_each(|item| rewrite_offer_ids(item, grant_id)), - _ => {} + rewrite_protocol_offer_list(data.get_mut("offers"), grant_id); } + rewrite_protocol_offer_list(value.get_mut("offers"), grant_id); } /// Append the offers each approved grant currently shares. An unreachable @@ -581,7 +585,18 @@ pub(crate) async fn route_run_operation( return Err(err); } }; - if let Some(run_id) = run_id_of(&result) { + let reply = match normalize_remote_model_reply(&result) { + Ok(reply) => reply, + Err(_) => { + if held_slot { + release_route_slot(data_dir, now, &key); + } + return Err(RemoteRouteError::Invalid( + "model reply is ambiguous".to_string(), + )); + } + }; + if let Some(run_id) = reply.run_id { let route = RemoteRunRoute { principal_id: context.principal_id.clone(), grant_id: grant.grant_id.clone(), @@ -757,10 +772,20 @@ mod tests { #[test] fn offer_ids_in_results_are_rewritten_once() { - let mut result = json!({ "data": { "run": { "id": "run:sha256:aa", "offer_id": "qwen" }, "offer_id": "remote:g:qwen" } }); + let mut result = json!({ + "data": { + "run": { "id": "run:sha256:aa", "offer_id": "qwen" }, + "offer_id": "remote:g:qwen", + "terminal": { "output": { "offer_id": "nested-user-offer", "text": "hello" } } + } + }); rewrite_offer_ids(&mut result, "g"); assert_eq!(result["data"]["run"]["offer_id"], "remote:g:qwen"); assert_eq!(result["data"]["offer_id"], "remote:g:qwen"); + assert_eq!( + result["data"]["terminal"]["output"]["offer_id"], + "nested-user-offer" + ); } #[test] diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index d4b690fa8..ee4407dd6 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -360,29 +360,37 @@ fn release_run_slot(data_dir: &Path, now: u64, key: &str) { } } -/// Best-effort cancel of a run this Runtime dispatched but could not record. +/// Best-effort cancel of a run this Runtime dispatched but could not accept. +/// Returns true when the provider accepted the cancel request. async fn cancel_unrecorded_run( registry: &ProviderRegistry, context: &HomeLaunchTokenContext, capsule_id: &str, run_id: &str, -) { +) -> bool { let request = json!({ "run_id": run_id, "request_id": format!("unrecorded:{run_id}") }); let Ok(normalized) = normalize_model_provider_request("runs_cancel", &request, context, capsule_id) else { tracing::warn!("unrecorded model run {run_id} could not form a cancel request"); - return; + return false; }; match registry.send_raw("model", &normalized).await { Ok(result) if provider_status_error(&result).is_none() => { - tracing::info!("unrecorded model run {run_id}: cancel requested") + tracing::info!("unrecorded model run {run_id}: cancel requested"); + true + } + Ok(result) => { + tracing::warn!( + "unrecorded model run {run_id}: cancel returned {}", + provider_status_error(&result).unwrap_or_default() + ); + false + } + Err(err) => { + tracing::warn!("unrecorded model run {run_id}: cancel failed: {err}"); + false } - Ok(result) => tracing::warn!( - "unrecorded model run {run_id}: cancel returned {}", - provider_status_error(&result).unwrap_or_default() - ), - Err(err) => tracing::warn!("unrecorded model run {run_id}: cancel failed: {err}"), } } @@ -675,6 +683,110 @@ async fn read_authority( .context("model authority deadline")?? } +async fn grant_is_active( + data_dir: &Path, + network: &crate::collaboration_network::VerifiedCollaborationNetworkProfile, + source: iroh::PublicKey, + grant_id: &str, + requester_principal_id: &str, +) -> bool { + let now = crate::auth::now_ts(); + match read_authority(data_dir, network, source, grant_id, requester_principal_id).await { + Ok(grant) => grant.validate(&source, requester_principal_id, now).is_ok(), + Err(_) => false, + } +} + +#[cfg(test)] +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub(crate) enum CreateRacePoint { + BeforeDispatch, + BeforeCommit, +} + +#[cfg(test)] +struct CreateRaceBarrier { + prepared: Arc, + release: tokio::sync::watch::Sender, +} + +#[cfg(test)] +static CREATE_RACE_BARRIERS: std::sync::OnceLock< + std::sync::Mutex< + std::collections::BTreeMap<(PathBuf, CreateRacePoint), Arc>, + >, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +pub(crate) struct CreateRaceBarrierGuard { + data_dir: PathBuf, + point: CreateRacePoint, + barrier: Arc, +} + +#[cfg(test)] +impl CreateRaceBarrierGuard { + pub(crate) async fn wait_prepared(&self) { + self.barrier.prepared.wait().await; + } + + pub(crate) fn release(&self) { + let _ = self.barrier.release.send(true); + } +} + +#[cfg(test)] +impl Drop for CreateRaceBarrierGuard { + fn drop(&mut self) { + if let Ok(mut slots) = CREATE_RACE_BARRIERS.get_or_init(Default::default).lock() { + slots.remove(&(self.data_dir.clone(), self.point)); + } + let _ = self.barrier.release.send(true); + } +} + +#[cfg(test)] +pub(crate) fn install_create_race_barrier( + data_dir: impl AsRef, + point: CreateRacePoint, +) -> CreateRaceBarrierGuard { + let data_dir = data_dir.as_ref().to_path_buf(); + let (release, _) = tokio::sync::watch::channel(false); + let barrier = Arc::new(CreateRaceBarrier { + prepared: Arc::new(tokio::sync::Barrier::new(2)), + release, + }); + CREATE_RACE_BARRIERS + .get_or_init(Default::default) + .lock() + .expect("create race barrier") + .insert((data_dir.clone(), point), barrier.clone()); + CreateRaceBarrierGuard { + data_dir, + point, + barrier, + } +} + +#[cfg(test)] +async fn await_create_race_barrier(data_dir: &Path, point: CreateRacePoint) { + let barrier = CREATE_RACE_BARRIERS + .get_or_init(Default::default) + .lock() + .ok() + .and_then(|slots| slots.get(&(data_dir.to_path_buf(), point)).cloned()); + let Some(barrier) = barrier else { + return; + }; + let mut released = barrier.release.subscribe(); + barrier.prepared.wait().await; + while !*released.borrow() { + if released.changed().await.is_err() { + break; + } + } +} + /// Whether a typed model reply shows the run has settled. `runs_create`, /// `runs_get` and `runs_cancel` answer with a run view whose `status` is one of /// the terminal statuses (a `terminal` outcome accompanies it). `runs_events` @@ -710,6 +822,130 @@ pub(super) fn run_result_terminal_status(result: &Value) -> Option { }) } +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct NormalizedRemoteModelReply { + pub run_id: Option, + pub offer_id: Option, + pub terminal_status: Option, +} + +fn unique_string_pointers( + result: &Value, + pointers: &[&str], +) -> Result, &'static str> { + let mut found = None; + for pointer in pointers { + let Some(text) = result.pointer(pointer).and_then(Value::as_str) else { + continue; + }; + match &found { + None => found = Some(text.to_string()), + Some(existing) if existing == text => {} + Some(_) => return Err("ambiguous remote model reply"), + } + } + Ok(found) +} + +fn protocol_offer_list(result: &Value) -> Result>, &'static str> { + let nested = result.pointer("/data/offers"); + let top = result.get("offers"); + match (nested, top) { + (Some(left), Some(right)) if left != right => Err("ambiguous remote model reply"), + (Some(Value::Array(items)), _) | (_, Some(Value::Array(items))) => Ok(Some(items)), + (Some(_), _) | (_, Some(_)) => Err("ambiguous remote model reply"), + (None, None) => Ok(None), + } +} + +/// Read run id, offer id, and terminal status from a supported peer reply. +/// Nested user and output objects stay on the original value. Two disagreeing +/// identities are an ambiguous envelope. +pub(super) fn normalize_remote_model_reply( + result: &Value, +) -> Result { + let run_id = unique_string_pointers(result, &REMOTE_RUN_ID_POINTERS)?; + let offer_id = unique_string_pointers(result, &["/data/offer_id", "/offer_id"])?; + protocol_offer_list(result)?; + Ok(NormalizedRemoteModelReply { + run_id, + offer_id, + terminal_status: run_result_terminal_status(result), + }) +} + +const REMOTE_RUN_ID_POINTERS: [&str; 4] = ["/data/run/id", "/data/run_id", "/run/id", "/run_id"]; + +/// After dispatch, keep ownership of accepted work. A trustworthy run ID is +/// recorded and cancelled. A reservation without a run ID stays held so a +/// retry cannot dispatch again. +async fn reject_dispatched_create( + registry: &ProviderRegistry, + data_dir: &Path, + now: u64, + reservation_key: &str, + held_slot: bool, + context: &HomeLaunchTokenContext, + capsule_id: &str, + grant_id: &str, + source_did: &str, + requester_principal_id: &str, + request_id: &str, + normalized: &Value, + result: &Value, +) -> Value { + let trustworthy_run_id = unique_string_pointers(result, &REMOTE_RUN_ID_POINTERS) + .ok() + .flatten(); + if let Some(run_id) = trustworthy_run_id { + let record = RemoteModelRunRecord { + grant_id: grant_id.to_string(), + source_endpoint_did: source_did.to_string(), + requester_principal_id: requester_principal_id.to_string(), + remote_principal_id: context.principal_id.clone(), + capsule_id: capsule_id.to_string(), + offer_id: normalized["offer_id"] + .as_str() + .unwrap_or_default() + .to_string(), + operation: normalized["operation"] + .as_str() + .unwrap_or_default() + .to_string(), + request_id: request_id.to_string(), + input_hash: normalized + .pointer("/runtime_binding/input_hash") + .and_then(Value::as_str) + .unwrap_or_default() + .to_string(), + created_at: now, + terminal_status: None, + terminal_at: None, + }; + match commit_run_slot(data_dir, now, reservation_key, &run_id, record) { + Ok(()) => { + let status = if cancel_unrecorded_run(registry, context, capsule_id, &run_id).await + { + "cancelled" + } else { + "settlement_unknown" + }; + mark_run_settled(data_dir, now, &run_id, status); + } + Err(err) => { + tracing::warn!("remote model run index write failed: {err}"); + let _ = cancel_unrecorded_run(registry, context, capsule_id, &run_id).await; + if held_slot { + release_run_slot(data_dir, now, reservation_key); + } + } + } + } else { + tracing::warn!("dispatched model create kept its reservation without a trustworthy run id"); + } + denied("invalid_provider_invocation", "model reply is ambiguous") +} + /// Serve one model operation for a remote consumer. Every path returns a /// Carrier `provider_invoke` response object. pub(crate) async fn invoke( @@ -926,6 +1162,15 @@ pub(crate) async fn invoke( false }; + if held_slot { + #[cfg(test)] + await_create_race_barrier(data_dir, CreateRacePoint::BeforeDispatch).await; + if !grant_is_active(data_dir, &network, source, grant_id, requester_principal_id).await { + release_run_slot(data_dir, now, &reservation_key); + return denied("denied", "model grant is not active for this requester"); + } + } + let result = match registry.send_raw("model", &normalized).await { Ok(result) => result, Err(err) => { @@ -964,17 +1209,50 @@ pub(crate) async fn invoke( return denied(class, "model provider rejected the operation"); } + if held_slot { + #[cfg(test)] + await_create_race_barrier(data_dir, CreateRacePoint::BeforeCommit).await; + } + + let reply = normalize_remote_model_reply(&result); + if operation == "runs_create" { + let expected_offer = normalized["offer_id"].as_str().unwrap_or_default(); + let offer_mismatch = reply.as_ref().ok().is_some_and(|reply| { + reply + .offer_id + .as_deref() + .is_some_and(|got| !expected_offer.is_empty() && got != expected_offer) + }); + if reply.is_err() || offer_mismatch { + return reject_dispatched_create( + ®istry, + data_dir, + now, + &reservation_key, + held_slot, + &context, + capsule_id, + grant_id, + &source_did, + requester_principal_id, + &request_id, + &normalized, + &result, + ) + .await; + } + } + let reply = match reply { + Ok(reply) => reply, + Err(_) => { + return denied("invalid_provider_invocation", "model reply is ambiguous"); + } + }; + let result = match operation { "offers_list" => with_offers(result.clone(), shareable_offers(&result)), "runs_create" => { - let run_id = result - .pointer("/data/run/id") - .or_else(|| result.pointer("/data/run_id")) - .or_else(|| result.pointer("/run/id")) - .or_else(|| result.get("run_id")) - .and_then(Value::as_str) - .map(str::to_string); - if let Some(run_id) = run_id { + if let Some(run_id) = reply.run_id.clone() { let record = RemoteModelRunRecord { grant_id: grant_id.to_string(), source_endpoint_did: source_did.clone(), @@ -996,15 +1274,15 @@ pub(crate) async fn invoke( .unwrap_or_default() .to_string(), created_at: now, - terminal_status: run_result_terminal_status(&result), - terminal_at: run_result_is_terminal(&result).then_some(now), + terminal_status: reply.terminal_status.clone(), + terminal_at: reply.terminal_status.is_some().then_some(now), }; if let Err(err) = commit_run_slot(data_dir, now, &reservation_key, &run_id, record) { // The run exists on this Runtime but has no owner record; // settle it now rather than leave it running unaccounted. tracing::warn!("remote model run index write failed: {err}"); - cancel_unrecorded_run(®istry, &context, capsule_id, &run_id).await; + let _ = cancel_unrecorded_run(®istry, &context, capsule_id, &run_id).await; if held_slot { release_run_slot(data_dir, now, &reservation_key); } @@ -1020,8 +1298,10 @@ pub(crate) async fn invoke( } _ => { if let Some((run_id, record)) = indexed_run { - if let Some(status) = - run_result_terminal_status(&result).filter(|_| record.terminal_at.is_none()) + if let Some(status) = reply + .terminal_status + .clone() + .filter(|_| record.terminal_at.is_none()) { let _ = update_run_index(data_dir, now, |index| { if let Some(entry) = index.runs.get_mut(&run_id) { @@ -1035,6 +1315,11 @@ pub(crate) async fn invoke( result } }; + if held_slot + && !grant_is_active(data_dir, &network, source, grant_id, requester_principal_id).await + { + let _ = cancel_grant_runs(registry, data_dir, grant_id).await; + } json!({ "ok": true, "result": result }) } @@ -1198,6 +1483,47 @@ mod tests { assert_eq!(filtered["data"]["offers"].as_array().unwrap().len(), 2); } + #[test] + fn remote_reply_keeps_matching_identities_and_nested_output() { + let result = json!({ + "status": "ok", + "data": { + "run_id": "run:sha256:aa", + "run": { "id": "run:sha256:aa" }, + "offer_id": "qwen-local", + "status": "completed", + "terminal": { + "status": "completed", + "output": { "schema": "elastos.model.output.text/v1", "text": "hello" } + } + }, + "run_id": "run:sha256:aa", + "offer_id": "qwen-local" + }); + let reply = normalize_remote_model_reply(&result).expect("matching identities"); + assert_eq!(reply.run_id.as_deref(), Some("run:sha256:aa")); + assert_eq!(reply.offer_id.as_deref(), Some("qwen-local")); + assert_eq!(reply.terminal_status.as_deref(), Some("completed")); + assert_eq!( + result["data"]["terminal"]["output"]["text"], "hello", + "normalization must leave nested output on the original reply" + ); + } + + #[test] + fn remote_reply_rejects_disagreeing_run_ids_and_offer_lists() { + assert!(normalize_remote_model_reply(&json!({ + "data": { "run_id": "run:a" }, + "run_id": "run:b" + })) + .is_err()); + assert!(normalize_remote_model_reply(&json!({ + "data": { "offers": [{ "id": "qwen" }] }, + "offers": [{ "id": "other" }] + })) + .is_err()); + } + #[test] fn settlement_events_page_emits_one_terminal_event() { let record = RemoteModelRunRecord { diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs index 2e3acdaca..d4d11f074 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_model.rs @@ -4,7 +4,8 @@ use super::home_system::{ }; use super::*; use crate::api::gateway::gateway_model_service::{ - cancel_grant_runs, model_grant_id, remote_principal_id, + cancel_grant_runs, install_create_race_barrier, model_grant_id, remote_principal_id, + CreateRacePoint, }; use crate::collaboration_contact_store::CollaborationContactStore; use crate::collaboration_discovery::*; @@ -92,6 +93,10 @@ struct FakeModelProvider { run_status: std::sync::Mutex>, /// Injected once onto the next `runs_get` or `runs_events` reply. next_error: std::sync::Mutex>, + /// Overlay applied to the next `runs_create` reply `data` object. + next_create_data: std::sync::Mutex>, + /// Extra top-level `run_id` that disagrees with `data.run_id`. + next_create_alias_run_id: std::sync::Mutex>, } #[async_trait::async_trait] @@ -146,7 +151,20 @@ impl Provider for FakeModelProvider { .lock() .unwrap() .insert(run_id.clone(), "running".to_string()); - return Ok(run_view(&run_id, "running")); + let mut reply = run_view(&run_id, "running"); + if let Some(overlay) = self.next_create_data.lock().unwrap().take() { + if let Some(data) = reply.get_mut("data").and_then(Value::as_object_mut) { + if let Some(object) = overlay.as_object() { + for (key, value) in object { + data.insert(key.clone(), value.clone()); + } + } + } + } + if let Some(alias) = self.next_create_alias_run_id.lock().unwrap().take() { + reply["run_id"] = json!(alias); + } + return Ok(reply); } let binding: RuntimeAccessBinding = serde_json::from_value(request["runtime_binding"].clone()).map_err(provider_error)?; @@ -591,6 +609,11 @@ impl TwoRuntimes { .count() } + async fn revoke_grant(&self) -> anyhow::Result> { + self.write_request_record("denied"); + cancel_grant_runs(self.registry.clone(), self.owner.path(), &self.grant_id).await + } + async fn shutdown(mut self) { self.owner_service.shutdown().await.unwrap(); self.seed_node.shutdown().await; @@ -720,6 +743,109 @@ async fn run_lifecycle_stays_with_the_creating_principal_through_denial_and_revo fx.shutdown().await; } +#[tokio::test] +async fn revoke_before_dispatch_sends_no_create() { + let fx = TwoRuntimes::start().await; + let barrier = install_create_race_barrier(fx.owner.path(), CreateRacePoint::BeforeDispatch); + let (created, _) = tokio::join!( + fx.create_run("seed-req-revoke-before", "qwen-local"), + async { + barrier.wait_prepared().await; + fx.revoke_grant().await.unwrap(); + barrier.release(); + } + ); + assert_eq!(created["ok"], false, "{created}"); + assert_eq!(created["code"], "denied"); + assert_eq!(fx.provider_ops().await, vec!["offers_list"]); + assert!(fx + .run_record(&run_id_for( + &fx.remote_principal(), + "seed-req-revoke-before" + )) + .is_null()); + fx.shutdown().await; +} + +#[tokio::test] +async fn revoke_after_dispatch_keeps_ownership_and_cancels() { + let fx = TwoRuntimes::start().await; + let barrier = install_create_race_barrier(fx.owner.path(), CreateRacePoint::BeforeCommit); + let (created, _) = tokio::join!( + fx.create_run("seed-req-revoke-after", "qwen-local"), + async { + barrier.wait_prepared().await; + fx.revoke_grant().await.unwrap(); + barrier.release(); + } + ); + assert_eq!(created["ok"], true, "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-revoke-after"); + let record = fx.run_record(&run_id); + assert_eq!(record["grant_id"], fx.grant_id); + assert_eq!(record["request_id"], "seed-req-revoke-after"); + assert_eq!(record["terminal_status"], "cancelled"); + let ops = fx.provider_ops().await; + assert_eq!( + ops.iter().filter(|op| *op == "runs_create").count(), + 1, + "{ops:?}" + ); + assert!(ops.iter().any(|op| op == "runs_cancel"), "{ops:?}"); + fx.shutdown().await; +} + +#[tokio::test] +async fn a_mismatched_create_reply_keeps_the_run_and_cancels() { + let fx = TwoRuntimes::start().await; + *fx.provider.next_create_data.lock().unwrap() = Some(json!({ "offer_id": "other-local" })); + let created = fx.create_run("seed-req-mismatch", "qwen-local").await; + assert_eq!(created["ok"], false, "{created}"); + assert_eq!(created["code"], "invalid_provider_invocation", "{created}"); + let run_id = run_id_for(&fx.remote_principal(), "seed-req-mismatch"); + let record = fx.run_record(&run_id); + assert_eq!(record["request_id"], "seed-req-mismatch"); + assert_eq!(record["offer_id"], "qwen-local"); + assert_eq!(record["terminal_status"], "cancelled"); + assert_eq!(fx.pending_len(), 0); + assert_eq!(fx.create_count().await, 1); + let ops = fx.provider_ops().await; + assert!(ops.iter().any(|op| op == "runs_cancel"), "{ops:?}"); + let retried = fx.create_run("seed-req-mismatch", "qwen-local").await; + assert_eq!(retried["ok"], true, "{retried}"); + assert_eq!(retried["result"]["data"]["run_id"], run_id); + assert_eq!(fx.create_count().await, 1, "retry must not dispatch again"); + fx.shutdown().await; +} + +#[tokio::test] +async fn an_ambiguous_create_reply_keeps_the_reservation() { + let fx = TwoRuntimes::start().await; + *fx.provider.next_create_alias_run_id.lock().unwrap() = + Some("run:sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb".into()); + let created = fx.create_run("seed-req-ambiguous", "qwen-local").await; + assert_eq!(created["ok"], false, "{created}"); + assert_eq!(created["code"], "invalid_provider_invocation", "{created}"); + assert!(fx + .run_record(&run_id_for(&fx.remote_principal(), "seed-req-ambiguous")) + .is_null()); + assert_eq!(fx.pending_len(), 1); + assert_eq!(fx.create_count().await, 1); + let ops = fx.provider_ops().await; + assert!( + ops.iter().all(|op| op != "runs_cancel"), + "no trustworthy run id, no cancel: {ops:?}" + ); + let retried = fx.create_run("seed-req-ambiguous", "qwen-local").await; + assert_eq!(retried["ok"], false, "{retried}"); + assert_eq!( + fx.create_count().await, + 1, + "held reservation blocks redispatch" + ); + fx.shutdown().await; +} + impl TwoRuntimes { /// Fill the owner's run index with `count` open records, optionally /// including the record a specific earlier request would have left. @@ -771,6 +897,12 @@ impl TwoRuntimes { index["runs"].as_object().unwrap().len() + index["pending"].as_object().map_or(0, |p| p.len()) } + + fn pending_len(&self) -> usize { + let path = self.owner.path().join("services-model-runs.json"); + let index: Value = serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap(); + index["pending"].as_object().map_or(0, |p| p.len()) + } } #[tokio::test] @@ -952,6 +1084,37 @@ mod consumer_path { "runtime_binding": { "principal_id": SEED_PRINCIPAL, "request_id": request_id } }) } + #[tokio::test] + async fn route_run_operation_rewrites_protocol_offer_ids_only() { + let fx = TwoRuntimes::start().await; + let grant = fx.consumer_grant(); + *fx.provider.next_create_data.lock().unwrap() = Some(json!({ + "output": { "offer_id": "nested-user-offer", "text": "keep" } + })); + let created = fx + .route( + std::slice::from_ref(&grant), + "runs_create", + json!({ + "op": "runs_create", + "offer_id": remote_offer_id(&fx.grant_id, "qwen-local"), + "operation": "text.generate", + "input": { "prompt": "hello" }, + "runtime_binding": { "principal_id": SEED_PRINCIPAL, "request_id": "seed-req-offer" } + }), + ) + .await + .unwrap() + .expect("remote route"); + assert_eq!( + created["data"]["offer_id"], + remote_offer_id(&fx.grant_id, "qwen-local") + ); + assert_eq!(created["data"]["output"]["offer_id"], "nested-user-offer"); + assert_eq!(created["data"]["output"]["text"], "keep"); + fx.shutdown().await; + } + #[tokio::test] async fn existing_runs_settle_after_the_grant_is_denied_while_new_runs_are_refused() { let fx = TwoRuntimes::start().await; diff --git a/elastos/crates/elastos-server/src/carrier.rs b/elastos/crates/elastos-server/src/carrier.rs index 06894a236..82c55cfc9 100644 --- a/elastos/crates/elastos-server/src/carrier.rs +++ b/elastos/crates/elastos-server/src/carrier.rs @@ -41,7 +41,7 @@ use futures_lite::StreamExt; use iroh_gossip::net::Gossip; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256, Sha512}; -use tokio::io::{self, AsyncBufReadExt, AsyncWriteExt, BufReader}; +use tokio::io::{self, AsyncBufReadExt, AsyncReadExt as _, AsyncWriteExt, BufReader}; #[cfg(unix)] use tokio::net::UnixStream; use tokio::sync::Mutex; @@ -122,10 +122,18 @@ const MAX_CARRIER_OBJECT_IMPORT_BYTES: usize = 64 * 1024 * 1024; const MAX_REMOTE_RECEIPT_REPLICA_SUMMARY_ROWS: usize = 5; const GOSSIP_SEND_TIMEOUT: Duration = Duration::from_millis(1_500); const GOSSIP_JOIN_PEERS_TIMEOUT: Duration = Duration::from_millis(1_500); +/// A consumer that joins a CID topic after the holder announced still needs a +/// window for DHT/mDNS to form a gossip neighbor and for the signed fetch +/// ticket to arrive. One absolute deadline covers late holders. Operator peer +/// tickets remain the fallback when this window stays empty. Installed late-join +/// on LAN needed more than 8s (2026-09-16). +const CONTENT_AVAILABILITY_DISCOVERY_WAIT: Duration = Duration::from_secs(30); +const CONTENT_AVAILABILITY_DISCOVERY_POLL: Duration = Duration::from_millis(100); const GOSSIP_JOIN_EXACT_MAX_PEERS: usize = 16; const GOSSIP_CARRIER_PUSH_MAX_TOPIC_LEN: usize = 256; // Maximum complete JSON GossipMessage admitted to the iroh-gossip wire. const GOSSIP_WIRE_MESSAGE_MAX_BYTES: usize = 192 * 1024; +const CARRIER_BOUNDED_INVOKE_MAX_RESPONSE_BYTES: u64 = 128 * 1024; const GOSSIP_TOPIC_BUFFER_MAX_BYTES: usize = 12 * 1024 * 1024; const GOSSIP_CURSOR_MAX_CONSUMER_ID_BYTES: usize = 128; const GOSSIP_PEEK_MAX_MESSAGES: usize = 256; @@ -523,6 +531,8 @@ fn add_ticket_endpoints( /// node peer add --did ... --ticket ...`) into the endpoint's /// `MemoryLookup`, so a later `connect_resolved_peer` by peer DID can /// resolve an address without waiting on gossip/mDNS/DHT discovery. +/// Returns those endpoint IDs as gossip bootstrap peers so a content-topic +/// join can reach an operator-named holder. /// /// Never aborts Carrier startup: an absent or empty store is normal, and a /// malformed ticket is skipped with a warning rather than failing the @@ -530,7 +540,7 @@ fn add_ticket_endpoints( fn seed_address_book_from_operator_peer_store( memory_lookup: &MemoryLookup, data_dir: &std::path::Path, -) { +) -> Vec { let config = match load_operator_control(data_dir) { Ok(config) => config, Err(err) => { @@ -538,7 +548,7 @@ fn seed_address_book_from_operator_peer_store( "carrier: no operator peer store to seed address book from: {}", err ); - return; + return Vec::new(); } }; @@ -559,12 +569,13 @@ fn seed_address_book_from_operator_peer_store( continue; } - add_ticket_endpoints(memory_lookup, &mut bootstrap_peers, &endpoints, false); + add_ticket_endpoints(memory_lookup, &mut bootstrap_peers, &endpoints, true); debug!( peer_did, "seeded Carrier address book from operator peer store" ); } + bootstrap_peers } #[derive(Deserialize)] @@ -789,6 +800,8 @@ async fn join_gossip_topic_direct( let peers = state.peers.clone(); let topic_peers = state.topic_peers.clone(); let topic_key = topic_name.to_string(); + let sender_for_recv = dtt_sender.clone(); + let local_did = state.did.clone(); let receiver_task = tokio::spawn(async move { recv_loop( CarrierGossipReceiver::Direct(iroh_receiver), @@ -796,6 +809,8 @@ async fn join_gossip_topic_direct( peers, topic_peers, topic_key, + sender_for_recv, + local_did, ) .await; }); @@ -846,6 +861,8 @@ async fn join_gossip_topic( let peers = state.peers.clone(); let topic_peers = state.topic_peers.clone(); let topic_key = topic_name.to_string(); + let sender_for_recv = sender.clone(); + let local_did = state.did.clone(); let receiver_task = tokio::spawn(async move { recv_loop( CarrierGossipReceiver::Discovered(receiver), @@ -853,6 +870,8 @@ async fn join_gossip_topic( peers, topic_peers, topic_key, + sender_for_recv, + local_did, ) .await; }); @@ -1083,7 +1102,7 @@ async fn start_carrier_node_with_network( // so a peer-DID custody dial can resolve without a live discovery // round trip. Shared by every caller of this startup path (serve, // gateway, and future custody-node roles). - seed_address_book_from_operator_peer_store(&memory_lookup, &data_dir); + let bootstrap_peers = seed_address_book_from_operator_peer_store(&memory_lookup, &data_dir); let gossip = spawn_carrier_gossip(&endpoint); @@ -1094,6 +1113,10 @@ async fn start_carrier_node_with_network( Some(signing_key.clone()), Some(did.to_string()), ))); + { + let mut state = gossip_state.lock().await; + state.bootstrap_peers = bootstrap_peers; + } let file_handler = FileHandler { data_dir: data_dir.clone(), @@ -2190,6 +2213,7 @@ pub struct CarrierAvailabilityProvider { peer_reputation: Arc>>, data_dir: Option, peer_attestation_exchange: Option, + discovery_wait: Duration, } #[derive(Debug, Clone)] @@ -2214,6 +2238,7 @@ impl CarrierAvailabilityProvider { peer_reputation: Arc::new(Mutex::new(HashMap::new())), data_dir: None, peer_attestation_exchange: None, + discovery_wait: CONTENT_AVAILABILITY_DISCOVERY_WAIT, } } @@ -2227,6 +2252,7 @@ impl CarrierAvailabilityProvider { peer_reputation: Arc::new(Mutex::new(HashMap::new())), data_dir: None, peer_attestation_exchange: None, + discovery_wait: CONTENT_AVAILABILITY_DISCOVERY_WAIT, } } @@ -2265,9 +2291,16 @@ impl CarrierAvailabilityProvider { peer_reputation: Arc::new(Mutex::new(peer_reputation)), data_dir: Some(data_dir), peer_attestation_exchange, + discovery_wait: CONTENT_AVAILABILITY_DISCOVERY_WAIT, } } + #[cfg(test)] + fn with_discovery_wait(mut self, wait: Duration) -> Self { + self.discovery_wait = wait; + self + } + async fn record_peer_reputation(&self, node_did: &str, success: bool) { let snapshot = { let mut reputation = self.peer_reputation.lock().await; @@ -3467,8 +3500,12 @@ impl CarrierAvailabilityProvider { if let Err(err) = validate_carrier_content_path(path) { return Ok(carrier_availability_error("invalid_path", err)); } + let bound = match BoundedContentRead::from_request(request) { + Ok(bound) => bound, + Err(err) => return Ok(carrier_availability_error("invalid_request", err)), + }; let topic_name = content_availability_topic_name(cid); - let messages = { + { let mut state = self.state.lock().await; if !state.joined_topics.contains(&topic_name) { if state.joined_topics.len() >= MAX_TOPICS { @@ -3478,26 +3515,11 @@ impl CarrierAvailabilityProvider { )); } if let Err(err) = join_gossip_topic(&mut state, &topic_name, false).await { - return Ok(carrier_availability_error( - "join_failed", - format!("Carrier availability topic join failed: {err}"), - )); + tracing::debug!( + "Carrier availability topic join failed; announced holders are unavailable: {err}" + ); } } - let buffers = state.buffers.clone(); - drop(state); - let buffers = buffers.lock().await; - buffers - .get(&topic_name) - .map(|buffer| buffer.messages.iter().rev().cloned().collect::>()) - .unwrap_or_default() - }; - let tickets = content_availability_fetch_tickets(&messages, cid); - if tickets.is_empty() { - return Ok(carrier_availability_error( - "carrier_fetch_unavailable", - "no Carrier availability announcement with a fetch ticket is available for this CID", - )); } let Some(registry) = self @@ -3511,29 +3533,91 @@ impl CarrierAvailabilityProvider { )); }; + let deadline = tokio::time::Instant::now() + self.discovery_wait; + let mut tried = HashSet::new(); let mut errors = Vec::new(); - for ticket in tickets { - match fetch_content_via_carrier_provider_invocation(®istry, &ticket, cid, path).await - { - Ok((bytes, remote_transfer)) => { - return Ok(serde_json::json!({ - "status": "ok", - "data": { - "data": base64::engine::general_purpose::STANDARD.encode(bytes), - "availability": { - "status": "network_available", - "provider": "carrier-availability", - "policy": "carrier_provider_invoke", - "replicas": 1, - "transport": "carrier-provider-plane", - "remote_transfer": remote_transfer, - "checked_at": now_secs(), + loop { + let (messages, self_did) = { + let state = self.state.lock().await; + let self_did = state.did.clone().unwrap_or_default(); + let buffers = state.buffers.lock().await; + let messages = buffers + .get(&topic_name) + .map(|buffer| buffer.messages.iter().rev().cloned().collect::>()) + .unwrap_or_default(); + (messages, self_did) + }; + // Holders are tried one at a time in the same reputation order the + // replication path uses, so a holder that failed a previous piece drops + // behind one that served, instead of costing its timeout on every piece. + let reputation = self.peer_reputation.lock().await.clone(); + let mut replicas = + content_availability_replicas_with_reputation(&messages, cid, &reputation); + // Signed announcements are the product holder set. Operator-named + // peers fill the same gap they fill on ensure: a known holder that + // has not yet announced this CID can still serve a bounded fetch. + if let Some(data_dir) = self.data_dir.as_deref() { + append_operator_peer_store_replica_candidates( + &mut replicas, + data_dir, + &self_did, + &reputation, + now_secs(), + ); + } + let next = replicas + .into_iter() + .find(|replica| !tried.contains(&replica.node_did)); + if let Some(replica) = next { + if tried.len() >= MAX_CARRIER_REPLICATION_CANDIDATES { + break; + } + tried.insert(replica.node_did.clone()); + match fetch_content_via_carrier_provider_invocation( + ®istry, + &replica.node_did, + &replica.connect_ticket, + cid, + path, + bound, + ) + .await + { + Ok((bytes, remote_transfer)) => { + self.record_peer_reputation(&replica.node_did, true).await; + return Ok(serde_json::json!({ + "status": "ok", + "data": { + "data": base64::engine::general_purpose::STANDARD.encode(bytes), + "availability": { + "status": "network_available", + "provider": "carrier-availability", + "policy": "carrier_provider_invoke", + "replicas": 1, + "transport": "carrier-provider-plane", + "remote_transfer": remote_transfer, + "checked_at": now_secs(), + } } - } - })) + })); + } + Err(err) => { + self.record_peer_reputation(&replica.node_did, false).await; + errors.push(err.to_string()); + continue; + } } - Err(err) => errors.push(err.to_string()), } + if tokio::time::Instant::now() >= deadline { + break; + } + tokio::time::sleep(CONTENT_AVAILABILITY_DISCOVERY_POLL).await; + } + if errors.is_empty() { + return Ok(carrier_availability_error( + "carrier_fetch_unavailable", + "no Carrier availability announcement with a fetch ticket is available for this CID", + )); } Ok(carrier_availability_error( @@ -3543,11 +3627,88 @@ impl CarrierAvailabilityProvider { } } +/// Bound one Content read forwarded to a holder: a closed file range or a +/// complete-metadata byte limit. The holder's Runtime applies it once, so a +/// consumer moves a large package as 64 KiB pieces instead of whole files. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum BoundedContentRead { + Range { start: u64, end: u64 }, + Metadata { max_bytes: u64 }, +} + +const MAX_BOUNDED_CONTENT_READ_BYTES: u64 = 64 * 1024; + +impl BoundedContentRead { + fn from_request(request: &serde_json::Value) -> std::result::Result, String> { + match request.get("bounded_read") { + None | Some(serde_json::Value::Bool(false)) => return Ok(None), + Some(serde_json::Value::Bool(true)) => {} + Some(_) => return Err("bounded_read must be a boolean".to_string()), + } + let range = request.get("range"); + let max_bytes = request.get("max_bytes"); + match (range, max_bytes) { + (Some(range), None) => { + let start = range.get("start").and_then(serde_json::Value::as_u64); + let end = range.get("end").and_then(serde_json::Value::as_u64); + let (Some(start), Some(end)) = (start, end) else { + return Err("bounded read requires a closed range".to_string()); + }; + let length = end + .checked_sub(start) + .and_then(|size| size.checked_add(1)) + .filter(|size| *size <= MAX_BOUNDED_CONTENT_READ_BYTES); + if length.is_none() { + return Err("bounded read range exceeds limit".to_string()); + } + Ok(Some(Self::Range { start, end })) + } + (None, Some(max_bytes)) => { + let max_bytes = max_bytes + .as_u64() + .filter(|n| (1..=MAX_BOUNDED_CONTENT_READ_BYTES).contains(n)) + .ok_or_else(|| "bounded metadata read exceeds limit".to_string())?; + if request.get("path").and_then(serde_json::Value::as_str) + != Some(crate::content::CONTENT_OBJECT_MANIFEST_PATH) + { + return Err("bounded metadata read requires the object manifest path".into()); + } + Ok(Some(Self::Metadata { max_bytes })) + } + _ => Err("bounded read requires a range or a metadata bound, not both".to_string()), + } + } + + fn apply(self, request: &mut serde_json::Value) { + request["bounded_read"] = serde_json::Value::Bool(true); + match self { + Self::Range { start, end } => { + request["range"] = serde_json::json!({ "start": start, "end": end }); + } + Self::Metadata { max_bytes } => { + request["max_bytes"] = serde_json::json!(max_bytes); + } + } + } + + fn accepts(self, length: u64) -> bool { + match self { + Self::Range { start, end } => length == end - start + 1, + Self::Metadata { max_bytes } => (1..=max_bytes).contains(&length), + } + } +} + +/// The announcement's verified signer is the peer the ticket must reach. The +/// invoker rejects a ticket whose endpoint is another key before connecting, +/// so a holder is credited only for bytes its own authenticated peer served. async fn fetch_content_via_carrier_provider_invocation( registry: &ProviderRegistry, + holder_did: &str, ticket: &str, cid: &str, path: &str, + bound: Option, ) -> Result<(Vec, Option)> { validate_content_cid(cid).map_err(anyhow::Error::msg)?; validate_carrier_content_path(path).map_err(anyhow::Error::msg)?; @@ -3561,6 +3722,9 @@ async fn fetch_content_via_carrier_provider_invocation( if !path.is_empty() { request["path"] = serde_json::Value::String(path.to_string()); } + if let Some(bound) = bound { + bound.apply(&mut request); + } let response = registry .invoke_provider(ProviderInvocation { @@ -3573,7 +3737,7 @@ async fn fetch_content_via_carrier_provider_invocation( progress: None, transport: ProviderInvocationTransport::Carrier(ProviderCarrierRoute::ConnectTicket { connect_ticket: ticket.to_string(), - peer_did: None, + peer_did: Some(holder_did.to_string()), timeout_ms: Some(5_000), }), }) @@ -3589,6 +3753,13 @@ async fn fetch_content_via_carrier_provider_invocation( } let remote_transfer = response.get("_runtime_transfer").cloned(); let bytes = remote_content_provider_response_bytes(&response)?; + if let Some(bound) = bound { + anyhow::ensure!( + bound.accepts(bytes.len() as u64), + "holder returned {} bytes for a bounded read of {bound:?}", + bytes.len() + ); + } Ok((bytes, remote_transfer)) } @@ -4432,6 +4603,13 @@ async fn import_content_via_carrier_provider_invocation( .await { Ok(response) => Ok(response), + Err(object_err) + if object_err + .to_string() + .contains("local content object import exceeds") => + { + Err(object_err) + } Err(object_err) => import_exact_content_via_carrier_provider_invocation( registry, replica, @@ -4625,6 +4803,20 @@ async fn local_content_fetch_bytes_for_import( remote_content_provider_response_bytes(&response) } +fn carrier_legacy_object_import_declared_bytes(manifest: &ContentObjectManifest) -> Result { + let mut total_bytes = 0_u64; + for file in &manifest.files { + total_bytes = total_bytes.saturating_add(file.size); + if total_bytes > MAX_CARRIER_OBJECT_IMPORT_BYTES as u64 { + anyhow::bail!( + "local content object import exceeds {} bytes", + MAX_CARRIER_OBJECT_IMPORT_BYTES + ); + } + } + Ok(total_bytes) +} + async fn import_object_content_via_carrier_provider_invocation( registry: &ProviderRegistry, replica: &CarrierAvailabilityReplica, @@ -4648,6 +4840,7 @@ async fn import_object_content_via_carrier_provider_invocation( MAX_CARRIER_OBJECT_IMPORT_FILES ); } + carrier_legacy_object_import_declared_bytes(&manifest)?; let mut files = Vec::with_capacity(manifest.files.len()); let mut total_bytes = 0_usize; for file in &manifest.files { @@ -5355,6 +5548,7 @@ fn carrier_repair_reason( } } +#[cfg(test)] fn content_availability_replicas( messages: &[GossipMessage], cid: &str, @@ -5429,7 +5623,7 @@ fn content_availability_replicas_with_reputation( carrier_replica_candidate_score( endpoint_id.as_deref(), announced_at, - message.ts, + now_secs(), reputation.get(&node_did), ); replicas.push(CarrierAvailabilityReplica { @@ -5456,7 +5650,8 @@ fn sort_replica_candidates(replicas: &mut [CarrierAvailabilityReplica]) { }); } -/// Operator-registered peers as replication candidates. They rank below a +/// Operator-registered peers as replication candidates and as fetch +/// holders when the CID has no signed announcement yet. They rank below a /// signed announcement of the same CID (base score 40 vs 50) so an actual /// holder is always preferred, carry the local reputation adjustment like /// every other candidate, and are skipped when the DID is this node or is @@ -5515,7 +5710,7 @@ fn append_operator_peer_store_replica_candidates( fn carrier_replica_candidate_score( endpoint_id: Option<&str>, announced_at: u64, - message_ts: u64, + observed_at: u64, reputation: Option<&CarrierPeerReputation>, ) -> (u32, String, i32, String) { let mut score = 50_u32; @@ -5527,7 +5722,7 @@ fn carrier_replica_candidate_score( score = score.saturating_add(20); reasons.push("endpoint_advertised"); } - if announced_at >= message_ts.saturating_sub(60 * 60) { + if observed_at.saturating_sub(announced_at) <= 60 * 60 { score = score.saturating_add(20); reasons.push("fresh"); } else { @@ -5617,13 +5812,6 @@ fn save_carrier_peer_reputation( Ok(()) } -fn content_availability_fetch_tickets(messages: &[GossipMessage], cid: &str) -> Vec { - content_availability_replicas(messages, cid) - .into_iter() - .map(|replica| replica.connect_ticket) - .collect() -} - fn gossip_cursor_error(code: &str, message: &str) -> serde_json::Value { serde_json::json!({"status":"error","code":code,"message":message}) } @@ -6475,6 +6663,79 @@ impl Provider for CarrierGossipProvider { } } +fn is_content_availability_topic(topic: &str) -> bool { + topic.starts_with("__elastos_content/v1/") +} + +fn gossip_message_is_local_fetch_announcement(message: &GossipMessage, local_did: &str) -> bool { + let Ok(envelope) = serde_json::from_str::(&message.content) else { + return false; + }; + let Some(cid) = envelope + .get("payload") + .and_then(|payload| payload.get("cid")) + .and_then(|cid| cid.as_str()) + else { + return false; + }; + content_availability_replicas_with_reputation( + std::slice::from_ref(message), + cid, + &HashMap::new(), + ) + .iter() + .any(|replica| replica.node_did == local_did) +} + +fn local_content_availability_catchup_message( + messages: &VecDeque, + local_did: &str, +) -> Option { + messages + .iter() + .rev() + .find(|message| gossip_message_is_local_fetch_announcement(message, local_did)) + .cloned() +} + +async fn rebroadcast_local_content_availability_catchup( + buffers: &Arc>>, + topic: &str, + sender: &distributed_topic_tracker::GossipSender, + local_did: Option<&str>, +) { + if !is_content_availability_topic(topic) { + return; + } + let Some(local_did) = local_did.filter(|did| !did.is_empty()) else { + return; + }; + let message = { + let buffers = buffers.lock().await; + let Some(buffer) = buffers.get(topic) else { + return; + }; + local_content_availability_catchup_message(&buffer.messages, local_did) + }; + let Some(message) = message else { + return; + }; + let Ok(bytes) = serialize_gossip_wire_message(&message) else { + return; + }; + match tokio::time::timeout(GOSSIP_SEND_TIMEOUT, sender.broadcast(bytes)).await { + Ok(Ok(())) => { + tracing::debug!("Carrier availability catch-up broadcast on neighbor up"); + } + Ok(Err(err)) => { + tracing::debug!("Carrier availability catch-up broadcast failed: {err}"); + } + Err(_) => { + tracing::debug!("Carrier availability catch-up broadcast timed out"); + } + } +} + /// Background task: receive gossip messages and buffer them. async fn handle_gossip_event( event: iroh_gossip::api::Event, @@ -6559,11 +6820,23 @@ async fn recv_loop( peers: Arc>>, topic_peers: Arc>>>, topic: String, + sender: distributed_topic_tracker::GossipSender, + local_did: Option, ) { loop { match receiver.next().await { Ok(Some(event)) => { + let neighbor_up = matches!(&event, iroh_gossip::api::Event::NeighborUp(_)); handle_gossip_event(event, &buffers, &peers, &topic_peers, &topic).await; + if neighbor_up { + rebroadcast_local_content_availability_catchup( + &buffers, + &topic, + &sender, + local_did.as_deref(), + ) + .await; + } } Err(e) => { tracing::warn!("carrier recv_loop error on '{}': {}", topic, e); @@ -6592,6 +6865,7 @@ async fn recv_loop( pub struct CarrierProviderInvoker { peer_endpoint: Option, provider_registry: Weak, + peer_clients: Mutex>>, } #[derive(Debug, Clone)] @@ -6615,10 +6889,7 @@ pub struct BrowserCarrierStream { impl CarrierProviderInvoker { pub fn new() -> Self { - Self { - peer_endpoint: None, - provider_registry: Weak::new(), - } + Self::default() } /// Binds the invoker to the long-lived Carrier endpoint that owns peer @@ -6627,7 +6898,7 @@ impl CarrierProviderInvoker { pub fn with_carrier_endpoint(endpoint: Endpoint) -> Self { Self { peer_endpoint: Some(endpoint), - provider_registry: Weak::new(), + ..Self::default() } } @@ -6641,7 +6912,34 @@ impl CarrierProviderInvoker { Self { peer_endpoint: Some(endpoint), provider_registry, + ..Self::default() + } + } + + async fn connected_client( + &self, + peer_endpoint: &Endpoint, + addr: iroh::EndpointAddr, + timeout_secs: u64, + ) -> Result> { + let peer = addr.id; + if let Some(client) = self.peer_clients.lock().await.get(&peer).cloned() { + return Ok(client); } + let client = Arc::new( + CarrierClient::connect_known_endpoint(peer_endpoint, addr, timeout_secs).await?, + ); + let mut peers = self.peer_clients.lock().await; + Ok(peers.entry(peer).or_insert_with(|| client.clone()).clone()) + } + + async fn forget_peer(&self, peer: iroh::PublicKey) { + self.peer_clients.lock().await.remove(&peer); + } + + #[cfg(test)] + async fn retained_peer_count(&self) -> usize { + self.peer_clients.lock().await.len() } async fn invoke_loopback_provider( @@ -6746,17 +7044,41 @@ impl ProviderCarrierInvoker for CarrierProviderInvoker { let mut errors = Vec::new(); for (index, endpoint) in endpoints.into_iter().enumerate() { - match CarrierClient::connect_known_endpoint( - peer_endpoint, - endpoint, - timeout_secs, - ) - .await + let peer = endpoint.id; + match self + .connected_client(peer_endpoint, endpoint, timeout_secs) + .await { Ok(client) => { - match client.invoke_provider(invocation, request.clone()).await { + let invoke = client.invoke_provider(invocation, request.clone()); + // A bounded Content read carries no effect, so its answer + // gets the same budget as the connect and a silent holder + // yields to the next one. Every other operation keeps its + // open-ended answer: cutting it could discard a receipt for + // an effect that is still running on the peer. + let result = if bounded_content_fetch(invocation, &request) { + match tokio::time::timeout( + std::time::Duration::from_secs(timeout_secs), + invoke, + ) + .await + { + Ok(result) => result, + Err(_) => { + self.forget_peer(peer).await; + errors.push(format!( + "ticket[{index}] response deadline of {timeout_secs}s passed" + )); + continue; + } + } + } else { + invoke.await + }; + match result { Ok(response) => return Ok(response), Err(err) => { + self.forget_peer(peer).await; errors.push(carrier_provider_public_invoke_error(index, &err)) } } @@ -6789,35 +7111,40 @@ impl ProviderCarrierInvoker for CarrierProviderInvoker { .invoke_loopback_provider(peer_endpoint, invocation, request) .await; } - let client = - CarrierClient::connect_resolved_peer(peer_endpoint, public_key, timeout_secs) - .await - .map_err(|err| { - // Keep the cause. A message that will not send is - // hard enough to diagnose without the Runtime - // throwing away the reason on its way out. - tracing::debug!( - peer = %peer_did, - error = %err, - "Carrier peer connect failed" - ); - ProviderError::Unavailable( - "no verified Carrier route to the requested peer".to_string(), - ) - })?; - client - .invoke_provider(invocation, request) + let client = self + .connected_client( + peer_endpoint, + iroh::EndpointAddr::from(public_key), + timeout_secs, + ) .await .map_err(|err| { + // Keep the cause. A message that will not send is + // hard enough to diagnose without the Runtime + // throwing away the reason on its way out. + tracing::debug!( + peer = %peer_did, + error = %err, + "Carrier peer connect failed" + ); + ProviderError::Unavailable( + "no verified Carrier route to the requested peer".to_string(), + ) + })?; + match client.invoke_provider(invocation, request).await { + Ok(response) => Ok(response), + Err(err) => { + self.forget_peer(public_key).await; tracing::debug!( peer = %peer_did, error = %err, "Carrier peer invocation failed" ); - ProviderError::Provider( + Err(ProviderError::Provider( "Carrier provider invocation peer_did route failed".to_string(), - ) - }) + )) + } + } } } } @@ -6883,6 +7210,14 @@ fn carrier_route_timeout_secs(route: &ProviderCarrierRoute) -> u64 { timeout_ms.div_ceil(1_000) } +/// The one Carrier operation whose answer is bounded by its request: Content's +/// validated bounded fetch of a closed range or a metadata bound. +fn bounded_content_fetch(invocation: &ProviderInvocation, request: &serde_json::Value) -> bool { + invocation.target == "content" + && invocation.op == "fetch" + && request.get("bounded_read") == Some(&serde_json::Value::Bool(true)) +} + fn carrier_endpoint_matches_peer(endpoint: &iroh::EndpointAddr, peer_did: &str) -> bool { did_to_public_key(peer_did).is_some_and(|public_key| endpoint.id == public_key) } @@ -7152,6 +7487,7 @@ impl CarrierClient { invocation: &ProviderInvocation, request: serde_json::Value, ) -> Result { + let bounded = request.get("bounded_read") == Some(&serde_json::Value::Bool(true)); let (mut send, recv) = self.conn.open_bi().await?; let msg = carrier_provider_invoke_message(invocation, request); let mut bytes = serde_json::to_vec(&msg)?; @@ -7161,7 +7497,18 @@ impl CarrierClient { let mut reader = BufReader::new(recv); let mut line = String::new(); - reader.read_line(&mut line).await?; + if bounded { + // A bounded read holds at most 64 KiB of base64 plus its envelope; + // a holder that sends more is refused before it is buffered. + let mut limited = (&mut reader).take(CARRIER_BOUNDED_INVOKE_MAX_RESPONSE_BYTES); + limited.read_line(&mut line).await?; + anyhow::ensure!( + line.ends_with('\n'), + "bounded Carrier provider response exceeds {CARRIER_BOUNDED_INVOKE_MAX_RESPONSE_BYTES} bytes" + ); + } else { + reader.read_line(&mut line).await?; + } let response: serde_json::Value = serde_json::from_str(line.trim())?; carrier_provider_invoke_result(response) } @@ -7368,7 +7715,7 @@ pub async fn try_p2p_discovery( } #[cfg(test)] -mod tests { +pub(crate) mod tests { use super::*; use std::sync::Mutex as StdMutex; @@ -7427,7 +7774,7 @@ mod tests { message } - async fn shutdown_test_carrier_node(node: CarrierNode) { + pub(crate) async fn shutdown_test_carrier_node(node: CarrierNode) { let tasks = { let mut state = node.gossip_state.lock().await; let tasks = state @@ -7975,6 +8322,9 @@ mod tests { "request": request, })); if invocation.transfer == ProviderTransfer::Stream { + if self.fail_tickets.iter().any(|dead| dead == ticket) { + return Err(ProviderError::Provider("mock remote fetch failed".into())); + } return Ok(serde_json::json!({ "status": "ok", "data": { @@ -8364,6 +8714,77 @@ mod tests { assert!(uri.ends_with("/availability")); assert!(!topic.contains(cid)); assert!(!uri.contains(cid)); + assert!(is_content_availability_topic(&topic)); + assert!(!is_content_availability_topic( + "__elastos_internal/chat-presence-v1/#general" + )); + } + + #[test] + fn test_local_content_availability_catchup_selects_own_fetch_ticket() { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let (message, did) = signed_content_availability_message( + cid, + [26u8; 32], + "ticket:holder", + "holder-endpoint", + 1_700_000_000, + ); + let (foreign, _) = signed_content_availability_message( + cid, + [27u8; 32], + "ticket:foreign", + "foreign-endpoint", + 1_700_000_001, + ); + let buffer = test_topic_buffer([foreign, message.clone()], 0); + let selected = local_content_availability_catchup_message(&buffer.messages, &did) + .expect("holder announcement with a fetch ticket is selected"); + assert_eq!(selected.sender_id, did); + assert!(selected.content.contains("ticket:holder")); + } + + #[test] + fn test_local_content_availability_catchup_skips_ticketless_local_announce() { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let (sk, did) = elastos_identity::derive_did(&[28u8; 32]); + let payload = serde_json::json!({ + "schema": CONTENT_AVAILABILITY_ANNOUNCEMENT_SCHEMA, + "cid": cid, + "uri": format!("elastos://{cid}"), + "policy": "network_default", + "provider": "carrier-availability", + "node_did": did, + "topic": content_availability_topic_uri(cid), + "local": { + "status": "local_pinned", + "provider": "ipfs-provider", + "replicas": 0 + }, + "announced_at": 1_700_000_000 + }); + let canonical = serde_json::to_string(&payload).unwrap(); + let (signature, signer_did) = crate::crypto::domain_separated_sign( + &sk, + CONTENT_AVAILABILITY_ANNOUNCEMENT_DOMAIN, + canonical.as_bytes(), + ); + let message = GossipMessage { + sender_id: signer_did.clone(), + sender_nick: "content-provider".to_string(), + content: serde_json::json!({ + "payload": payload, + "signature": signature, + "signer_did": signer_did, + }) + .to_string(), + ts: 1_700_000_000, + nonce: 1, + signature: None, + sender_session_id: None, + }; + let buffer = test_topic_buffer([message], 0); + assert!(local_content_availability_catchup_message(&buffer.messages, &did).is_none()); } #[test] @@ -8441,7 +8862,11 @@ mod tests { ..signed_message.clone() }; - let tickets = content_availability_fetch_tickets(&[unsigned_message, signed_message], cid); + let tickets: Vec = + content_availability_replicas(&[unsigned_message, signed_message], cid) + .into_iter() + .map(|replica| replica.connect_ticket) + .collect(); assert_eq!(tickets, vec!["ticket:test".to_string()]); } @@ -8521,15 +8946,20 @@ mod tests { #[test] fn test_content_availability_replicas_are_scored_and_sorted() { let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; - let (mut stale, stale_did) = - signed_content_availability_message(cid, [26u8; 32], "ticket:stale", "", 10); - stale.ts = 1_700_000_000; + let now = now_secs(); + let (stale, stale_did) = signed_content_availability_message( + cid, + [26u8; 32], + "ticket:stale", + "", + now.saturating_sub(2 * 60 * 60), + ); let (fresh, fresh_did) = signed_content_availability_message( cid, [27u8; 32], "ticket:fresh", "remote-endpoint", - 1_700_000_000, + now, ); let replicas = content_availability_replicas(&[stale, fresh], cid); @@ -8552,22 +8982,63 @@ mod tests { ); } + #[test] + fn test_carrier_legacy_object_import_rejects_declared_oversize_before_allocation() { + let oversized = ContentObjectManifest { + schema: "elastos.content.object.manifest/v1".into(), + kind: "capsule".into(), + content_digest: "sha256:00".into(), + files: vec![crate::content::ContentObjectFile { + path: "weights.gguf".into(), + sha256: "00".into(), + size: MAX_CARRIER_OBJECT_IMPORT_BYTES as u64 + 1, + }], + links: Vec::new(), + object_did: None, + publisher_did: None, + }; + let err = carrier_legacy_object_import_declared_bytes(&oversized).unwrap_err(); + assert!( + err.to_string() + .contains("local content object import exceeds"), + "{err}" + ); + let small = ContentObjectManifest { + schema: "elastos.content.object.manifest/v1".into(), + kind: "capsule".into(), + content_digest: "sha256:00".into(), + files: vec![crate::content::ContentObjectFile { + path: "capsule.json".into(), + sha256: "00".into(), + size: 12, + }], + links: Vec::new(), + object_did: None, + publisher_did: None, + }; + assert_eq!( + carrier_legacy_object_import_declared_bytes(&small).unwrap(), + 12 + ); + } + #[test] fn test_content_availability_replicas_apply_local_runtime_reputation() { let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let now = now_secs(); let (preferred, preferred_did) = signed_content_availability_message( cid, [28u8; 32], "ticket:preferred", "remote-endpoint", - 1_700_000_000, + now, ); let (penalized, penalized_did) = signed_content_availability_message( cid, [29u8; 32], "ticket:penalized", "remote-endpoint", - 1_700_000_000, + now, ); let mut reputation = HashMap::new(); reputation.insert( @@ -8625,7 +9096,7 @@ mod tests { assert!(carrier_peer_reputation_path(data_dir.path()).is_file()); } - fn signed_content_availability_message( + pub(crate) fn signed_content_availability_message( cid: &str, key_seed: [u8; 32], connect_ticket: &str, @@ -9146,7 +9617,7 @@ mod tests { /// publication under load and hand out an id with no dialable address. /// Production waits for `endpoint.online()` before snapshotting; tests /// only need a direct IP transport address, so wait for exactly that. - async fn wait_for_direct_endpoint_addr(endpoint: &Endpoint) -> iroh::EndpointAddr { + pub(crate) async fn wait_for_direct_endpoint_addr(endpoint: &Endpoint) -> iroh::EndpointAddr { let mut watcher = endpoint.watch_addr(); for _ in 0..100 { let addr = watcher.get(); @@ -9162,17 +9633,814 @@ mod tests { panic!("carrier test endpoint never published a direct address"); } - /// Deterministic peer-DID resolver setup. - /// - /// Runtime keeps one long-lived Carrier endpoint and owns peer resolution. - /// Seeding the remote node's verified address into that endpoint's address - /// book makes resolution in-process, so a routed invocation does not depend - /// on public discovery timing. Nothing here is visible to apps: the route - /// stays a peer DID and the address never leaves Runtime. - async fn peer_did_route_fixture(local_seed: u8, remote_seed: u8) -> PeerDidRouteFixture { - let local_dir = tempfile::tempdir().unwrap(); - let local_registry = Arc::new(ProviderRegistry::new()); - let (local_sk, local_did) = elastos_identity::derive_did(&[local_seed; 32]); + /// A holder's local IPFS backend for one synthetic closure. It answers only + /// bounded reads with the Runtime receipts a real ipfs-provider returns. + pub(crate) struct BoundedClosureBackend { + pub(crate) files: std::collections::BTreeMap>, + pub(crate) requests: Arc>>, + } + + #[async_trait::async_trait] + impl Provider for BoundedClosureBackend { + async fn handle( + &self, + _: elastos_runtime::provider::ResourceRequest, + ) -> std::result::Result { + Err(ProviderError::Provider("fixture raw only".into())) + } + fn schemes(&self) -> Vec<&'static str> { + vec![] + } + fn name(&self) -> &'static str { + "bounded-closure-backend" + } + async fn send_raw( + &self, + request: &serde_json::Value, + ) -> std::result::Result { + self.requests.lock().unwrap().push(request.clone()); + if request["op"] != "cat" || request["bounded_read"] != true { + return Ok(serde_json::json!({"status":"error","code":"unbounded", + "message":"fixture holder serves bounded reads only"})); + } + let path = request["path"].as_str().unwrap_or_default(); + let Some(file) = self.files.get(path) else { + return Ok(serde_json::json!({"status":"error","code":"not_found", + "message":"fixture path missing"})); + }; + let (bytes, receipt) = if let Some(max) = request.get("max_bytes") { + ( + file.clone(), + serde_json::json!({"_runtime_complete_metadata":{ + "schema":"elastos.provider.complete-metadata/v1", "cid":request["cid"], + "path":path, "max_bytes":max, "actual_bytes":file.len(), "completed":true + }}), + ) + } else { + let range = &request["_runtime_invocation"]["range"]; + let start = range["start"].as_u64().unwrap() as usize; + let end = range["end"].as_u64().unwrap() as usize; + if end >= file.len() || start > end { + return Ok(serde_json::json!({"status":"error","code":"range", + "message":"fixture range exceeds the file"})); + } + ( + file[start..=end].to_vec(), + serde_json::json!({"_runtime_applied_range":{ + "schema":"elastos.provider.applied-range/v1", "cid":request["cid"], + "path":path, "start":start, "end":end + }}), + ) + }; + let mut data = receipt; + data["data"] = + serde_json::json!(base64::engine::general_purpose::STANDARD.encode(bytes)); + Ok(serde_json::json!({"status":"ok","data":data})) + } + } + + /// An empty consumer cache: every local read misses. + pub(crate) struct EmptyIpfsBackend; + + #[async_trait::async_trait] + impl Provider for EmptyIpfsBackend { + async fn handle( + &self, + _: elastos_runtime::provider::ResourceRequest, + ) -> std::result::Result { + Err(ProviderError::Provider("fixture raw only".into())) + } + fn schemes(&self) -> Vec<&'static str> { + vec![] + } + fn name(&self) -> &'static str { + "empty-ipfs-backend" + } + async fn send_raw( + &self, + _: &serde_json::Value, + ) -> std::result::Result { + Ok(serde_json::json!({"status":"error","code":"not_found", + "message":"consumer cache is empty"})) + } + } + + /// A holder whose Content answers a bounded read with more bytes than asked. + struct OversizedContentProvider { + extra: usize, + requests: Arc>>, + } + + #[async_trait::async_trait] + impl Provider for OversizedContentProvider { + async fn handle( + &self, + _: elastos_runtime::provider::ResourceRequest, + ) -> std::result::Result { + Err(ProviderError::Provider("fixture raw only".into())) + } + fn schemes(&self) -> Vec<&'static str> { + vec![] + } + fn name(&self) -> &'static str { + "oversized-content" + } + async fn send_raw( + &self, + request: &serde_json::Value, + ) -> std::result::Result { + self.requests.lock().unwrap().push(request.clone()); + let range = &request["range"]; + let asked = + range["end"].as_u64().unwrap_or(0) - range["start"].as_u64().unwrap_or(0) + 1; + let bytes = vec![b'x'; asked as usize + self.extra]; + Ok(serde_json::json!({"status":"ok","data":{ + "data": base64::engine::general_purpose::STANDARD.encode(bytes) + }})) + } + } + + /// A connected holder that accepts the invocation and never answers. + pub(crate) struct SilentContentProvider { + pub(crate) requests: Arc>>, + } + + #[async_trait::async_trait] + impl Provider for SilentContentProvider { + async fn handle( + &self, + _: elastos_runtime::provider::ResourceRequest, + ) -> std::result::Result { + Err(ProviderError::Provider("fixture raw only".into())) + } + fn schemes(&self) -> Vec<&'static str> { + vec![] + } + fn name(&self) -> &'static str { + "silent-content" + } + async fn send_raw( + &self, + request: &serde_json::Value, + ) -> std::result::Result { + self.requests.lock().unwrap().push(request.clone()); + std::future::pending().await + } + } + + /// Signed holder announcements placed in the consumer's topic buffer, newest + /// last in the slice; the consumer ranks them by score, then newest first. + pub(crate) async fn seed_content_availability_announcements( + node: &CarrierNode, + cid: &str, + announcements: &[(String, [u8; 32], String, u64)], + ) { + let messages: Vec = announcements + .iter() + .map(|(ticket, seed, endpoint, announced_at)| { + signed_content_availability_message(cid, *seed, ticket, endpoint, *announced_at).0 + }) + .collect(); + let topic = content_availability_topic_name(cid); + let mut guard = node.gossip_state.lock().await; + guard.joined_topics.insert(topic.clone()); + guard + .buffers + .lock() + .await + .insert(topic, test_topic_buffer(messages, 0)); + } + + /// A holder Runtime with real Content over a bounded closure backend. + pub(crate) async fn start_content_holder_runtime( + seed: u8, + files: std::collections::BTreeMap>, + requests: Arc>>, + ) -> HolderRuntime { + let dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + registry + .register_sub_provider( + "content", + Arc::new(crate::content::ContentProvider::new( + dir.path().to_path_buf(), + Arc::downgrade(®istry), + )), + ) + .await + .unwrap(); + registry + .register_sub_provider("ipfs", Arc::new(BoundedClosureBackend { files, requests })) + .await + .unwrap(); + let (sk, did) = elastos_identity::derive_did(&[seed; 32]); + let node = start_isolated_carrier_node_with_registry( + &sk, + &did, + dir.path().to_path_buf(), + Some(Arc::downgrade(®istry)), + ) + .await + .unwrap(); + let addr = wait_for_direct_endpoint_addr(&node.endpoint).await; + HolderRuntime { + ticket: encode_ticket_for(addr.clone()), + node, + did, + addr, + _registry: registry, + _dir: dir, + } + } + + pub(crate) struct HolderRuntime { + pub(crate) node: CarrierNode, + pub(crate) did: String, + pub(crate) ticket: String, + pub(crate) addr: iroh::EndpointAddr, + _registry: Arc, + _dir: tempfile::TempDir, + } + + pub(crate) async fn start_holder_runtime( + seed: u8, + content: Arc, + backend: Option>, + ) -> HolderRuntime { + let dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + registry + .register_sub_provider("content", content) + .await + .unwrap(); + if let Some(backend) = backend { + registry + .register_sub_provider("ipfs", backend) + .await + .unwrap(); + } + let (sk, did) = elastos_identity::derive_did(&[seed; 32]); + let node = start_isolated_carrier_node_with_registry( + &sk, + &did, + dir.path().to_path_buf(), + Some(Arc::downgrade(®istry)), + ) + .await + .unwrap(); + let addr = wait_for_direct_endpoint_addr(&node.endpoint).await; + HolderRuntime { + ticket: encode_ticket_for(addr.clone()), + node, + did, + addr, + _registry: registry, + _dir: dir, + } + } + + #[tokio::test] + async fn test_bounded_content_transfer_between_runtimes_fails_over_holders_and_stays_bounded() { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let weights: Vec = (0..200_000u32) + .map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8) + .collect(); + let index = serde_json::to_vec(&serde_json::json!({ + "schema":"elastos.content.object.manifest/v1", "kind":"capsule", + "files":[{"path":"weights.gguf","size":weights.len()}] + })) + .unwrap(); + let files = std::collections::BTreeMap::from([ + ("weights.gguf".to_string(), weights.clone()), + ( + crate::content::CONTENT_OBJECT_MANIFEST_PATH.to_string(), + index.clone(), + ), + ]); + + // Consumer Runtime: empty cache, real Content, real Carrier availability. + let consumer_dir = tempfile::tempdir().unwrap(); + let consumer_registry = Arc::new(ProviderRegistry::new()); + let (consumer_sk, consumer_did) = elastos_identity::derive_did(&[80u8; 32]); + let consumer_node = start_isolated_carrier_node_with_registry( + &consumer_sk, + &consumer_did, + consumer_dir.path().to_path_buf(), + Some(Arc::downgrade(&consumer_registry)), + ) + .await + .unwrap(); + consumer_registry + .set_carrier_invoker(Arc::new( + CarrierProviderInvoker::with_carrier_endpoint_and_registry( + consumer_node.endpoint.clone(), + Arc::downgrade(&consumer_registry), + ), + )) + .await; + consumer_registry + .register_sub_provider("ipfs", Arc::new(EmptyIpfsBackend)) + .await + .unwrap(); + consumer_registry + .register_sub_provider( + "content", + Arc::new(crate::content::ContentProvider::new( + consumer_dir.path().to_path_buf(), + Arc::downgrade(&consumer_registry), + )), + ) + .await + .unwrap(); + let availability = Arc::new(CarrierAvailabilityProvider::with_provider_registry( + consumer_node.gossip_state.clone(), + Arc::downgrade(&consumer_registry), + )); + consumer_registry.register(availability.clone()).await; + + // Holders: one that announced and is gone, one that oversizes, one real. + let dead = start_holder_runtime(81, Arc::new(EmptyIpfsBackend), None).await; + let dead_ticket = dead.ticket.clone(); + let dead_did = dead.did.clone(); + shutdown_test_carrier_node(dead.node).await; + let oversized_requests = Arc::new(StdMutex::new(Vec::new())); + let oversized = start_holder_runtime( + 82, + Arc::new(OversizedContentProvider { + extra: 1, + requests: oversized_requests.clone(), + }), + None, + ) + .await; + let holder_requests = Arc::new(StdMutex::new(Vec::new())); + let holder = start_content_holder_runtime(83, files, holder_requests.clone()).await; + consumer_node + .memory_lookup + .add_endpoint_info(holder.addr.clone()); + + let now = now_secs(); + let announce = |ticket: &str, seed: u8, did: &str, at: u64| { + (ticket.to_string(), [seed; 32], did.to_string(), at) + }; + seed_content_availability_announcements( + &consumer_node, + cid, + &[ + announce(&holder.ticket, 83, &holder.did, now - 20), + announce(&oversized.ticket, 82, &oversized.did, now - 10), + announce(&dead_ticket, 81, &dead_did, now), + ], + ) + .await; + + // Exact bytes at every boundary, assembled from 64 KiB pieces. + let started = std::time::Instant::now(); + let mut assembled = Vec::new(); + let mut offset = 0u64; + while offset < weights.len() as u64 { + let length = (weights.len() as u64 - offset).min(65536); + let piece = crate::content::fetch_model_part( + &consumer_registry, + cid, + "weights.gguf", + Some((offset, length)), + ) + .await + .unwrap(); + assert_eq!(piece.len() as u64, length); + assembled.extend_from_slice(&piece); + offset += length; + } + let elapsed = started.elapsed(); + assert_eq!(assembled, weights); + assert_eq!( + Sha256::digest(&assembled).as_slice(), + Sha256::digest(&weights).as_slice() + ); + let last = crate::content::fetch_model_part( + &consumer_registry, + cid, + "weights.gguf", + Some((weights.len() as u64 - 1, 1)), + ) + .await + .unwrap(); + assert_eq!(last, weights[weights.len() - 1..]); + assert!(crate::content::fetch_model_part( + &consumer_registry, + cid, + "weights.gguf", + Some((weights.len() as u64 - 1, 2)), + ) + .await + .is_err()); + // Complete metadata is bounded by a maximum, so a holder ranked ahead + // of the real one can answer within bound with the wrong bytes; the + // transport layer delivers them and the caller's comparison against + // the signed catalog is what rejects them (prepare() does exactly that). + let metadata = crate::content::fetch_model_part( + &consumer_registry, + cid, + crate::content::CONTENT_OBJECT_MANIFEST_PATH, + None, + ) + .await + .unwrap(); + assert!(metadata.len() <= 65536); + let metadata_from_real_holder = metadata == index; + + // The real holder saw only bounded reads, one per piece, none whole-file. + let served = holder_requests.lock().unwrap().clone(); + assert_eq!( + served.len(), + 4 + 1 + 1 + usize::from(metadata_from_real_holder), + "{served:#?}" + ); + for request in &served { + assert_eq!(request["op"], "cat"); + assert_eq!(request["bounded_read"], true); + if request["path"] == "weights.gguf" { + let range = &request["_runtime_invocation"]["range"]; + let length = range["end"].as_u64().unwrap() - range["start"].as_u64().unwrap() + 1; + assert!(length <= 65536, "{range}"); + } else { + assert_eq!(request["max_bytes"], 65536); + } + } + // The oversizing holder was consulted before the real one and refused. + assert!(!oversized_requests.lock().unwrap().is_empty()); + for request in oversized_requests.lock().unwrap().iter() { + assert_eq!(request["bounded_read"], true); + assert_eq!(request["local_only"], true); + } + eprintln!( + "bounded transfer: {} bytes in 4 pieces over real Carrier with dead+oversized holders first: {:?}", + weights.len(), + elapsed + ); + + // With only misbehaving holders the read fails closed, at the payload + // bound and at the wire cap. + seed_content_availability_announcements( + &consumer_node, + cid, + &[announce(&oversized.ticket, 82, &oversized.did, now)], + ) + .await; + let bounded_probe = serde_json::json!({"op":"fetch","cid":cid,"path":"weights.gguf", + "bounded_read":true,"range":{"start":0,"end":3}}); + let refusal = consumer_registry + .send_raw("availability", &bounded_probe) + .await + .unwrap(); + assert_eq!(refusal["code"], "carrier_fetch_failed", "{refusal}"); + assert!( + refusal["message"] + .as_str() + .unwrap() + .contains("holder returned 5 bytes"), + "{refusal}" + ); + assert!(crate::content::fetch_model_part( + &consumer_registry, + cid, + "weights.gguf", + Some((0, 4)) + ) + .await + .is_err()); + let flood_requests = Arc::new(StdMutex::new(Vec::new())); + let flood = start_holder_runtime( + 84, + Arc::new(OversizedContentProvider { + extra: 200_000, + requests: flood_requests.clone(), + }), + None, + ) + .await; + seed_content_availability_announcements( + &consumer_node, + cid, + &[announce(&flood.ticket, 84, &flood.did, now)], + ) + .await; + let refusal = consumer_registry + .send_raw("availability", &bounded_probe) + .await + .unwrap(); + assert_eq!(refusal["code"], "carrier_fetch_failed", "{refusal}"); + assert_eq!(flood_requests.lock().unwrap().len(), 1); + assert!(crate::content::fetch_model_part( + &consumer_registry, + cid, + "weights.gguf", + Some((0, 4)) + ) + .await + .is_err()); + // The public invocation error is redacted; the cap itself is visible + // on the direct client hop, before the flood is buffered. + let client = + CarrierClient::connect_known_endpoint(&consumer_node.endpoint, flood.addr.clone(), 5) + .await + .unwrap(); + let mut flood_request = bounded_probe.clone(); + flood_request["local_only"] = serde_json::json!(true); + flood_request["transfer"] = serde_json::json!("bytes"); + flood_request["_runtime_invocation"] = serde_json::json!({ + "schema": "elastos.provider.invocation/v1", + "source": "carrier-availability", + "target": "content", + "op": "fetch", + "capability": "provider:carrier-availability->content:fetch", + "transport": "carrier-provider-plane", + "carrier": null, + "transfer": "bytes", + "range": null, + "progress": null + }); + let invocation = ProviderInvocation { + source: "carrier-availability".into(), + target: "content".into(), + op: "fetch".into(), + request: flood_request.clone(), + transfer: ProviderTransfer::Bytes, + range: None, + progress: None, + transport: ProviderInvocationTransport::Local, + }; + let err = client + .invoke_provider(&invocation, flood_request) + .await + .unwrap_err() + .to_string(); + assert!( + err.contains(&format!( + "exceeds {CARRIER_BOUNDED_INVOKE_MAX_RESPONSE_BYTES} bytes" + )), + "{err}" + ); + + // A connected holder that never answers holds the caller for its answer + // budget (connect already succeeded), then the next holder serves the piece. + let silent_requests = Arc::new(StdMutex::new(Vec::new())); + let silent = start_holder_runtime( + 85, + Arc::new(SilentContentProvider { + requests: silent_requests.clone(), + }), + None, + ) + .await; + // A fresh consumer has no history, so the newest announcement is tried first. + availability.peer_reputation.lock().await.clear(); + seed_content_availability_announcements( + &consumer_node, + cid, + &[ + announce(&holder.ticket, 83, &holder.did, now - 20), + announce(&silent.ticket, 85, &silent.did, now), + ], + ) + .await; + let before = holder_requests.lock().unwrap().len(); + let started = std::time::Instant::now(); + let piece = + crate::content::fetch_model_part(&consumer_registry, cid, "weights.gguf", Some((0, 4))) + .await + .unwrap(); + let waited = started.elapsed(); + assert_eq!(piece, weights[..4]); + assert_eq!(silent_requests.lock().unwrap().len(), 1); + assert_eq!(holder_requests.lock().unwrap().len(), before + 1); + assert!( + waited >= std::time::Duration::from_secs(5) + && waited < std::time::Duration::from_secs(9), + "a connected silent holder costs its answer budget, then the next holder serves: {waited:?}" + ); + eprintln!("silent connected holder failed over to the real holder after {waited:?}"); + + // An announcement signed by one key that points its ticket at another + // peer is refused before connecting and earns that signer no credit. + let (_, impostor_did) = elastos_identity::derive_did(&[86u8; 32]); + seed_content_availability_announcements( + &consumer_node, + cid, + &[announce(&holder.ticket, 86, &impostor_did, now)], + ) + .await; + let before = holder_requests.lock().unwrap().len(); + let refusal = consumer_registry + .send_raw("availability", &bounded_probe) + .await + .unwrap(); + assert_eq!(refusal["code"], "carrier_fetch_failed", "{refusal}"); + assert_eq!( + holder_requests.lock().unwrap().len(), + before, + "the real holder must not be reached through a foreign signer's announcement" + ); + { + let reputation = availability.peer_reputation.lock().await; + let impostor = reputation.get(&impostor_did).expect("mismatch is recorded"); + assert_eq!((impostor.successes, impostor.failures), (0, 1)); + let real = reputation + .get(&holder.did) + .expect("the real holder has history"); + assert!(real.successes >= 1); + assert_eq!(real.failures, 0); + assert_eq!( + reputation + .get(&silent.did) + .map(|r| (r.successes, r.failures)), + Some((0, 1)) + ); + } + + // Cancellation drops an in-flight attempt against the dead holder; the + // retry against the real holder completes with one more bounded read. + seed_content_availability_announcements( + &consumer_node, + cid, + &[announce(&dead_ticket, 81, &dead_did, now)], + ) + .await; + let cancelled = tokio::time::timeout( + std::time::Duration::from_millis(300), + crate::content::fetch_model_part(&consumer_registry, cid, "weights.gguf", Some((0, 4))), + ) + .await; + assert!( + cancelled.is_err(), + "the attempt must still be waiting on the dead holder" + ); + seed_content_availability_announcements( + &consumer_node, + cid, + &[announce(&holder.ticket, 83, &holder.did, now)], + ) + .await; + let before = holder_requests.lock().unwrap().len(); + let retried = + crate::content::fetch_model_part(&consumer_registry, cid, "weights.gguf", Some((0, 4))) + .await + .unwrap(); + assert_eq!(retried, weights[..4]); + assert_eq!(holder_requests.lock().unwrap().len(), before + 1); + let metadata = crate::content::fetch_model_part( + &consumer_registry, + cid, + crate::content::CONTENT_OBJECT_MANIFEST_PATH, + None, + ) + .await + .unwrap(); + assert_eq!( + metadata, index, + "the real holder's index arrives exactly within its bound" + ); + + shutdown_test_carrier_node(silent.node).await; + shutdown_test_carrier_node(flood.node).await; + shutdown_test_carrier_node(oversized.node).await; + shutdown_test_carrier_node(holder.node).await; + shutdown_test_carrier_node(consumer_node).await; + } + + /// A holder whose Content answers every operation after a fixed delay. + struct SlowContentProvider { + delay: std::time::Duration, + } + + #[async_trait::async_trait] + impl Provider for SlowContentProvider { + async fn handle( + &self, + _: elastos_runtime::provider::ResourceRequest, + ) -> std::result::Result { + Err(ProviderError::Provider("fixture raw only".into())) + } + fn schemes(&self) -> Vec<&'static str> { + vec![] + } + fn name(&self) -> &'static str { + "slow-content" + } + async fn send_raw( + &self, + request: &serde_json::Value, + ) -> std::result::Result { + tokio::time::sleep(self.delay).await; + Ok(serde_json::json!({"status":"ok","data":{ + "op": request["op"], "receipt": "effect-completed" + }})) + } + } + + #[tokio::test] + async fn test_response_deadline_applies_only_to_bounded_content_fetch() { + let local_dir = tempfile::tempdir().unwrap(); + let local_registry = Arc::new(ProviderRegistry::new()); + let (local_sk, local_did) = elastos_identity::derive_did(&[87u8; 32]); + let local_node = start_isolated_carrier_node_with_registry( + &local_sk, + &local_did, + local_dir.path().to_path_buf(), + Some(Arc::downgrade(&local_registry)), + ) + .await + .unwrap(); + local_registry + .set_carrier_invoker(Arc::new( + CarrierProviderInvoker::with_carrier_endpoint_and_registry( + local_node.endpoint.clone(), + Arc::downgrade(&local_registry), + ), + )) + .await; + let slow = start_holder_runtime( + 88, + Arc::new(SlowContentProvider { + delay: std::time::Duration::from_millis(2_500), + }), + None, + ) + .await; + let route = + |op: &str, request: serde_json::Value, transfer: ProviderTransfer| ProviderInvocation { + source: "carrier-availability".into(), + target: "content".into(), + op: op.into(), + request, + transfer, + range: None, + progress: None, + transport: ProviderInvocationTransport::Carrier( + ProviderCarrierRoute::ConnectTicket { + connect_ticket: slow.ticket.clone(), + peer_did: Some(slow.did.clone()), + timeout_ms: Some(1_000), + }, + ), + }; + + // An effectful operation keeps its open-ended answer past the route budget. + let started = std::time::Instant::now(); + let response = local_registry + .invoke_provider(route( + "import_object", + serde_json::json!({"op":"import_object","cid":"bafyfixture"}), + ProviderTransfer::Json, + )) + .await + .unwrap(); + assert_eq!( + response["data"]["receipt"], "effect-completed", + "{response}" + ); + assert!( + started.elapsed() >= std::time::Duration::from_millis(2_500), + "the receipt arrived after the route budget without being cut" + ); + + // The bounded fetch to the same holder is cut at the route budget. + let started = std::time::Instant::now(); + let err = local_registry + .invoke_provider(route( + "fetch", + serde_json::json!({"op":"fetch","cid":"bafyfixture","path":"weights.gguf", + "local_only":true,"bounded_read":true,"range":{"start":0,"end":3}, + "transfer":"bytes"}), + ProviderTransfer::Bytes, + )) + .await + .unwrap_err() + .to_string(); + let cut_after = started.elapsed(); + assert!(err.contains("response deadline of 1s passed"), "{err}"); + assert!( + cut_after >= std::time::Duration::from_secs(1) + && cut_after < std::time::Duration::from_millis(2_400), + "bounded fetch is cut at its budget, before the slow answer: {cut_after:?}" + ); + + shutdown_test_carrier_node(slow.node).await; + shutdown_test_carrier_node(local_node).await; + } + + /// Deterministic peer-DID resolver setup. + /// + /// Runtime keeps one long-lived Carrier endpoint and owns peer resolution. + /// Seeding the remote node's verified address into that endpoint's address + /// book makes resolution in-process, so a routed invocation does not depend + /// on public discovery timing. Nothing here is visible to apps: the route + /// stays a peer DID and the address never leaves Runtime. + async fn peer_did_route_fixture(local_seed: u8, remote_seed: u8) -> PeerDidRouteFixture { + let local_dir = tempfile::tempdir().unwrap(); + let local_registry = Arc::new(ProviderRegistry::new()); + let (local_sk, local_did) = elastos_identity::derive_did(&[local_seed; 32]); let local_node = start_isolated_carrier_node_with_registry( &local_sk, &local_did, @@ -9446,6 +10714,10 @@ mod tests { assert_eq!(response["status"], "ok"); assert_eq!(response["data"]["op"], "evaluate"); + assert_eq!( + local_node.gossip_state.lock().await.bootstrap_peers, + vec![remote_addr.id] + ); let requests = remote_requests.lock().unwrap().clone(); assert_eq!(requests.len(), 1); assert_eq!(requests[0]["_runtime_invocation"]["target"], "custody"); @@ -9673,6 +10945,36 @@ mod tests { shutdown_test_carrier_node(fixture.local_node).await; } + #[tokio::test] + async fn test_connect_ticket_route_retains_one_peer_connection_across_invokes() { + let fixture = peer_did_route_fixture(80, 81).await; + let invoker = Arc::new(CarrierProviderInvoker::with_carrier_endpoint_and_registry( + fixture.local_node.endpoint.clone(), + Arc::downgrade(&fixture.local_registry), + )); + fixture + .local_registry + .set_carrier_invoker(invoker.clone()) + .await; + + for expected in 1..=2usize { + let response = fixture + .local_registry + .invoke_provider(connect_ticket_invocation( + fixture.remote_addr.clone(), + &fixture.remote_did, + )) + .await + .unwrap(); + assert_eq!(response["status"], "ok"); + assert_eq!(fixture.remote_requests.lock().unwrap().len(), expected); + } + assert_eq!(invoker.retained_peer_count().await, 1); + + shutdown_test_carrier_node(fixture.remote_node).await; + shutdown_test_carrier_node(fixture.local_node).await; + } + #[tokio::test] async fn test_connect_ticket_route_without_runtime_endpoint_fails_before_provider_effect() { let fixture = peer_did_route_fixture(74, 75).await; @@ -10225,9 +11527,11 @@ mod tests { let (bytes, remote_transfer) = fetch_content_via_carrier_provider_invocation( ®istry, + "did:key:z6MkholderFixture", "ticket:internal-secret", "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi", "docs/readme.md", + None, ) .await .unwrap(); @@ -10273,9 +11577,11 @@ mod tests { let err = fetch_content_via_carrier_provider_invocation( ®istry, + "did:key:z6MkholderFixture", "ticket:internal-secret", "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi", "docs/readme.md", + None, ) .await .expect_err("failing carrier invocation must stay bounded"); @@ -11279,37 +12585,322 @@ mod tests { assert_eq!(quota["requirements_exceed_quota"], true); } - #[test] - fn test_carrier_remote_candidate_limit_keeps_live_multi_peer_requirement() { - let requirements = CarrierAvailabilityRequirements { - min_replicas: 2, - max_replicas: None, - require_live_multi_peer_proof: true, - repair_graph_kind: CarrierRepairGraphKind::Auto, - }; - - assert_eq!(carrier_remote_candidate_limit(requirements, 2, 2), 1); - - let quota_blocked = CarrierAvailabilityRequirements { - min_replicas: 2, - max_replicas: Some(2), - require_live_multi_peer_proof: true, - repair_graph_kind: CarrierRepairGraphKind::Auto, - }; - assert_eq!(carrier_remote_candidate_limit(quota_blocked, 2, 2), 0); + #[test] + fn test_carrier_remote_candidate_limit_keeps_live_multi_peer_requirement() { + let requirements = CarrierAvailabilityRequirements { + min_replicas: 2, + max_replicas: None, + require_live_multi_peer_proof: true, + repair_graph_kind: CarrierRepairGraphKind::Auto, + }; + + assert_eq!(carrier_remote_candidate_limit(requirements, 2, 2), 1); + + let quota_blocked = CarrierAvailabilityRequirements { + min_replicas: 2, + max_replicas: Some(2), + require_live_multi_peer_proof: true, + repair_graph_kind: CarrierRepairGraphKind::Auto, + }; + assert_eq!(carrier_remote_candidate_limit(quota_blocked, 2, 2), 0); + } + + #[tokio::test] + async fn test_carrier_availability_ensure_replicates_to_operator_peer_store_peers_without_announcements( + ) { + // A freshly published CID has no announcements on its topic. The + // peers registered through `elastos node peer add` are the replica + // targets this node was told to use, so `ensure` must invoke them + // and count their signed proofs as live remote replicas. + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (_remote_sk, remote_did) = elastos_identity::derive_did(&[22u8; 32]); + let (local_sk, local_did) = elastos_identity::derive_did(&[21u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let remote_ticket = carrier_connect_ticket(&endpoint); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name, test_topic_buffer([], 0)); + } + let data_dir = tempfile::tempdir().unwrap(); + crate::operator_control::upsert_peer( + data_dir.path(), + crate::operator_control::OperatorPeer { + did: remote_did.clone(), + label: "custody-a".to_string(), + connect_ticket: remote_ticket.clone(), + allow: Vec::new(), + }, + ) + .unwrap(); + // This node itself in the store must never become its own candidate. + crate::operator_control::upsert_peer( + data_dir.path(), + crate::operator_control::OperatorPeer { + did: local_did.clone(), + label: "self".to_string(), + connect_ticket: remote_ticket.clone(), + allow: Vec::new(), + }, + ) + .unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + let response = provider + .send_raw(&serde_json::json!({ + "op": "ensure", + "cid": cid, + "uri": format!("elastos://{cid}"), + "policy": "network_default", + "local": { + "status": "local_pinned", + "provider": "ipfs-provider", + "replicas": 1 + }, + "requirements": { + "min_replicas": 2, + "max_replicas": 2, + "require_live_multi_peer_proof": true + }, + "object_did": "did:key:zObject", + "publisher_did": "did:key:zPublisher" + })) + .await + .unwrap(); + let availability = &response["data"]["availability"]; + assert_eq!(response["status"], "ok", "{response}"); + assert_eq!(availability["status"], "network_available"); + assert_eq!(availability["replicas"], 2); + assert_eq!( + availability["peer_selection"]["mode"], + "carrier_provider_replication" + ); + assert_eq!( + availability["peer_selection"]["live_multi_peer_proof"], + true + ); + assert_eq!(availability["abuse_controls"]["candidate_count"], 1); + let remote = &availability["peer_selection"]["replicas"][1]; + assert_eq!(remote["node_did"], remote_did); + assert!(remote["selection_reason"] + .as_str() + .unwrap() + .starts_with("operator_peer_store")); + let requests = invoker.requests.lock().await; + assert!(requests + .iter() + .all(|request| request["ticket"] == remote_ticket)); + assert!(requests.iter().any(|request| request["op"] == "ensure")); + } + + #[tokio::test] + async fn test_carrier_availability_fetch_uses_operator_peer_store_without_announcements() { + // A consumer Get has no signed announcement until a holder publishes + // one. The peers registered through `elastos node peer add` are the + // holders this node was told to use, so `fetch` must invoke them. + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (_remote_sk, remote_did) = elastos_identity::derive_did(&[22u8; 32]); + let (local_sk, local_did) = elastos_identity::derive_did(&[21u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let remote_ticket = carrier_connect_ticket(&endpoint); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name, test_topic_buffer([], 0)); + } + let data_dir = tempfile::tempdir().unwrap(); + crate::operator_control::upsert_peer( + data_dir.path(), + crate::operator_control::OperatorPeer { + did: remote_did.clone(), + label: "holder-a".to_string(), + connect_ticket: remote_ticket.clone(), + allow: Vec::new(), + }, + ) + .unwrap(); + crate::operator_control::upsert_peer( + data_dir.path(), + crate::operator_control::OperatorPeer { + did: local_did.clone(), + label: "self".to_string(), + connect_ticket: remote_ticket.clone(), + allow: Vec::new(), + }, + ) + .unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + assert_eq!( + response["data"]["availability"]["policy"], + "carrier_provider_invoke" + ); + let requests = invoker.requests.lock().await; + assert!( + requests + .iter() + .all(|request| request["ticket"] == remote_ticket), + "{requests:?}" + ); + assert!(requests.iter().any(|request| request["op"] == "fetch")); + assert_eq!(requests.len(), 1); + } + + #[tokio::test] + async fn test_carrier_availability_fetch_waits_for_a_late_signed_announcement() { + // A consumer that joins after the holder announced still has an empty + // local buffer. The fetch waits for the signed announcement instead of + // returning carrier_fetch_unavailable before gossip delivers it. + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[25u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let holder_ticket = carrier_connect_ticket(&endpoint); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name.clone(), test_topic_buffer([], 0)); + } + let buffers = state.lock().await.buffers.clone(); + let ticket = holder_ticket.clone(); + let topic = topic_name.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(200)).await; + let (message, _) = + signed_content_availability_message(cid, [26u8; 32], &ticket, "", now_secs()); + let mut buffers = buffers.lock().await; + let buffer = buffers.get_mut(&topic).expect("topic buffer"); + assert!(push_gossip_buffer_message(buffer, message)); + }); + let data_dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + let started = std::time::Instant::now(); + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + assert!(started.elapsed() >= Duration::from_millis(200)); + assert!(started.elapsed() < Duration::from_secs(2)); + let requests = invoker.requests.lock().await; + assert!( + requests + .iter() + .all(|request| request["ticket"] == holder_ticket), + "{requests:?}" + ); + assert!(requests.iter().any(|request| request["op"] == "fetch")); + assert_eq!(requests.len(), 1); } #[tokio::test] - async fn test_carrier_availability_ensure_replicates_to_operator_peer_store_peers_without_announcements( - ) { - // A freshly published CID has no announcements on its topic. The - // peers registered through `elastos node peer add` are the replica - // targets this node was told to use, so `ensure` must invoke them - // and count their signed proofs as live remote replicas. + async fn test_carrier_availability_fetch_waits_past_a_non_holder_neighbor_for_a_later_holder() { let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; let topic_name = content_availability_topic_name(cid); - let (_remote_sk, remote_did) = elastos_identity::derive_did(&[22u8; 32]); - let (local_sk, local_did) = elastos_identity::derive_did(&[21u8; 32]); + let (local_sk, local_did) = elastos_identity::derive_did(&[31u8; 32]); let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) .bind() @@ -11321,7 +12912,7 @@ mod tests { .unwrap() .add(memory_lookup.clone()); let gossip = spawn_carrier_gossip(&endpoint); - let remote_ticket = carrier_connect_ticket(&endpoint); + let holder_ticket = carrier_connect_ticket(&endpoint); let state = Arc::new(Mutex::new(GossipState::new( endpoint.clone(), gossip, @@ -11336,30 +12927,27 @@ mod tests { .buffers .lock() .await - .insert(topic_name, test_topic_buffer([], 0)); + .insert(topic_name.clone(), test_topic_buffer([], 0)); + guard + .topic_peers + .lock() + .await + .entry(topic_name.clone()) + .or_default() + .insert("did:key:zNonHolder".to_string()); } + let buffers = state.lock().await.buffers.clone(); + let ticket = holder_ticket.clone(); + let topic = topic_name.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(2500)).await; + let (message, _) = + signed_content_availability_message(cid, [32u8; 32], &ticket, "", now_secs()); + let mut buffers = buffers.lock().await; + let buffer = buffers.get_mut(&topic).expect("topic buffer"); + assert!(push_gossip_buffer_message(buffer, message)); + }); let data_dir = tempfile::tempdir().unwrap(); - crate::operator_control::upsert_peer( - data_dir.path(), - crate::operator_control::OperatorPeer { - did: remote_did.clone(), - label: "custody-a".to_string(), - connect_ticket: remote_ticket.clone(), - allow: Vec::new(), - }, - ) - .unwrap(); - // This node itself in the store must never become its own candidate. - crate::operator_control::upsert_peer( - data_dir.path(), - crate::operator_control::OperatorPeer { - did: local_did.clone(), - label: "self".to_string(), - connect_ticket: remote_ticket.clone(), - allow: Vec::new(), - }, - ) - .unwrap(); let registry = Arc::new(ProviderRegistry::new()); let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); registry.set_carrier_invoker(invoker.clone()).await; @@ -11367,52 +12955,182 @@ mod tests { state, Arc::downgrade(®istry), data_dir.path().to_path_buf(), + ) + .with_discovery_wait(Duration::from_secs(4)); + let started = std::time::Instant::now(); + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + assert!(started.elapsed() >= Duration::from_millis(2500)); + assert!(started.elapsed() < Duration::from_secs(4)); + let requests = invoker.requests.lock().await; + assert_eq!(requests.len(), 1); + assert_eq!(requests[0]["ticket"], holder_ticket); + } + + #[tokio::test] + async fn test_carrier_availability_fetch_walks_a_stale_candidate_then_a_later_live_holder() { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[33u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let live_ticket = carrier_connect_ticket(&endpoint); + let (stale_message, _) = signed_content_availability_message( + cid, + [34u8; 32], + "ticket:stale", + "", + now_secs().saturating_sub(2 * 60 * 60), ); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name.clone(), test_topic_buffer([stale_message], 0)); + } + let buffers = state.lock().await.buffers.clone(); + let ticket = live_ticket.clone(); + let topic = topic_name.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(300)).await; + let (message, _) = + signed_content_availability_message(cid, [35u8; 32], &ticket, "", now_secs()); + let mut buffers = buffers.lock().await; + let buffer = buffers.get_mut(&topic).expect("topic buffer"); + assert!(push_gossip_buffer_message(buffer, message)); + }); + let data_dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker { + fail_tickets: vec!["ticket:stale".to_string()], + ..Default::default() + }); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ) + .with_discovery_wait(Duration::from_secs(2)); let response = provider .send_raw(&serde_json::json!({ - "op": "ensure", + "op": "fetch", "cid": cid, - "uri": format!("elastos://{cid}"), - "policy": "network_default", - "local": { - "status": "local_pinned", - "provider": "ipfs-provider", - "replicas": 1 - }, - "requirements": { - "min_replicas": 2, - "max_replicas": 2, - "require_live_multi_peer_proof": true - }, - "object_did": "did:key:zObject", - "publisher_did": "did:key:zPublisher" + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" })) .await .unwrap(); - let availability = &response["data"]["availability"]; assert_eq!(response["status"], "ok", "{response}"); - assert_eq!(availability["status"], "network_available"); - assert_eq!(availability["replicas"], 2); - assert_eq!( - availability["peer_selection"]["mode"], - "carrier_provider_replication" + let requests = invoker.requests.lock().await; + assert!( + requests + .iter() + .any(|request| request["ticket"] == "ticket:stale"), + "{requests:?}" ); - assert_eq!( - availability["peer_selection"]["live_multi_peer_proof"], - true + assert!( + requests + .iter() + .any(|request| request["ticket"] == live_ticket), + "{requests:?}" ); - assert_eq!(availability["abuse_controls"]["candidate_count"], 1); - let remote = &availability["peer_selection"]["replicas"][1]; - assert_eq!(remote["node_did"], remote_did); - assert!(remote["selection_reason"] - .as_str() + } + + #[tokio::test] + async fn test_carrier_availability_fetch_ends_at_the_deadline_when_holders_stay_unavailable() { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[36u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() .unwrap() - .starts_with("operator_peer_store")); - let requests = invoker.requests.lock().await; - assert!(requests - .iter() - .all(|request| request["ticket"] == remote_ticket)); - assert!(requests.iter().any(|request| request["op"] == "ensure")); + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let (dead_message, _) = + signed_content_availability_message(cid, [37u8; 32], "ticket:dead", "", now_secs()); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name, test_topic_buffer([dead_message], 0)); + } + let data_dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker { + fail_tickets: vec!["ticket:dead".to_string()], + ..Default::default() + }); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ) + .with_discovery_wait(Duration::from_millis(400)); + let started = std::time::Instant::now(); + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "error", "{response}"); + assert_eq!(response["code"], "carrier_fetch_failed"); + assert!(started.elapsed() >= Duration::from_millis(400)); + assert!(started.elapsed() < Duration::from_secs(2)); } #[tokio::test] @@ -11881,7 +13599,7 @@ mod tests { assert!(decoded["endpoints"].is_array()); } - fn encode_ticket_for(endpoint: iroh::EndpointAddr) -> String { + pub(crate) fn encode_ticket_for(endpoint: iroh::EndpointAddr) -> String { let ticket_json = serde_json::json!({ "topic": null, "endpoints": [endpoint], diff --git a/elastos/crates/elastos-server/src/content.rs b/elastos/crates/elastos-server/src/content.rs index cbb605a7c..ef9b7a662 100644 --- a/elastos/crates/elastos-server/src/content.rs +++ b/elastos/crates/elastos-server/src/content.rs @@ -3074,9 +3074,7 @@ impl ContentProvider { { Ok(result) => result, Err(local_err) - if transfer.bounded_read - || request.get("local_only").and_then(|value| value.as_bool()) - == Some(true) => + if request.get("local_only").and_then(|value| value.as_bool()) == Some(true) => { return Err(local_err); } @@ -3180,15 +3178,24 @@ impl ContentProvider { if !path.is_empty() { request["path"] = Value::String(path.to_string()); } + // A bounded read travels as its own request so the holder applies the + // exact range or metadata bound once and returns only those bytes. + // Runtime's outer invocation carries no range to slice again. + let mut hop = transfer.clone(); + if transfer.bounded_read { + request["bounded_read"] = Value::Bool(true); + if let Some(range) = transfer.range { + request["range"] = json!({ "start": range.start, "end": range.end }); + } + if let Some(max_bytes) = transfer.max_bytes { + request["max_bytes"] = json!(max_bytes); + } + hop.range = None; + hop.progress = None; + } let response = match self - .invoke_provider_with_fetch_transfer( - registry, - "availability", - "fetch", - request, - transfer, - ) + .invoke_provider_with_fetch_transfer(registry, "availability", "fetch", request, &hop) .await { Ok(response) => response, @@ -3196,7 +3203,20 @@ impl ContentProvider { Err(err) => return Err(err), }; if response.get("status").and_then(|status| status.as_str()) == Some("error") { - return Ok(None); + let code = response + .get("code") + .and_then(|value| value.as_str()) + .unwrap_or("availability_error"); + let message = response + .get("message") + .and_then(|value| value.as_str()) + .unwrap_or("Carrier availability fetch failed"); + return Err(ProviderError::Provider(format!( + "availability fetch failed ({code}): {message}" + ))); + } + if transfer.bounded_read { + check_bounded_availability_payload(&response, transfer)?; } let availability = response .get("data") @@ -8386,6 +8406,54 @@ fn provider_transfer_value(response: &Value) -> Option { response.get("_runtime_transfer").cloned() } +/// A holder answering a bounded read returns base64 bytes whose length is the +/// requested range exactly, or one to `max_bytes` for complete metadata. +fn check_bounded_availability_payload( + response: &Value, + transfer: &ContentFetchTransfer, +) -> Result<(), ProviderError> { + let invalid = + |detail: &str| ProviderError::Provider(format!("bounded availability payload {detail}")); + let expected = match (transfer.range, transfer.max_bytes) { + (Some(range), None) => range + .end + .and_then(|end| end.checked_sub(range.start)) + .and_then(|size| size.checked_add(1)) + .ok_or_else(|| invalid("requires a closed range"))?, + (None, Some(max_bytes)) => max_bytes, + _ => return Err(invalid("requires a range or a metadata bound")), + }; + // Runtime normalized a stream transfer from the decoded bytes already; + // a bytes transfer still carries the holder's base64 payload. + let length = if let Some(stream) = response.pointer("/data/stream") { + stream + .get("total_bytes") + .and_then(Value::as_u64) + .ok_or_else(|| invalid("stream lacks total_bytes"))? + } else { + let encoded = response + .pointer("/data/data") + .and_then(Value::as_str) + .ok_or_else(|| invalid("must be base64 data or a stream"))?; + if encoded.len() as u64 > expected.div_ceil(3) * 4 { + return Err(invalid("exceeds the requested bound")); + } + base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|_| invalid("is not valid base64"))? + .len() as u64 + }; + let accepted = if transfer.range.is_some() { + length == expected + } else { + (1..=expected).contains(&length) + }; + if !accepted { + return Err(invalid("length does not match the requested bound")); + } + Ok(()) +} + fn is_valid_cid(value: &str) -> bool { cid::Cid::try_from(value).is_ok() } @@ -11533,7 +11601,120 @@ mod tests { .is_err()); } assert_eq!(backend.requests.lock().await.len(), 6); - assert!(availability.requests.lock().await.is_empty()); + // Each local failure consults the availability plane once with the + // same bound; its empty answer leaves the read failed. + let forwarded = availability.requests.lock().await; + assert_eq!(forwarded.len(), 3); + for request in forwarded.iter() { + assert_eq!(request["bounded_read"], true); + assert_eq!(request["range"], json!({ "start": 8, "end": 11 })); + assert!(request["_runtime_invocation"]["range"].is_null()); + } + } + + #[tokio::test] + async fn content_bounded_read_retrieves_exact_range_from_availability_when_local_misses() { + let (_root, registry, ipfs, _content) = registry_with_content_and_ipfs().await; + { + let mut missing = ipfs.missing_paths.lock().await; + missing.push("weights.gguf".to_string()); + missing.push(CONTENT_OBJECT_MANIFEST_PATH.to_string()); + } + let encode = |bytes: &[u8]| base64::engine::general_purpose::STANDARD.encode(bytes); + let availability = Arc::new(MockAvailabilityProvider { + requests: Mutex::new(Vec::new()), + response: Mutex::new(provider_ok(json!({ + "data": encode(b"89ab"), + "availability": { + "status": "network_available", "provider": "mock-availability", + "policy": "carrier_provider_invoke", "replicas": 1 + } + }))), + }); + registry.register(availability.clone()).await; + let invoke = |request: Value, transfer: ProviderTransfer| { + registry.invoke_provider(ProviderInvocation { + source: "runtime-model-preparation".into(), + target: "content".into(), + op: "fetch".into(), + request, + transfer, + range: None, + progress: None, + transport: ProviderInvocationTransport::Local, + }) + }; + let range_request = json!({ "op": "fetch", "cid": TEST_CID, "path": "weights.gguf", + "bounded_read": true, "range": { "start": 8, "end": 11 }, "transfer": "bytes" }); + let response = invoke(range_request.clone(), ProviderTransfer::Bytes) + .await + .unwrap(); + assert_eq!(response["data"]["data"], encode(b"89ab")); + assert_eq!( + response["data"]["availability"]["provider"], + "mock-availability" + ); + { + let sent = availability.requests.lock().await; + assert_eq!(sent.len(), 1); + assert_eq!(sent[0]["cid"], TEST_CID); + assert_eq!(sent[0]["path"], "weights.gguf"); + assert_eq!(sent[0]["bounded_read"], true); + assert_eq!(sent[0]["range"], json!({ "start": 8, "end": 11 })); + assert!( + sent[0]["_runtime_invocation"]["range"].is_null(), + "the holder applies the range once; Runtime must not slice again" + ); + } + let mut stream_request = range_request.clone(); + stream_request["transfer"] = json!("stream"); + let response = invoke(stream_request, ProviderTransfer::Stream) + .await + .unwrap(); + assert_eq!( + decode_test_stream_payload(&response["data"]["stream"]), + b"89ab" + ); + for bytes in [b"89abc".as_slice(), b"89a".as_slice(), b"".as_slice()] { + *availability.response.lock().await = provider_ok(json!({ "data": encode(bytes) })); + assert!( + invoke(range_request.clone(), ProviderTransfer::Bytes) + .await + .is_err(), + "{} bytes for a 4-byte range", + bytes.len() + ); + } + let metadata_request = json!({ "op": "fetch", "cid": TEST_CID, + "path": CONTENT_OBJECT_MANIFEST_PATH, "bounded_read": true, + "max_bytes": 65536, "transfer": "bytes" }); + *availability.response.lock().await = + provider_ok(json!({ "data": encode(b"{\"schema\":\"fixture\"}") })); + let response = invoke(metadata_request.clone(), ProviderTransfer::Bytes) + .await + .unwrap(); + assert_eq!( + response["data"]["data"], + encode(b"{\"schema\":\"fixture\"}") + ); + assert_eq!( + availability.requests.lock().await.last().unwrap()["max_bytes"], + 65536 + ); + *availability.response.lock().await = + provider_ok(json!({ "data": encode(&vec![b'x'; 65537]) })); + assert!(invoke(metadata_request, ProviderTransfer::Bytes) + .await + .is_err()); + let sent_before_local_only = availability.requests.lock().await.len(); + let mut local_only = range_request.clone(); + local_only["local_only"] = json!(true); + assert!(invoke(local_only, ProviderTransfer::Bytes).await.is_err()); + assert_eq!( + availability.requests.lock().await.len(), + sent_before_local_only, + "local_only never reaches the availability plane" + ); } #[tokio::test] @@ -11661,7 +11842,17 @@ mod tests { assert!(request["_runtime_invocation"]["range"].is_null()); assert!(request["_runtime_invocation"]["progress"]["expected_bytes"].is_null()); } - assert!(availability.requests.lock().await.is_empty()); + drop(sent); + // Each local failure forwards the same metadata bound to the + // availability plane once; its empty answer leaves the read failed. + let forwarded = availability.requests.lock().await; + assert_eq!(forwarded.len(), count - 2); + for request in forwarded.iter() { + assert_eq!(request["bounded_read"], true); + assert_eq!(request["max_bytes"], 65536); + assert_eq!(request["path"], OBJECT_MANIFEST_PATH); + assert!(request["_runtime_invocation"]["range"].is_null()); + } } #[tokio::test] diff --git a/elastos/crates/elastos-server/src/provider_resource.rs b/elastos/crates/elastos-server/src/provider_resource.rs index ac64afe79..d520636e1 100644 --- a/elastos/crates/elastos-server/src/provider_resource.rs +++ b/elastos/crates/elastos-server/src/provider_resource.rs @@ -1377,6 +1377,9 @@ mod tests { build_capability_resource("browser-engine", "raw_socket", &serde_json::json!({})) .is_err() ); + assert!( + build_capability_resource("browser-engine", "wallet", &serde_json::json!({})).is_err() + ); } #[test] diff --git a/elastos/crates/elastos-server/src/publish.rs b/elastos/crates/elastos-server/src/publish.rs index 470c44234..56a8256d0 100644 --- a/elastos/crates/elastos-server/src/publish.rs +++ b/elastos/crates/elastos-server/src/publish.rs @@ -39,6 +39,7 @@ const HOME_PUBLISH_CAPSULES: &[&str] = &[ "inbox", "assistant", "elacity-player", + "model-provider", ]; const DEFAULT_PUBLISH_CAPSULES: &[&str] = HOME_PUBLISH_CAPSULES; const DEMO_PUBLISH_CAPSULES: &[&str] = @@ -75,6 +76,7 @@ const REQUIRED_SUPPORTED_PUBLISH_CAPSULES: &[&str] = &[ "inbox", "assistant", "elacity-player", + "model-provider", ]; const ALLOWED_RELEASE_CHANNELS: &[&str] = &["stable", "canary", "jetson-test"]; diff --git a/elastos/crates/elastos-server/src/runtime_control.rs b/elastos/crates/elastos-server/src/runtime_control.rs index 93319fdbd..aecbae1a1 100644 --- a/elastos/crates/elastos-server/src/runtime_control.rs +++ b/elastos/crates/elastos-server/src/runtime_control.rs @@ -558,7 +558,16 @@ fn sync_managed_runtime_child_components_manifest( let child_data_dir = managed_runtime_child_data_dir(child_home); let child_bin_dir = child_data_dir.join("bin"); std::fs::create_dir_all(&child_bin_dir)?; - std::fs::write(child_data_dir.join("components.json"), manifest_bytes)?; + std::fs::write(child_data_dir.join("components.json"), &manifest_bytes)?; + let verified_manifest: crate::setup::ComponentsManifest = + serde_json::from_slice(&manifest_bytes).map_err(|error| { + anyhow::anyhow!("managed home runtime requires a valid parent manifest: {error}") + })?; + crate::setup::install_signed_model_catalog( + &child_data_dir, + &verified_manifest, + &parent_manifest, + )?; for (name, parent_binary) in verified_binaries { std::fs::copy(parent_binary, child_bin_dir.join(name))?; } @@ -1840,6 +1849,71 @@ mod tests { ) .is_err() ); + assert!(!child_data_dir.join("model-catalog.json").exists()); + } + + #[test] + fn subordinate_managed_runtime_copies_signed_model_catalog_into_child_data_dir() { + let tmp = tempfile::tempdir().unwrap(); + let home = managed_runtime_child_home_dir(tmp.path(), RUNTIME_KIND_MANAGED_HOME); + write_managed_runtime_parent_support_artifacts(tmp.path()); + let catalog = br#"{"payload":{"schema":"elastos.model.catalog/v1"},"signature":"ab","signer_did":"did:key:z"}"#; + write_parent_signed_model_catalog(tmp.path(), catalog); + + sync_managed_runtime_child_components_manifest(tmp.path(), &home).unwrap(); + + let child_catalog = managed_runtime_child_data_dir(&home).join("model-catalog.json"); + assert_eq!(std::fs::read(&child_catalog).unwrap(), catalog); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::metadata(&child_catalog) + .unwrap() + .permissions() + .mode() + & 0o777, + 0o600 + ); + } + } + + #[test] + fn subordinate_managed_runtime_rejects_missing_signed_model_catalog_snapshot() { + let tmp = tempfile::tempdir().unwrap(); + let home = managed_runtime_child_home_dir(tmp.path(), RUNTIME_KIND_MANAGED_HOME); + write_managed_runtime_parent_support_artifacts(tmp.path()); + pin_parent_model_catalog_head( + tmp.path(), + &crate::setup::catalog_head_cid(b"absent").unwrap(), + ); + + let error = sync_managed_runtime_child_components_manifest(tmp.path(), &home).unwrap_err(); + + assert!(error.to_string().contains("model-catalog.json")); + assert!(error.to_string().contains("missing")); + assert!(!managed_runtime_child_data_dir(&home) + .join("model-catalog.json") + .exists()); + } + + #[test] + fn subordinate_managed_runtime_rejects_mismatched_signed_model_catalog_snapshot() { + let tmp = tempfile::tempdir().unwrap(); + let home = managed_runtime_child_home_dir(tmp.path(), RUNTIME_KIND_MANAGED_HOME); + write_managed_runtime_parent_support_artifacts(tmp.path()); + std::fs::write(tmp.path().join("model-catalog.json"), b"catalog-a").unwrap(); + pin_parent_model_catalog_head( + tmp.path(), + &crate::setup::catalog_head_cid(b"catalog-b").unwrap(), + ); + + let error = sync_managed_runtime_child_components_manifest(tmp.path(), &home).unwrap_err(); + + assert!(error.to_string().contains("does not match the pinned")); + assert!(!managed_runtime_child_data_dir(&home) + .join("model-catalog.json") + .exists()); } #[test] @@ -1971,6 +2045,22 @@ mod tests { manifest_bytes } + fn pin_parent_model_catalog_head(data_dir: &Path, head_cid: &str) { + let path = data_dir.join("components.json"); + let mut manifest: serde_json::Value = + serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + manifest["model_catalog"] = serde_json::json!({ + "head_cid": head_cid, + "publisher_dids": ["did:key:z6Mkabcdefghijklmnopqrstuvwxyz0123456789ABCDE"] + }); + std::fs::write(&path, serde_json::to_vec_pretty(&manifest).unwrap()).unwrap(); + } + + fn write_parent_signed_model_catalog(data_dir: &Path, catalog: &[u8]) { + std::fs::write(data_dir.join("model-catalog.json"), catalog).unwrap(); + pin_parent_model_catalog_head(data_dir, &crate::setup::catalog_head_cid(catalog).unwrap()); + } + #[test] #[cfg(unix)] fn gateway_owned_terminal_child_allows_parent_gateway_subordinate_runtime() { diff --git a/elastos/crates/elastos-server/src/setup.rs b/elastos/crates/elastos-server/src/setup.rs index 09c419cfc..3e461a41f 100644 --- a/elastos/crates/elastos-server/src/setup.rs +++ b/elastos/crates/elastos-server/src/setup.rs @@ -256,7 +256,7 @@ async fn run_with_data_dir( if media_tools_dir.is_some() && !prerequisites_only { anyhow::bail!("--media-tools-dir requires --prerequisites-only"); } - let manifest = load_manifest()?; + let (manifest_path, manifest) = load_manifest_with_path()?; let platform = detect_platform(); eprintln!( @@ -525,6 +525,7 @@ async fn run_with_data_dir( } let stamped = write_installed_manifest(&data_dir, &manifest, &platform)?; + install_signed_model_catalog(&data_dir, &manifest, &manifest_path)?; println!(); if !stamped.is_empty() { @@ -545,9 +546,15 @@ async fn run_with_data_dir( const COMPONENTS_MANIFEST_ENV: &str = "ELASTOS_COMPONENTS_MANIFEST"; +#[cfg(test)] fn load_manifest() -> anyhow::Result { + Ok(load_manifest_with_path()?.1) +} + +fn load_manifest_with_path() -> anyhow::Result<(PathBuf, ComponentsManifest)> { if let Some(path) = explicit_components_manifest_path() { - return load_manifest_from_path(&path); + let manifest = load_manifest_from_path(&path)?; + return Ok((path, manifest)); } let exe_path = std::env::current_exe().ok(); @@ -565,7 +572,7 @@ fn load_manifest() -> anyhow::Result { for path in manifest_paths.iter().flatten() { if let Ok(content) = fs::read_to_string(path) { - return parse_manifest_at(path, &content); + return parse_manifest_at(path, &content).map(|manifest| (path.clone(), manifest)); } } @@ -1910,6 +1917,75 @@ pub fn write_installed_manifest( Ok(stamped) } +const MODEL_CATALOG_FILE: &str = "model-catalog.json"; +const MAX_MODEL_CATALOG_BYTES: usize = 128 * 1024; + +pub(crate) fn catalog_head_cid(bytes: &[u8]) -> anyhow::Result { + let hash = cid::multihash::Multihash::<64>::wrap(0x12, &sha2::Sha256::digest(bytes)) + .map_err(|err| anyhow::anyhow!("model catalog digest is not a SHA-256 multihash: {err}"))?; + Ok(cid::Cid::new_v1(0x55, hash).to_string()) +} + +fn verify_catalog_head(head_cid: &str, bytes: &[u8]) -> anyhow::Result<()> { + let actual = catalog_head_cid(bytes)?; + if actual != head_cid { + anyhow::bail!( + "model catalog head {actual} does not match the pinned raw SHA-256 CIDv1 {head_cid}" + ); + } + Ok(()) +} + +fn resolve_model_catalog_source(manifest_path: &Path, dest: &Path) -> anyhow::Result { + let sibling = manifest_path + .parent() + .ok_or_else(|| anyhow::anyhow!("components manifest has no parent directory"))? + .join(MODEL_CATALOG_FILE); + if sibling.is_file() { + return Ok(sibling); + } + if dest.is_file() { + return Ok(dest.to_path_buf()); + } + anyhow::bail!( + "model catalog pin is present but {MODEL_CATALOG_FILE} is missing beside {} and in the data directory", + manifest_path.display() + ); +} + +pub(crate) fn install_signed_model_catalog( + data_dir: &Path, + manifest: &ComponentsManifest, + manifest_path: &Path, +) -> anyhow::Result<()> { + let Some(trust) = &manifest.model_catalog else { + return Ok(()); + }; + if trust.head_cid.is_empty() || trust.head_cid.len() > 128 { + anyhow::bail!("model catalog trust head is invalid"); + } + let dest = data_dir.join(MODEL_CATALOG_FILE); + let source = resolve_model_catalog_source(manifest_path, &dest)?; + let bytes = fs::read(&source)?; + if bytes.len() > MAX_MODEL_CATALOG_BYTES { + anyhow::bail!( + "model catalog exceeds its {}-byte bound", + MAX_MODEL_CATALOG_BYTES + ); + } + verify_catalog_head(&trust.head_cid, &bytes)?; + if source != dest { + atomic_write_file(&dest, &bytes)?; + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + fs::set_permissions(&dest, fs::Permissions::from_mode(0o600))?; + } + println!("Installed signed model catalog: {}", dest.display()); + Ok(()) +} + pub fn write_installed_manifest_bytes( data_dir: &Path, manifest_bytes: &[u8], @@ -2942,13 +3018,40 @@ fn copy_dir_recursive(src: &Path, dest: &Path) -> anyhow::Result<()> { let entry = entry?; let path = entry.path(); let target = dest.join(entry.file_name()); - if path.is_dir() { + let metadata = fs::symlink_metadata(&path)?; + if metadata.file_type().is_dir() { copy_dir_recursive(&path, &target)?; - } else { + } else if metadata.file_type().is_symlink() { + let link_target = fs::read_link(&path)?; + if link_target.is_absolute() + || link_target.components().any(|part| { + matches!( + part, + std::path::Component::ParentDir + | std::path::Component::RootDir + | std::path::Component::Prefix(_) + ) + }) + { + anyhow::bail!("bundle symlink target is unsafe"); + } + #[cfg(unix)] + { + use std::os::unix::fs::symlink; + if let Some(parent) = target.parent() { + fs::create_dir_all(parent)?; + } + symlink(&link_target, &target)?; + } + #[cfg(not(unix))] + anyhow::bail!("bundle symlinks are unsupported on this platform"); + } else if metadata.file_type().is_file() { if let Some(parent) = target.parent() { fs::create_dir_all(parent)?; } fs::copy(&path, &target)?; + } else { + anyhow::bail!("bundle contains a special file"); } } Ok(()) @@ -3004,6 +3107,50 @@ mod tests { assert!(verify_checksum("home-cli", &corrupt, &info).is_err()); } + #[cfg(unix)] + #[test] + fn extract_from_tarball_preserves_relative_symlinks() { + use std::os::unix::fs::symlink; + + let temp = tempfile::tempdir().unwrap(); + let src = temp.path().join("llama-b10516"); + fs::create_dir_all(&src).unwrap(); + fs::write(src.join("lib.0.dylib"), b"dylib\n").unwrap(); + symlink("lib.0.dylib", src.join("lib.dylib")).unwrap(); + let archive = temp.path().join("bundle.tar.gz"); + let status = Command::new("tar") + .args([ + "czf", + archive.to_str().unwrap(), + "-C", + temp.path().to_str().unwrap(), + "llama-b10516", + ]) + .status() + .unwrap(); + assert!(status.success()); + let bytes = fs::read(&archive).unwrap(); + let info: PlatformInfo = serde_json::from_value(serde_json::json!({ + "release_path": "llama-b10516.tar.gz", + "install_path": "libexec/llama.cpp/b10516/darwin-arm64", + "extract_path": "llama-b10516", + "checksum": format!("sha256:{}", hex::encode(sha2::Sha256::digest(&bytes))) + })) + .unwrap(); + let dest = temp.path().join("libexec/llama.cpp/b10516/darwin-arm64"); + extract_from_tarball(&bytes, &dest, &info).unwrap(); + let link = dest.join("lib.dylib"); + assert!( + fs::symlink_metadata(&link) + .unwrap() + .file_type() + .is_symlink(), + "installer must keep the signed dylib symlink" + ); + assert_eq!(fs::read_link(&link).unwrap(), Path::new("lib.0.dylib")); + assert_eq!(fs::read(dest.join("lib.0.dylib")).unwrap(), b"dylib\n"); + } + #[test] fn test_detect_platform() { let p = detect_platform(); @@ -3350,6 +3497,37 @@ mod tests { assert!(manifest.external.contains_key("kubo")); assert!(manifest.external.contains_key("archive-manager")); + // Every prebuilt local model engine installs as a receipted bundle. The + // release executables load their libraries from their own directory, so + // a single copied executable would neither verify nor run. + let engine = &manifest.external["llama-server"]; + let version = engine.version.as_deref().unwrap(); + let mut prebuilt = 0; + for (platform, info) in &engine.platforms { + if info.url.is_none() { + continue; + } + prebuilt += 1; + assert_eq!( + info.binary_path.as_deref(), + Some("llama-server"), + "{platform}" + ); + assert_eq!( + info.extract_path.as_deref(), + Some(format!("llama-{version}").as_str()), + "{platform}" + ); + assert_eq!( + info.install_path.as_deref(), + Some(format!("libexec/llama.cpp/{version}/{platform}").as_str()), + "{platform}" + ); + } + assert!( + prebuilt >= 2, + "darwin-arm64 and linux-amd64 engines are prebuilt" + ); for provider in [ "net-provider", "exit-provider", @@ -3372,6 +3550,30 @@ mod tests { .components .iter() .any(|component| component == "archive-manager")); + for name in ["model-provider", "llama-server"] { + for profile_name in ["home", "demo"] { + assert!( + manifest.profiles[profile_name] + .components + .iter() + .any(|component| component == name), + "{profile_name} profile must install {name}" + ); + } + } + assert!( + !manifest.profiles["home"] + .components + .iter() + .any(|component| component.starts_with("model-qwen")), + "home profile must not install huggingface GGUF components" + ); + let catalog = manifest + .model_catalog + .as_ref() + .expect("home matching install pins a signed model catalog"); + assert!(!catalog.publisher_dids.is_empty()); + assert!(catalog.local_use.is_some()); for profile_name in ["home", "demo", "agent-local-ai", "public-gateway", "full"] { let profile = manifest .profiles @@ -5285,4 +5487,85 @@ mod tests { b"new-binary-with-more-bytes" ); } + + #[test] + fn install_signed_model_catalog_copies_sibling_snapshot() { + let tmp = tempfile::tempdir().unwrap(); + let source_dir = tmp.path().join("source"); + let data_dir = tmp.path().join("data"); + fs::create_dir_all(&source_dir).unwrap(); + fs::create_dir_all(&data_dir).unwrap(); + let catalog = br#"{"payload":{"schema":"elastos.model.catalog/v1"},"signature":"ab","signer_did":"did:key:z"}"#; + fs::write(source_dir.join(MODEL_CATALOG_FILE), catalog).unwrap(); + let head = catalog_head_cid(catalog).unwrap(); + let manifest: ComponentsManifest = serde_json::from_value(serde_json::json!({ + "external": {}, + "profiles": {}, + "model_catalog": { + "head_cid": head, + "publisher_dids": ["did:key:z6Mkabcdefghijklmnopqrstuvwxyz0123456789ABCDE"] + } + })) + .unwrap(); + fs::write( + source_dir.join("components.json"), + serde_json::to_vec(&manifest).unwrap(), + ) + .unwrap(); + install_signed_model_catalog(&data_dir, &manifest, &source_dir.join("components.json")) + .unwrap(); + assert_eq!( + fs::read(data_dir.join(MODEL_CATALOG_FILE)).unwrap(), + catalog + ); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + fs::metadata(data_dir.join(MODEL_CATALOG_FILE)) + .unwrap() + .permissions() + .mode() + & 0o777, + 0o600 + ); + } + } + + #[test] + fn install_signed_model_catalog_skips_when_unpinned() { + let tmp = tempfile::tempdir().unwrap(); + let data_dir = tmp.path(); + let manifest: ComponentsManifest = serde_json::from_value(serde_json::json!({ + "external": {}, + "profiles": {} + })) + .unwrap(); + install_signed_model_catalog(data_dir, &manifest, &data_dir.join("components.json")) + .unwrap(); + assert!(!data_dir.join(MODEL_CATALOG_FILE).exists()); + } + + #[test] + fn install_signed_model_catalog_rejects_head_mismatch() { + let tmp = tempfile::tempdir().unwrap(); + let source_dir = tmp.path().join("source"); + let data_dir = tmp.path().join("data"); + fs::create_dir_all(&source_dir).unwrap(); + fs::create_dir_all(&data_dir).unwrap(); + fs::write(source_dir.join(MODEL_CATALOG_FILE), b"catalog-a").unwrap(); + let manifest: ComponentsManifest = serde_json::from_value(serde_json::json!({ + "external": {}, + "profiles": {}, + "model_catalog": { + "head_cid": catalog_head_cid(b"catalog-b").unwrap(), + "publisher_dids": ["did:key:z6Mkabcdefghijklmnopqrstuvwxyz0123456789ABCDE"] + } + })) + .unwrap(); + let err = + install_signed_model_catalog(&data_dir, &manifest, &source_dir.join("components.json")) + .unwrap_err(); + assert!(err.to_string().contains("does not match the pinned")); + } } diff --git a/model-catalog.json b/model-catalog.json new file mode 100644 index 000000000..fd031950d --- /dev/null +++ b/model-catalog.json @@ -0,0 +1 @@ +{"payload":{"entries":[{"capsule_manifest":{"entrypoint":"weights.gguf","model_content":{"consumer_interface":"elastos.provider.model","consumer_interface_version":"0.1.0","engine":"llama.cpp","format":"gguf","license":{"path":"LICENSE","spdx_id":"Apache-2.0"},"minimum_memory_mb":8192,"provenance":{"base_license":{"path":"LICENSE.base","spdx_id":"Apache-2.0"},"base_repository":"Qwen/Qwen3.5-9B","base_revision":"c202236235762e1c871ad0ccb60c8ee5ba337b9a","path":"PROVENANCE.md","quantized_repository":"bartowski/Qwen_Qwen3.5-9B-GGUF","quantized_revision":"bc658f7a1ae222ee901a565aafd2f5b8c839e176"},"quantization":"Q4_K_M"},"name":"qwen3-5-9b-q4-k-m-local","projections":["content"],"role":"content","schema":"elastos.capsule/v1","type":"data","version":"0.1.0"},"cid":"bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi","object_manifest":{"content_digest":"sha256:93a66d041debbef5daad88c5673666e38187ee64ec136d51b526ab34cdcce212","files":[{"path":"LICENSE","sha256":"bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a","size":11544},{"path":"LICENSE.base","sha256":"bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a","size":11544},{"path":"PROVENANCE.md","sha256":"e7f5d1b457faa5c42070106cea7c0de2f67aedddaae7bb848b2de4658b8ad57d","size":594},{"path":"capsule.json","sha256":"f95e193acd689473e36e9e962afaf57c3dd564815904ef39f29359c6ccdf521d","size":721},{"path":"weights.gguf","sha256":"d784ce9eda1a5a7b51e8f705a9e6310844bf4f173654d115823c775fdea56d43","size":6169341984}],"kind":"capsule","schema":"elastos.content.object.manifest/v1"}},{"capsule_manifest":{"entrypoint":"weights.gguf","model_content":{"consumer_interface":"elastos.provider.model","consumer_interface_version":"0.1.0","engine":"llama.cpp","format":"gguf","license":{"path":"LICENSE","spdx_id":"Apache-2.0"},"minimum_memory_mb":512,"provenance":{"base_license":{"path":"LICENSE.base","spdx_id":"Apache-2.0"},"base_repository":"HuggingFaceTB/SmolLM2-135M-Instruct","base_revision":"12fd25f77366fa6b3b4b768ec3050bf629380bac","path":"PROVENANCE.md","quantized_repository":"unsloth/SmolLM2-135M-Instruct-GGUF","quantized_revision":"9e6855bc4be717fca1ef21360a1db4b29d5c559a"},"quantization":"Q8_0"},"name":"smollm2-135m-instruct-q8-0-local","projections":["content"],"role":"content","schema":"elastos.capsule/v1","type":"data","version":"0.1.0"},"cid":"bafybeidy5kfvqwg6g6pfgdfwslmhijosbeskt5b2duqdqxnc7e6fwmr72y","object_manifest":{"content_digest":"sha256:9599f2dcce56595e67c50c5f4434e04f14800504b45f78b2d7e895dc183cd619","files":[{"path":"LICENSE","sha256":"bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a","size":11544},{"path":"LICENSE.base","sha256":"bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a","size":11544},{"path":"PROVENANCE.md","sha256":"df5a337b439c3511487a4849ff658ecf65671e72a81a375e441ae200f535efcd","size":537},{"path":"capsule.json","sha256":"8f916fbe895beaa561631624f94f7db90a1ce5816cc1a193070629dc3e01d629","size":751},{"path":"weights.gguf","sha256":"c4a3dd037301b6ecea31d6da37f5cd793ead920dd5ddfe6d589294628d6ce66a","size":144811072}],"kind":"capsule","schema":"elastos.content.object.manifest/v1"}}],"published_at":1789592973,"schema":"elastos.model.catalog/v1"},"signature":"2d3177d67b83c36b2afb282728cc7ea76b3a571c80c5bc853975c879e8e9965c7cf973ba4c7bda26f746cb07f1590c9ee01d66547fa72a3affdadf2ae12bc504","signer_did":"did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA"} \ No newline at end of file diff --git a/scripts/assistant-shell-smoke.mjs b/scripts/assistant-shell-smoke.mjs index 4567fb44e..a85e30b8f 100644 --- a/scripts/assistant-shell-smoke.mjs +++ b/scripts/assistant-shell-smoke.mjs @@ -28,7 +28,8 @@ const source = `${originalSource .replace( '"./vendor/katex/katex.mjs"', JSON.stringify(katexModuleUrl), - ).replace('"./model-selection.js"', JSON.stringify(new URL("../capsules/_shared/model-selection.js", import.meta.url).href))} + ).replace('"./model-selection.js"', JSON.stringify(new URL("../capsules/_shared/model-selection.js", import.meta.url).href)) + .replace('"./model-contract.js"', JSON.stringify(new URL("../capsules/assistant/browser/model-contract.js", import.meta.url).href))} export { renderMarkdown, renderMessageBody, readHashParam, readQueryParam, readLaunchContext }; `; const assistantModule = await import( @@ -413,7 +414,7 @@ async function buildApp(options = {}) { assert.equal(await app.sendDraft(), false); assert.equal(app.openModels(), true); assert.deepEqual(posts.at(-1), { origin: "https://home.example", message: { - type: "home:open-target", homeToken: "token-1", target: "system", query: { settings: "models" }, + type: "home:open-target", homeToken: "token-1", target: "marketplace", query: { category: "models", model_cid: cid }, } }); assert.equal(app.snapshot().draft, "Keep exact text"); failed = false; @@ -1067,6 +1068,25 @@ for (const first of ["offers", "workspace"]) { ], }, }, + { + status: "ok", + data: { + schema: "elastos.model.run-events/v1", + run_id: + "run:sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + next_cursor: 2, + has_more: false, + events: [ + { + schema: "elastos.model.run-event/v1", + sequence: 2, + kind: "output", + terminal: true, + data: { schema: "elastos.model.output.text/v1", text: "Hello " }, + }, + ], + }, + }, ], }); app.setDraft("Do not duplicate text."); @@ -1078,10 +1098,10 @@ for (const first of ["offers", "workspace"]) { .currentSession.messages.filter((message) => message.role === "assistant"); assert.equal(assistantMessages.length, 1); assert.equal(assistantMessages[0].content, "Hello "); - assert.equal(app.snapshot().statusMessage, "Model provider unavailable."); + assert.equal(app.snapshot().statusMessage, ""); assert.equal( fetch.fetchCalls.filter(([url]) => url === "/api/provider/model/runs_events").length, - 2, + 3, ); assert.equal(pendingTimerCount(), 0); } diff --git a/scripts/assistant-studio-preservation.test.mjs b/scripts/assistant-studio-preservation.test.mjs index 3026d1bf8..d858517ff 100644 --- a/scripts/assistant-studio-preservation.test.mjs +++ b/scripts/assistant-studio-preservation.test.mjs @@ -10,7 +10,8 @@ const asset = path => new URL(`../capsules/${path}`, import.meta.url).href; const source = readFileSync(new URL("../capsules/assistant/browser/assistant.js", import.meta.url), "utf8") .replace('"/apps/home/home-clipboard-client.js?v=home-20260726a"', JSON.stringify(asset("home/browser/home-clipboard-client.js"))) .replace('"./vendor/katex/katex.mjs"', JSON.stringify(asset("assistant/browser/vendor/katex/katex.mjs"))) - .replace('"./model-selection.js"', JSON.stringify(asset("_shared/model-selection.js"))); + .replace('"./model-selection.js"', JSON.stringify(asset("_shared/model-selection.js"))) + .replace('"./model-contract.js"', JSON.stringify(asset("assistant/browser/model-contract.js"))); const { createAssistantApp } = await import(`data:text/javascript,${encodeURIComponent(source)}`); const clone = value => structuredClone(value); const runId = `run:sha256:${"a".repeat(64)}`; diff --git a/scripts/home-agent-cancellation.test.mjs b/scripts/home-agent-cancellation.test.mjs index 5554b495f..d91c4bff4 100644 --- a/scripts/home-agent-cancellation.test.mjs +++ b/scripts/home-agent-cancellation.test.mjs @@ -123,7 +123,7 @@ test("nested Agent Models intent registers top Home with exact origin and leaves assert.equal(openModelsFromAgent(), true); assert.deepEqual(messages, [ { origin: "https://home.example", message: { type: "home:app-ready", homeToken: "fixture" } }, - { origin: "https://home.example", message: { type: "home:open-target", homeToken: "fixture", target: "system", query: { settings: "models" } } }, + { origin: "https://home.example", message: { type: "home:open-target", homeToken: "fixture", target: "marketplace", query: { category: "models" } } }, ]); window.location.href = "https://home.example/apps/home-agent/?home_origin=null"; assert.equal(openModelsFromAgent(), false); diff --git a/scripts/home-agent-shell-smoke.mjs b/scripts/home-agent-shell-smoke.mjs index 28fd9aa6d..d2fcacec9 100644 --- a/scripts/home-agent-shell-smoke.mjs +++ b/scripts/home-agent-shell-smoke.mjs @@ -63,6 +63,10 @@ for (const gone of ["mock-agent-provider.js", "agent-grants.js", "agent-studio.j } assert.ok(agentLive.includes('from "./model-contract.js"'), "agent-live speaks the typed contract module"); +assert.ok( + readFileSync(new URL("assistant.js", capsuleDir), "utf8").includes('from "./model-contract.js"'), + "Assistant Chat and Studio decode events through the typed contract module", +); assert.ok(agentLive.includes("after_sequence: afterSequence"), "runs_events is polled by after_sequence"); assert.ok(agentLive.includes("textRunCreateBody({ offer, messages, requestId: createRequestId })")); assert.ok(agentLive.includes('modelRunCall("runs_cancel", { run_id: runId, request_id: newRequestId() })')); @@ -371,7 +375,13 @@ const page1 = contract.applyRunEventsPage( }, 0, ); -assert.deepEqual(page1, { nextCursor: 3, hasMore: true, textDeltas: ["Hel", "lo"], terminal: null }); +assert.deepEqual(page1, { + nextCursor: 3, + hasMore: true, + textDeltas: ["Hel", "lo"], + terminal: null, + studioProgress: null, +}); const page2 = contract.applyRunEventsPage( { @@ -408,16 +418,62 @@ assert.deepEqual(failed.terminal, { status: "failed", output: null, error: { cod assert.deepEqual( contract.applyRunEventsPage({ events: [], next_cursor: 5 }, 5), - { nextCursor: 5, hasMore: false, textDeltas: [], terminal: null }, + { nextCursor: 5, hasMore: false, textDeltas: [], terminal: null, studioProgress: null }, ); assert.throws(() => contract.applyRunEventsPage({ events: [], next_cursor: 2 }, 5), /cursor went backwards/); +assert.deepEqual( + contract.applyRunEventsPage( + { + events: [ + { sequence: 2, kind: "text_delta", data: { text: "a" } }, + { sequence: 2, kind: "text_delta", data: { text: "b" } }, + ], + next_cursor: 2, + }, + 0, + ), + { nextCursor: 2, hasMore: false, textDeltas: ["a"], terminal: null, studioProgress: null }, +); +assert.deepEqual( + contract.applyRunEventsPage( + { events: [{ sequence: 1, kind: "text_delta", data: { text: "a" } }], next_cursor: 1 }, + 1, + ), + { nextCursor: 1, hasMore: false, textDeltas: [], terminal: null, studioProgress: null }, +); +const replayedProgress = contract.applyRunEventsPage( + { + events: [ + { sequence: 1, kind: "progress", data: { phase: "rendering", completed: 1, total: 4 } }, + { sequence: 1, kind: "progress", data: { phase: "rendering", completed: 1, total: 4 } }, + { sequence: 2, kind: "progress", data: { phase: "rendering", completed: 2, total: 4 } }, + ], + next_cursor: 2, + }, + 0, +); +assert.deepEqual(replayedProgress.studioProgress, { phase: "rendering", completed: 2, total: 4 }); assert.throws( - () => contract.applyRunEventsPage({ events: [{ sequence: 2, kind: "text_delta", data: { text: "a" } }, { sequence: 2, kind: "text_delta", data: { text: "b" } }], next_cursor: 2 }, 0), - /strictly increasing/, + () => + contract.applyRunEventsPage( + { events: [{ sequence: 1, kind: "progress", data: { phase: "rendering", completed: 3, total: 2 } }], next_cursor: 1 }, + 0, + ), + /studio progress/, ); assert.throws( - () => contract.applyRunEventsPage({ events: [{ sequence: 1, kind: "text_delta", data: { text: "a" } }], next_cursor: 1 }, 1), - /strictly increasing/, + () => + contract.applyRunEventsPage( + { + events: [ + { sequence: 2, kind: "text_delta", data: { text: "B" } }, + { sequence: 1, kind: "text_delta", data: { text: "A" } }, + ], + next_cursor: 2, + }, + 0, + ), + /out of order/, ); assert.throws( () => contract.applyRunEventsPage({ events: [{ sequence: 7, kind: "text_delta", data: { text: "a" } }], next_cursor: 6 }, 0), diff --git a/scripts/marketplace-product-behavior-smoke.mjs b/scripts/marketplace-product-behavior-smoke.mjs index b4787de9d..73cf14886 100644 --- a/scripts/marketplace-product-behavior-smoke.mjs +++ b/scripts/marketplace-product-behavior-smoke.mjs @@ -58,6 +58,9 @@ assert( assert( js.includes('if (event.origin !== "null" || event.source !== window.parent) {') && js.includes('if (data?.type !== "elastos:menu-command" || typeof data.cmd !== "string") {') + && js.includes('data?.type === "elastos.marketplace.navigate/v1"') + && js.includes('target: "assistant"') + && js.includes("function applyModelsLaunch") && !js.includes("window.location.origin") && !js.includes('event.source !== window.top'), "Marketplace must accept inbound Home commands only from the opaque parent boundary.", diff --git a/scripts/model-package-handoff-test.py b/scripts/model-package-handoff-test.py new file mode 100755 index 000000000..3784b1fa1 --- /dev/null +++ b/scripts/model-package-handoff-test.py @@ -0,0 +1,496 @@ +#!/usr/bin/env python3 +"""Behavior tests for scripts/model-package-handoff.py against pinned Kubo 0.40.1; stdlib only. + +ELASTOS_TEST_KUBO_PATH=/path/to/kubo python3 scripts/model-package-handoff-test.py +""" + +import contextlib +import hashlib +import http.client +import http.server +import importlib.util +import io +import json +import os +from pathlib import Path +import socket +import stat +import subprocess +import tempfile +import threading +import time +import unittest +from unittest import mock + +HERE = Path(__file__).resolve().parent + + +def load_sibling(name): + spec = importlib.util.spec_from_file_location(name.replace("-", "_"), HERE / f"{name}.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +helper = load_sibling("model-package-handoff") +bootstrap = load_sibling("install-bootstrap-test") + +KUBO = os.environ.get("ELASTOS_TEST_KUBO_PATH") +KUBO_SKIP = "requires explicit pinned ELASTOS_TEST_KUBO_PATH" +OTHER_DID = bootstrap.PUBLISHER_DID +CACHE_BUDGET = 7 * 2**30 +MEMORY_BUDGET = 5 * 2**30 + + +def run_helper(*argv): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + try: + code = helper.main([str(arg) for arg in argv]) + except SystemExit as exit: + code = exit.code + return code, out.getvalue(), err.getvalue() + + +def sha256_file(path): + digest = hashlib.sha256() + with open(path, "rb") as source: + for chunk in iter(lambda: source.read(65536), b""): + digest.update(chunk) + return digest.hexdigest() + + +def unbound_port(): + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + return probe.getsockname()[1] + + +def synthetic_closure(root): + files = { + "capsule.json": b'{"name":"synthetic-model","role":"content","type":"data"}\n', + "LICENSE": b"Apache-2.0 (synthetic quantized model)\n", + "LICENSE.base": b"Apache-2.0 (synthetic base model)\n", + "PROVENANCE.md": b"# Provenance\n\nSynthetic bytes for the handoff test.\n", + "weights.gguf": b"GGUF\x03\x00\x00\x00" + bytes((index * 7919 + 13) % 256 for index in range(600 * 1024)), + } + for name, data in files.items(): + (root / name).write_bytes(data) + return sum(map(len, files.values())) + + +def signed_catalog(package_cid, published_at=1, expires_at=None): + payload = { + "schema": "elastos.model.catalog/v1", + "published_at": published_at, + "entries": [{ + "cid": package_cid, + "capsule_manifest": {"name": "synthetic-model", "role": "content", "type": "data"}, + "object_manifest": {"files": ["weights.gguf"], "publisher_did": None}, + }], + } + if expires_at is not None: + payload["expires_at"] = expires_at + signed = bootstrap.sign_envelope(payload, "elastos.model.catalog.v1") + return bootstrap.encode_envelope(signed), signed["signer_did"] + + +def fake_routes(**overrides): + routes = { + "/api/v0/version": (200, b'{"Version":"0.40.1"}\n'), + "/api/v0/config?arg=Import.UnixFSHAMTDirectorySizeThreshold": + (200, b'{"Key":"Import.UnixFSHAMTDirectorySizeThreshold","Value":null}\n'), + "/api/v0/config?arg=Import.UnixFSHAMTDirectorySizeEstimation": + (200, b'{"Key":"Import.UnixFSHAMTDirectorySizeEstimation","Value":null}\n'), + } + routes.update(overrides) + return routes + + +class FakeKuboHandler(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def do_POST(self): + answer = self.server.routes.get(self.path) + if callable(answer): + answer(self) + return + status, body = answer or (404, b'{"Message":"no such route","Type":"error"}\n') + self.send_response(status) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *_): + pass + + +@contextlib.contextmanager +def fake_kubo(routes): + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), FakeKuboHandler) + server.daemon_threads = True + server.routes = routes + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield f"http://127.0.0.1:{server.server_port}" + finally: + server.shutdown() + server.server_close() + thread.join() + + +class KuboDaemon: + def __init__(self, repo): + self.repo = Path(repo) + self.env = {**os.environ, "IPFS_PATH": str(self.repo)} + self.process = None + self.port = None + self.api_url = None + + def run(self, *args): + completed = subprocess.run([KUBO, *map(str, args)], env=self.env, check=True, capture_output=True, text=True) + return completed.stdout.strip() + + def start(self): + self.repo.mkdir() + self.run("init", "--empty-repo", "--profile=test") + self.run("config", "Addresses.API", "/ip4/127.0.0.1/tcp/0") + self.run("config", "Addresses.Gateway", "/ip4/127.0.0.1/tcp/0") + self.run("config", "--json", "Addresses.Swarm", "[]") + log_path = self.repo.parent / f"{self.repo.name}.log" + with open(log_path, "wb") as log: + self.process = subprocess.Popen([KUBO, "daemon", "--offline"], env=self.env, stdout=log, stderr=subprocess.STDOUT) + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + if self.process.poll() is not None: + raise RuntimeError(f"kubo daemon exited early:\n{log_path.read_text()}") + api_file = self.repo / "api" + if api_file.exists() and self._answers(int(api_file.read_text().strip().rsplit("/", 1)[1])): + return self + time.sleep(0.05) + raise RuntimeError(f"kubo daemon did not become ready:\n{log_path.read_text()}") + + def _answers(self, port): + connection = http.client.HTTPConnection("127.0.0.1", port, timeout=2) + try: + connection.request("POST", "/api/v0/version") + if connection.getresponse().status != 200: + return False + except OSError: + return False + finally: + connection.close() + self.port, self.api_url = port, f"http://127.0.0.1:{port}" + return True + + def stop(self): + if self.process is None or self.process.poll() is not None: + return + self.process.terminate() + try: + self.process.wait(timeout=15) + except subprocess.TimeoutExpired: + self.process.kill() + self.process.wait() + + +class OfflineTest(unittest.TestCase): + def test_head_cid_matches_recorded_kubo_vector(self): + # Recorded from `kubo add --cid-version=1 --raw-leaves -Q -n` on Kubo 0.40.1 for these exact bytes. + self.assertEqual(helper.raw_cid(b'{"a":1}\n'), "bafkreihdizbsainqif4vddmwctzvmdgnoe2uutxbaho4xcj5nfm2tvrqdq") + + +@unittest.skipUnless(KUBO and os.access(KUBO, os.X_OK), KUBO_SKIP) +class KuboHandoffTest(unittest.TestCase): + def setUp(self): + scratch = tempfile.TemporaryDirectory(prefix="model-package-handoff-") + self.addCleanup(scratch.cleanup) + self.root = Path(scratch.name) + self.sender = KuboDaemon(self.root / "sender").start() + self.addCleanup(self.sender.stop) + self.receiver = KuboDaemon(self.root / "receiver").start() + self.addCleanup(self.receiver.stop) + self.closure = self.root / "closure" + self.closure.mkdir() + self.payload_bytes = synthetic_closure(self.closure) + self.cid = self.sender.run("add", "-r", "--cid-version=1", "--raw-leaves", "--chunker=size-262144", "-Q", self.closure) + self.assertTrue(self.cid.startswith("bafybei"), self.cid) + self.other_cid = self.cid[:-1] + ("a" if self.cid[-1] != "a" else "b") + + def export_package(self): + car = self.root / "package.car" + code, out, err = run_helper("export", "--cid", self.cid, "--output", car, "--kubo-api", self.sender.api_url) + self.assertEqual((code, err), (0, "")) + return car, self.root / "package.car.receipt.json", json.loads(out) + + def dag_stat(self, node): + return json.loads(node.run("dag", "stat", "--enc=json", "--progress=false", self.cid))["DagStats"][0] + + def recursive_pins(self, node): + return sorted(node.run("pin", "ls", "--type=recursive").splitlines()) + + def receiver_data_dir(self): + data_dir = self.root / "receiver-data" + data_dir.mkdir(mode=0o700) + components = { + "schema": "elastos.components/v1", + "capsules": {"shell": {"cid": "", "sha256": "", "size": 0, "platforms": ["x86_64-linux"]}}, + "external": { + "kubo": {"version": "0.40.1", "platforms": {"x86_64-linux": { + "url": "https://example.invalid/kubo.tar.gz", "checksum": "sha256:" + "a" * 64, "extract_path": "kubo/ipfs"}}}, + "llama-server": {"version": "b1", "platforms": {}}, + "carrier": {"version": "1", "platforms": {}}, + }, + "profiles": {"default": {"external": ["kubo"], "capsules": ["shell"]}}, + "model_catalog": {"head_cid": "bafkreiold", "publisher_dids": ["did:key:z6MkOld"]}, + } + (data_dir / "components.json").write_bytes(json.dumps(components, indent=2).encode() + b"\n") + return data_dir, components + + def pin_catalog(self, data_dir, catalog_path, head, did, package_cid=None): + return run_helper( + "pin-catalog", "--data-dir", data_dir, "--catalog", catalog_path, "--head-cid", head, + "--publisher-did", did, "--publisher-did", OTHER_DID, "--package-cid", package_cid or self.cid, + "--max-cache-bytes", CACHE_BUDGET, "--max-model-memory-bytes", MEMORY_BUDGET) + + @staticmethod + def snapshot(data_dir): + return {path.name: path.read_bytes() for path in data_dir.iterdir()} + + def test_verify_kubo_accepts_pinned_version_and_canonical_profile(self): + code, out, err = run_helper("verify-kubo", "--kubo-api", self.sender.api_url) + self.assertEqual((code, err), (0, "")) + self.assertEqual(json.loads(out), {"kubo_version": "0.40.1", "import_profile": "canonical"}) + + data_dir = self.root / "sender-data" + data_dir.mkdir() + coords = {"kubo_pid": self.sender.process.pid, "api_port": self.sender.port, "gateway_port": 0, "started_at": 0, "last_used": 0} + (data_dir / "ipfs-coords.json").write_text(json.dumps(coords)) + code, out, err = run_helper("verify-kubo", "--data-dir", data_dir) + self.assertEqual((code, err, json.loads(out)["kubo_version"]), (0, "", "0.40.1")) + + with fake_kubo(fake_routes(**{"/api/v0/version": (200, b'{"Version":"0.39.0"}\n')})) as url: + code, out, err = run_helper("verify-kubo", "--kubo-api", url) + self.assertEqual((code, out, err), (1, "", "error: kubo version '0.39.0' is not 0.40.1\n")) + + threshold = "/api/v0/config?arg=Import.UnixFSHAMTDirectorySizeThreshold" + with fake_kubo(fake_routes(**{threshold: (200, b'{"Key":"Import.UnixFSHAMTDirectorySizeThreshold","Value":"1MiB"}\n')})) as url: + code, out, err = run_helper("verify-kubo", "--kubo-api", url) + self.assertEqual((code, out), (1, "")) + self.assertIn("Import.UnixFSHAMTDirectorySizeThreshold='1MiB' is not canonical", err) + + def test_export_streams_car_with_receipt_and_verify_car_refuses_corruption(self): + car = self.root / "package.car" + receipt_path = self.root / "package.car.receipt.json" + reads = [] + original_read = helper.TrailerResponse.read + + def counting_read(response, amt=None): + reads.append(amt) + return original_read(response, amt) + + with mock.patch.object(helper, "CHUNK_BYTES", 4096), mock.patch.object(helper.TrailerResponse, "read", counting_read): + code, out, err = run_helper("export", "--cid", self.cid, "--output", car, "--kubo-api", self.sender.api_url) + self.assertEqual((code, err), (0, "")) + receipt = json.loads(out) + size = car.stat().st_size + self.assertEqual(receipt, { + "schema": "elastos.model.package-car/v1", "package_cid": self.cid, "car_sha256": sha256_file(car), + "car_bytes": size, "kubo_version": "0.40.1", "exported_at": receipt["exported_at"]}) + self.assertGreater(size, self.payload_bytes) + self.assertGreaterEqual(reads.count(4096), size // 4096, "the CAR body was not read in 4 KiB chunks") + self.assertEqual(json.loads(receipt_path.read_bytes()), receipt) + self.assertFalse((self.root / "package.car.partial").exists()) + code, out, err = run_helper("verify-car", "--car", car, "--receipt", receipt_path, "--cid", self.cid) + self.assertEqual((code, err, json.loads(out)), (0, "", receipt)) + + truncated = self.root / "truncated.car" + truncated.write_bytes(car.read_bytes()[:-1000]) + flipped_bytes = bytearray(car.read_bytes()) + flipped_bytes[size // 2] ^= 0xFF + flipped = self.root / "flipped.car" + flipped.write_bytes(flipped_bytes) + other_receipt = self.root / "other.receipt.json" + other_receipt.write_text(json.dumps({**receipt, "package_cid": self.other_cid})) + dead_kubo = f"http://127.0.0.1:{unbound_port()}" + cases = [ + ("truncated", truncated, receipt_path, f"CAR size {size - 1000} does not match receipt car_bytes {size}"), + ("flipped byte", flipped, receipt_path, "does not match receipt car_sha256"), + ("other receipt", car, other_receipt, f"receipt package_cid {self.other_cid} does not match expected CID {self.cid}"), + ] + for label, car_path, receipt_file, expected in cases: + with self.subTest(label): + code, out, err = run_helper("verify-car", "--car", car_path, "--receipt", receipt_file, "--cid", self.cid) + self.assertEqual((code, out), (1, "")) + self.assertIn(expected, err) + code, out, err = run_helper( + "import", "--car", car_path, "--receipt", receipt_file, "--cid", self.cid, + "--free-space-floor-bytes", 0, "--kubo-api", dead_kubo) + self.assertEqual((code, out), (1, "")) + self.assertIn(expected, err) + self.assertNotIn("unreachable", err, "import contacted Kubo before the CAR gate") + + def failing_export(handler): + handler.send_response(200) + handler.send_header("Transfer-Encoding", "chunked") + handler.send_header("Trailer", "X-Stream-Error") + handler.end_headers() + handler.wfile.write(b"5\r\nCARv1\r\n0\r\nX-Stream-Error: block bafkreimissing not found\r\n\r\n") + + with fake_kubo(fake_routes(**{f"/api/v0/dag/export?arg={self.cid}": failing_export})) as url: + code, out, err = run_helper("export", "--cid", self.cid, "--output", self.root / "failed.car", "--kubo-api", url) + self.assertEqual((code, out), (1, "")) + self.assertIn("failed mid-stream: block bafkreimissing not found", err) + self.assertEqual(sorted(path.name for path in self.root.glob("failed.car*")), []) + + def test_import_pins_complete_root_on_receiver_and_is_idempotent(self): + car, receipt_path, _ = self.export_package() + code, out, err = run_helper("verify-kubo", "--kubo-api", self.receiver.api_url) + self.assertEqual((code, err), (0, "")) + base = ["import", "--car", car, "--receipt", receipt_path, "--cid", self.cid, "--kubo-api", self.receiver.api_url] + + code, out, err = run_helper(*base, "--free-space-floor-bytes", 2**62) + self.assertEqual((code, out), (1, "")) + self.assertIn(f"leave less than the {2**62}-byte --free-space-floor-bytes floor", err) + self.assertNotIn(f"{self.cid} recursive", self.recursive_pins(self.receiver)) + + code, out, err = run_helper(*base, "--free-space-floor-bytes", 0) + self.assertEqual((code, err), (0, "")) + first = json.loads(out) + sender_stat = self.dag_stat(self.sender) + self.assertEqual(first, { + "schema": "elastos.model.package-import/v1", "package_cid": self.cid, "pinned": True, + "blocks": sender_stat["NumBlocks"], "block_bytes": sender_stat["Size"], "kubo_version": "0.40.1"}) + pins = self.recursive_pins(self.receiver) + self.assertIn(f"{self.cid} recursive", pins) + self.assertEqual(self.dag_stat(self.receiver), sender_stat) + fetched = self.root / "fetched" + self.receiver.run("get", "-o", fetched, self.cid) + self.assertEqual(sorted(path.name for path in fetched.iterdir()), sorted(path.name for path in self.closure.iterdir())) + for path in self.closure.iterdir(): + self.assertEqual((fetched / path.name).read_bytes(), path.read_bytes(), path.name) + + code, out, err = run_helper(*base, "--free-space-floor-bytes", 0) + self.assertEqual((code, err, json.loads(out)), (0, "", first)) + self.assertEqual(self.recursive_pins(self.receiver), pins) + + def test_pin_catalog_patches_only_model_catalog_and_refuses_timed_snapshots(self): + data_dir, components = self.receiver_data_dir() + catalog, did = signed_catalog(self.cid) + catalog_path = self.root / "model-catalog.json" + catalog_path.write_bytes(catalog) + head = helper.raw_cid(catalog) + self.assertEqual(head, self.sender.run("add", "--cid-version=1", "--raw-leaves", "-Q", "-n", catalog_path)) + code, out, err = run_helper("head-cid", "--catalog", catalog_path) + self.assertEqual((code, err, json.loads(out)), (0, "", {"head_cid": head, "catalog_bytes": len(catalog)})) + + code, out, err = self.pin_catalog(data_dir, catalog_path, head, did) + self.assertEqual((code, err), (0, "")) + expected_pin = {"head_cid": head, "publisher_dids": [did, OTHER_DID], + "local_use": {"max_cache_bytes": CACHE_BUDGET, "max_model_memory_bytes": MEMORY_BUDGET}} + self.assertEqual(json.loads(out), { + "schema": "elastos.model.catalog-pin/v1", "head_cid": head, "package_cid": self.cid, + **expected_pin, "signature_verification": "pending_runtime_read"}) + patched = json.loads((data_dir / "components.json").read_bytes()) + self.assertEqual(list(patched), list(components)) + self.assertEqual({key: value for key, value in patched.items() if key != "model_catalog"}, + {key: value for key, value in components.items() if key != "model_catalog"}) + self.assertEqual(patched["model_catalog"], expected_pin) + self.assertEqual((data_dir / "model-catalog.json").read_bytes(), catalog) + for name in ("model-catalog.json", "components.json"): + info = (data_dir / name).stat() + self.assertEqual((stat.S_IMODE(info.st_mode), info.st_nlink), (0o600, 1), name) + snapshot = self.snapshot(data_dir) + self.assertEqual(sorted(snapshot), ["components.json", "model-catalog.json"]) + + timed, _ = signed_catalog(self.cid, expires_at=4000000000) + timed_path = self.root / "timed-catalog.json" + timed_path.write_bytes(timed) + refusals = [ + ("timed snapshot", (timed_path, helper.raw_cid(timed), did, None), + "expires_at=4000000000 is a timed snapshot; this helper pins permanent snapshots only"), + ("wrong head", (catalog_path, helper.raw_cid(b"other bytes"), did, None), f"catalog head is {head}, not --head-cid"), + ("entry cid", (catalog_path, head, did, self.other_cid), f"does not match --package-cid {self.other_cid}"), + ("unknown signer", (catalog_path, head, "did:key:z6MkUnknownSigner", None), "is not among the given --publisher-did values"), + ] + for label, arguments, expected in refusals: + with self.subTest(label): + code, out, err = self.pin_catalog(data_dir, *arguments) + self.assertEqual((code, out), (1, "")) + self.assertIn(expected, err) + self.assertEqual(self.snapshot(data_dir), snapshot) + data_dir.chmod(0o770) + try: + code, out, err = self.pin_catalog(data_dir, catalog_path, head, did) + finally: + data_dir.chmod(0o700) + self.assertEqual((code, out), (1, "")) + self.assertIn("not writable by group or others", err) + self.assertEqual(self.snapshot(data_dir), snapshot) + + def test_two_pins_stay_separate(self): + car, receipt_path, _ = self.export_package() + code, out, err = run_helper( + "import", "--car", car, "--receipt", receipt_path, "--cid", self.cid, + "--free-space-floor-bytes", 0, "--kubo-api", self.receiver.api_url) + self.assertEqual((code, err), (0, "")) + data_dir, _ = self.receiver_data_dir() + components_path = data_dir / "components.json" + catalog, did = signed_catalog(self.cid) + catalog_path = self.root / "model-catalog.json" + catalog_path.write_bytes(catalog) + code, out, err = self.pin_catalog(data_dir, catalog_path, helper.raw_cid(catalog), did) + self.assertEqual((code, err), (0, "")) + pinned = json.loads(components_path.read_bytes())["model_catalog"] + + self.receiver.run("pin", "rm", self.cid) + pins = self.recursive_pins(self.receiver) + self.assertNotIn(f"{self.cid} recursive", pins) + self.assertEqual(json.loads(components_path.read_bytes())["model_catalog"], pinned) + + second, _ = signed_catalog(self.cid, published_at=2) + second_path = self.root / "second-catalog.json" + second_path.write_bytes(second) + code, out, err = self.pin_catalog(data_dir, second_path, helper.raw_cid(second), did) + self.assertEqual((code, err), (0, "")) + self.assertEqual(json.loads(components_path.read_bytes())["model_catalog"]["head_cid"], helper.raw_cid(second)) + self.assertNotEqual(helper.raw_cid(second), pinned["head_cid"]) + self.assertEqual(self.recursive_pins(self.receiver), pins) + + +class RepoCatalogPinTests(unittest.TestCase): + def test_repo_catalog_matches_components_pin(self): + root = HERE.parent + catalog = (root / "model-catalog.json").read_bytes() + components = json.loads((root / "components.json").read_bytes()) + pin = components["model_catalog"] + self.assertEqual(helper.raw_cid(catalog), pin["head_cid"]) + signed = json.loads(catalog) + self.assertNotIn("expires_at", signed["payload"]) + self.assertEqual(len(signed["payload"]["entries"]), 2) + self.assertEqual( + signed["payload"]["entries"][0]["cid"], + "bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi", + ) + self.assertEqual( + signed["payload"]["entries"][1]["cid"], + "bafybeidy5kfvqwg6g6pfgdfwslmhijosbeskt5b2duqdqxnc7e6fwmr72y", + ) + self.assertIn(signed["signer_did"], pin["publisher_dids"]) + helper.check_catalog( + catalog, + signed["payload"]["entries"][0]["cid"], + pin["publisher_dids"], + ) + helper.check_catalog( + catalog, + signed["payload"]["entries"][1]["cid"], + pin["publisher_dids"], + ) + self.assertGreaterEqual(pin["local_use"]["max_cache_bytes"], 18516684377) + self.assertGreaterEqual(pin["local_use"]["max_model_memory_bytes"], 8589934592) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/scripts/model-package-handoff.py b/scripts/model-package-handoff.py new file mode 100755 index 000000000..7ff7712e6 --- /dev/null +++ b/scripts/model-package-handoff.py @@ -0,0 +1,555 @@ +#!/usr/bin/env python3 +"""Hand a signed model content package from one Runtime-managed Kubo to another. + +`export` streams the package closure out of the sender's Kubo as a CAR with a +receipt. `verify-car` and `import` replay that closure into the receiver's Kubo +and confirm the recursive retention pin. `pin-catalog` installs the sender's +signed permanent catalog snapshot and points `components.json.model_catalog` +at it without verifying the signature; the Runtime verifies the catalog +signature when it reads the snapshot, so the receipt reports that as pending. +The retention pin and the catalog trust pin are separate and stay separate. +Stdlib only. Every command prints one JSON receipt on success and one +`error: ...` line on refusal. +""" + +import argparse +import base64 +import contextlib +import dataclasses +import hashlib +import http.client +import json +import os +import re +import shutil +import stat +import sys +import tempfile +import time +import urllib.parse + +KUBO_VERSION = "0.40.1" +CHUNK_BYTES = 1024 * 1024 +MAX_CONTROL_BYTES = 64 * 1024 +MAX_CATALOG_BYTES = 128 * 1024 +MAX_COMPONENTS_BYTES = 4 * 1024 * 1024 +MAX_BUDGET = 2**63 - 1 +CATALOG_SCHEMA = "elastos.model.catalog/v1" +RECEIPT_SCHEMA = "elastos.model.package-car/v1" +IMPORT_SCHEMA = "elastos.model.package-import/v1" +CATALOG_PIN_SCHEMA = "elastos.model.catalog-pin/v1" +BASE32_ALPHABET = frozenset("abcdefghijklmnopqrstuvwxyz234567") + +# Kubo 0.40.1 config/import.go treats null as exactly these defaults; the +# Runtime's directory_hash.rs accepts the same spellings and nothing else. +IMPORT_PROFILE = { + "Import.UnixFSHAMTDirectorySizeThreshold": lambda value: value is None + or (isinstance(value, int) and not isinstance(value, bool) and value == 262144) + or value in ("256KiB", "262144"), + "Import.UnixFSHAMTDirectorySizeEstimation": lambda value: value is None or value == "links", +} + + +class Refusal(Exception): + pass + + +@dataclasses.dataclass(frozen=True) +class CarDigest: + sha256: str + size: int + + +@dataclasses.dataclass(frozen=True) +class ImportStats: + blocks: int + block_bytes: int + + +@dataclasses.dataclass(frozen=True) +class Receipt: + package_cid: str + car_sha256: str + car_bytes: int + kubo_version: str + exported_at: int + + def to_json(self): + return {"schema": RECEIPT_SCHEMA, **dataclasses.asdict(self)} + + @classmethod + def load(cls, path): + try: + record = json.loads(read_bounded(path, MAX_CONTROL_BYTES)) + except ValueError: + raise Refusal(f"receipt {path} is not JSON") + fields = {field.name: field.type for field in dataclasses.fields(cls)} + if ( + not isinstance(record, dict) + or record.get("schema") != RECEIPT_SCHEMA + or set(record) != {"schema", *fields} + or any(not isinstance(record[name], kind) or isinstance(record[name], bool) for name, kind in fields.items()) + or record["car_bytes"] <= 0 + or len(record["car_sha256"]) != 64 + ): + raise Refusal(f"receipt {path} is not a {RECEIPT_SCHEMA} record") + return cls(**{name: record[name] for name in fields}) + + +@dataclasses.dataclass(frozen=True) +class CatalogPin: + head_cid: str + publisher_dids: tuple + max_cache_bytes: int + max_model_memory_bytes: int + + def to_json(self): + return { + "head_cid": self.head_cid, + "publisher_dids": list(self.publisher_dids), + "local_use": { + "max_cache_bytes": self.max_cache_bytes, + "max_model_memory_bytes": self.max_model_memory_bytes, + }, + } + + +class TrailerResponse(http.client.HTTPResponse): + """http.client discards chunked trailers; Kubo reports a mid-stream failure only in X-Stream-Error.""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.trailers = {} + + def _read_and_discard_trailer(self): + while True: + line = self.fp.readline(65537) + if len(line) > 65536: + raise http.client.LineTooLong("trailer line") + if not line or line in (b"\r\n", b"\n"): + return + name, _, value = line.decode("latin-1").partition(":") + self.trailers[name.strip()] = value.strip() + + +class KuboApi: + def __init__(self, base_url, timeout): + match = re.fullmatch(r"http://([^/:]+):([1-9][0-9]{0,4})/?", base_url) + if not match: + raise Refusal(f"--kubo-api must look like http://127.0.0.1:PORT, got {base_url!r}") + self.host, self.port, self.timeout = match.group(1), int(match.group(2)), timeout + self.base_url = f"http://{self.host}:{self.port}" + + @classmethod + def from_data_dir(cls, data_dir, timeout): + coords_path = os.path.join(data_dir, "ipfs-coords.json") + try: + coords = json.loads(read_bounded(coords_path, MAX_CONTROL_BYTES)) + port = coords["api_port"] + except (OSError, ValueError, KeyError, TypeError): + raise Refusal(f"{coords_path} does not name the Runtime Kubo api_port; is the Runtime's Kubo running?") + return cls(f"http://127.0.0.1:{port}", timeout) + + def _connect(self): + connection = http.client.HTTPConnection(self.host, self.port, timeout=self.timeout) + connection.response_class = TrailerResponse + return connection + + def _open(self, operation, query=None): + target = "/api/v0/" + operation + ("?" + urllib.parse.urlencode(query) if query else "") + connection = self._connect() + try: + connection.request("POST", target) + return connection, connection.getresponse() + except OSError as error: + connection.close() + raise Refusal(f"kubo api {self.base_url} unreachable: {error}") + + def _json(self, operation, query=None): + connection, response = self._open(operation, query) + with contextlib.closing(connection): + body = response.read(MAX_CONTROL_BYTES + 1) + text = body[:300].decode("utf-8", "replace").strip() + if response.status != 200 or len(body) > MAX_CONTROL_BYTES: + raise Refusal(f"kubo {operation} answered {response.status}: {text}") + try: + return json.loads(body) + except ValueError: + raise Refusal(f"kubo {operation} returned invalid JSON: {text}") + + def verify_profile(self): + version = self._json("version").get("Version") + if version != KUBO_VERSION: + raise Refusal(f"kubo version {version!r} is not {KUBO_VERSION}") + for key, accepts in IMPORT_PROFILE.items(): + answer = self._json("config", {"arg": key}) + if answer.get("Key") != key or "Value" not in answer or not accepts(answer["Value"]): + raise Refusal(f"kubo import profile {key}={answer.get('Value')!r} is not canonical") + return version + + def export_car(self, cid, destination): + connection, response = self._open("dag/export", {"arg": cid}) + with contextlib.closing(connection): + if response.status != 200: + text = response.read(MAX_CONTROL_BYTES)[:300].decode("utf-8", "replace").strip() + raise Refusal(f"kubo dag/export {cid} answered {response.status}: {text}") + digest = hashlib.sha256() + size = 0 + with open(destination, "wb") as sink: + for chunk in iter(lambda: response.read(CHUNK_BYTES), b""): + sink.write(chunk) + digest.update(chunk) + size += len(chunk) + sink.flush() + os.fsync(sink.fileno()) + if response.trailers.get("X-Stream-Error"): + raise Refusal(f"kubo dag/export {cid} failed mid-stream: {response.trailers['X-Stream-Error']}") + if size == 0: + raise Refusal(f"kubo dag/export {cid} returned an empty CAR") + return CarDigest(digest.hexdigest(), size) + + def repo_path(self): + path = self._json("repo/stat").get("RepoPath") + if not isinstance(path, str) or not path: + raise Refusal("kubo repo/stat did not report RepoPath") + return path + + def import_car(self, car_path, cid): + boundary = "elastos-model-package-" + os.urandom(12).hex() + head = ( + f"--{boundary}\r\n" + 'Content-Disposition: form-data; name="file"; filename="package.car"\r\n' + "Content-Type: application/vnd.ipld.car\r\n\r\n" + ).encode("ascii") + foot = f"\r\n--{boundary}--\r\n".encode("ascii") + size = os.path.getsize(car_path) + connection = self._connect() + try: + connection.putrequest("POST", "/api/v0/dag/import?pin-roots=true&stats=true") + connection.putheader("Content-Type", f"multipart/form-data; boundary={boundary}") + connection.putheader("Content-Length", str(len(head) + size + len(foot))) + connection.endheaders() + connection.send(head) + sent = 0 + with open(car_path, "rb") as source: + for chunk in iter(lambda: source.read(CHUNK_BYTES), b""): + connection.send(chunk) + sent += len(chunk) + if sent != size: + raise Refusal(f"{car_path} changed size during upload") + connection.send(foot) + response = connection.getresponse() + body = response.read(MAX_CONTROL_BYTES + 1) + except OSError as error: + raise Refusal(f"kubo dag/import upload to {self.base_url} failed: {error}") + finally: + connection.close() + return parse_import_events(response.status, body, response.trailers, cid) + + def is_recursively_pinned(self, cid): + connection, response = self._open("pin/ls", {"arg": cid, "type": "recursive"}) + with contextlib.closing(connection): + body = response.read(MAX_CONTROL_BYTES + 1) + if response.status != 200 or len(body) > MAX_CONTROL_BYTES: + return False + try: + keys = json.loads(body).get("Keys", {}) + except (ValueError, AttributeError): + return False + return isinstance(keys, dict) and keys.get(cid, {}).get("Type") == "recursive" + + +def parse_import_events(status, body, trailers, cid): + text = body.decode("utf-8", "replace") + if status != 200 or len(body) > MAX_CONTROL_BYTES: + raise Refusal(f"kubo dag/import answered {status}: {text.strip()[:300]}") + if trailers.get("X-Stream-Error"): + raise Refusal(f"kubo dag/import failed mid-stream: {trailers['X-Stream-Error']}") + root = stats = None + for line in filter(None, text.splitlines()): + try: + event = json.loads(line) + except ValueError: + raise Refusal(f"kubo dag/import returned a non-JSON line: {line[:300]}") + if not isinstance(event, dict) or "Message" in event or "Type" in event: + raise Refusal(f"kubo dag/import reported an error: {line[:300]}") + if root is None and isinstance(event.get("Root"), dict): + root = event["Root"] + elif stats is None and isinstance(event.get("Stats"), dict): + stats = event["Stats"] + else: + raise Refusal(f"kubo dag/import returned an unexpected event: {line[:300]}") + if root is None or stats is None: + raise Refusal("kubo dag/import did not report exactly one root and its stats") + imported = root.get("Cid", {}).get("/") if isinstance(root.get("Cid"), dict) else None + if imported != cid: + raise Refusal(f"kubo dag/import root {imported!r} is not the expected package {cid}") + if root.get("PinErrorMsg"): + raise Refusal(f"kubo could not pin {cid}: {root['PinErrorMsg']}") + try: + return ImportStats(int(stats["BlockCount"]), int(stats["BlockBytesCount"])) + except (KeyError, TypeError, ValueError): + raise Refusal("kubo dag/import stats lack BlockCount and BlockBytesCount") + + +def raw_cid(data): + multihash = b"\x12\x20" + hashlib.sha256(data).digest() + return "b" + base64.b32encode(b"\x01\x55" + multihash).decode("ascii").lower().rstrip("=") + + +def digest_file(path): + digest = hashlib.sha256() + size = 0 + with open(path, "rb") as source: + for chunk in iter(lambda: source.read(CHUNK_BYTES), b""): + digest.update(chunk) + size += len(chunk) + return CarDigest(digest.hexdigest(), size) + + +def read_bounded(path, limit): + with open(path, "rb") as source: + data = source.read(limit + 1) + if len(data) > limit: + raise Refusal(f"{path} exceeds its {limit}-byte bound") + return data + + +def check_private_dir(path): + info = os.lstat(path) + if not stat.S_ISDIR(info.st_mode): + raise Refusal(f"{path} must be a real directory, not a symlink or file") + if info.st_uid != os.geteuid() or info.st_mode & 0o022: + raise Refusal(f"{path} must be owned by the current user and not writable by group or others") + + +def write_private(path, data): + handle, temp_path = tempfile.mkstemp(prefix=".handoff-", dir=os.path.dirname(path) or ".") + try: + with os.fdopen(handle, "wb") as sink: + os.fchmod(handle, 0o600) + sink.write(data) + sink.flush() + os.fsync(sink.fileno()) + os.replace(temp_path, path) + except BaseException: + if os.path.exists(temp_path): + os.unlink(temp_path) + raise + + +def check_car(car_path, receipt_path, cid): + receipt = Receipt.load(receipt_path) + if receipt.package_cid != cid: + raise Refusal(f"receipt package_cid {receipt.package_cid} does not match expected CID {cid}") + size = os.path.getsize(car_path) + if size != receipt.car_bytes: + raise Refusal(f"CAR size {size} does not match receipt car_bytes {receipt.car_bytes}") + digest = digest_file(car_path) + if digest.sha256 != receipt.car_sha256: + raise Refusal(f"CAR sha256 {digest.sha256} does not match receipt car_sha256 {receipt.car_sha256}") + return receipt + + +def check_catalog(catalog, package_cid, publisher_dids): + try: + signed = json.loads(catalog) + except ValueError: + raise Refusal("catalog is not JSON") + if not isinstance(signed, dict) or set(signed) != {"payload", "signature", "signer_did"}: + raise Refusal("catalog must be a {payload, signature, signer_did} envelope") + payload = signed["payload"] + if not isinstance(payload, dict) or payload.get("schema") != CATALOG_SCHEMA: + raise Refusal(f"catalog payload.schema must be {CATALOG_SCHEMA}") + if payload.get("expires_at") is not None: + raise Refusal( + f"catalog expires_at={payload['expires_at']} is a timed snapshot; this helper pins permanent" + " snapshots only (sign a payload that omits expires_at)" + ) + entries = payload.get("entries") + if not isinstance(entries, list) or not (1 <= len(entries) <= 8): + raise Refusal("catalog must carry 1 to 8 entries") + if any(not isinstance(entry, dict) for entry in entries): + raise Refusal("catalog entries must be objects") + cids = [entry.get("cid") for entry in entries] + if len(set(cids)) != len(cids): + raise Refusal("catalog entries must use unique package CIDs") + if package_cid not in cids: + raise Refusal(f"catalog entries do not include --package-cid {package_cid}") + if signed["signer_did"] not in publisher_dids: + raise Refusal(f"catalog signer_did {signed['signer_did']!r} is not among the given --publisher-did values") + + +def package_cid_argument(value): + if not (value.startswith("bafy") and 8 <= len(value) <= 128 and set(value) <= BASE32_ALPHABET): + raise argparse.ArgumentTypeError(f"{value!r} is not a base32 CIDv1 starting with bafy") + return value + + +def bounded_int(minimum): + def parse(value): + number = int(value) + if not minimum <= number <= MAX_BUDGET: + raise argparse.ArgumentTypeError(f"{value} is outside {minimum}..{MAX_BUDGET}") + return number + + return parse + + +def kubo_from_args(args): + if args.kubo_api: + return KuboApi(args.kubo_api, args.timeout_seconds) + return KuboApi.from_data_dir(args.data_dir, args.timeout_seconds) + + +def cmd_verify_kubo(args): + return {"kubo_version": kubo_from_args(args).verify_profile(), "import_profile": "canonical"} + + +def cmd_head_cid(args): + catalog = read_bounded(args.catalog, MAX_CATALOG_BYTES) + return {"head_cid": raw_cid(catalog), "catalog_bytes": len(catalog)} + + +def cmd_export(args): + kubo = kubo_from_args(args) + version = kubo.verify_profile() + partial = args.output + ".partial" + try: + digest = kubo.export_car(args.cid, partial) + except BaseException: + if os.path.exists(partial): + os.unlink(partial) + raise + os.replace(partial, args.output) + receipt = Receipt(args.cid, digest.sha256, digest.size, version, int(time.time())) + with open(args.output + ".receipt.json", "wb") as sink: + sink.write(json.dumps(receipt.to_json(), indent=2).encode("utf-8") + b"\n") + return receipt.to_json() + + +def cmd_verify_car(args): + return check_car(args.car, args.receipt, args.cid).to_json() + + +def cmd_import(args): + receipt = check_car(args.car, args.receipt, args.cid) + kubo = kubo_from_args(args) + version = kubo.verify_profile() + repo_path = kubo.repo_path() + try: + free = shutil.disk_usage(repo_path).free + except OSError as error: + raise Refusal(f"cannot measure free space at kubo repo {repo_path}: {error}") + if free - receipt.car_bytes < args.free_space_floor_bytes: + raise Refusal( + f"kubo repo {repo_path} has {free} free bytes; importing {receipt.car_bytes} bytes would" + f" leave less than the {args.free_space_floor_bytes}-byte --free-space-floor-bytes floor" + ) + stats = kubo.import_car(args.car, args.cid) + if not kubo.is_recursively_pinned(args.cid): + raise Refusal(f"kubo imported {args.cid} but does not list it as a recursive pin") + return { + "schema": IMPORT_SCHEMA, + "package_cid": args.cid, + "pinned": True, + "blocks": stats.blocks, + "block_bytes": stats.block_bytes, + "kubo_version": version, + } + + +def cmd_pin_catalog(args): + dids = args.publisher_dids + if len(set(dids)) != len(dids) or len(dids) > 8 or any(not did.startswith("did:key:z") or len(did) > 128 for did in dids): + raise Refusal("--publisher-did values must be unique did:key:z... identifiers, at most eight") + check_private_dir(args.data_dir) + catalog = read_bounded(args.catalog, MAX_CATALOG_BYTES) + head = raw_cid(catalog) + if head != args.head_cid: + raise Refusal(f"catalog head is {head}, not --head-cid {args.head_cid}") + check_catalog(catalog, args.package_cid, args.publisher_dids) + pin = CatalogPin(head, tuple(args.publisher_dids), args.max_cache_bytes, args.max_model_memory_bytes) + components_path = os.path.join(args.data_dir, "components.json") + try: + components = json.loads(read_bounded(components_path, MAX_COMPONENTS_BYTES)) + except ValueError: + raise Refusal(f"{components_path} is not JSON") + if not isinstance(components, dict): + raise Refusal(f"{components_path} must be a JSON object") + components["model_catalog"] = pin.to_json() + write_private(os.path.join(args.data_dir, "model-catalog.json"), catalog) + write_private(components_path, json.dumps(components, indent=2).encode("utf-8") + b"\n") + return { + "schema": CATALOG_PIN_SCHEMA, + "head_cid": head, + "package_cid": args.package_cid, + **pin.to_json(), + "signature_verification": "pending_runtime_read", + } + + +def add_kubo_arguments(parser): + target = parser.add_mutually_exclusive_group(required=True) + target.add_argument("--kubo-api", metavar="URL", help="Kubo HTTP API, e.g. http://127.0.0.1:5001") + target.add_argument("--data-dir", metavar="DIR", help="Runtime data dir holding ipfs-coords.json") + parser.add_argument("--timeout-seconds", type=float, default=60.0, help="per-request and per-chunk idle timeout") + + +def build_parser(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + commands = parser.add_subparsers(dest="command", required=True) + + verify_kubo = commands.add_parser("verify-kubo", help="check the pinned Kubo version and canonical import profile") + add_kubo_arguments(verify_kubo) + verify_kubo.set_defaults(run=cmd_verify_kubo) + + head_cid = commands.add_parser("head-cid", help="print the raw CIDv1 SHA-256 of a catalog file") + head_cid.add_argument("--catalog", required=True) + head_cid.set_defaults(run=cmd_head_cid) + + export = commands.add_parser("export", help="stream a package closure to FILE.car with a receipt") + export.add_argument("--cid", required=True, type=package_cid_argument) + export.add_argument("--output", required=True, metavar="FILE.car") + add_kubo_arguments(export) + export.set_defaults(run=cmd_export) + + verify_car = commands.add_parser("verify-car", help="check a CAR against its receipt and expected CID") + verify_car.add_argument("--car", required=True) + verify_car.add_argument("--receipt", required=True) + verify_car.add_argument("--cid", required=True, type=package_cid_argument) + verify_car.set_defaults(run=cmd_verify_car) + + import_ = commands.add_parser("import", help="verify, import and pin a CAR in the receiver's Kubo") + import_.add_argument("--car", required=True) + import_.add_argument("--receipt", required=True) + import_.add_argument("--cid", required=True, type=package_cid_argument) + import_.add_argument("--free-space-floor-bytes", required=True, type=bounded_int(0)) + add_kubo_arguments(import_) + import_.set_defaults(run=cmd_import) + + pin_catalog = commands.add_parser("pin-catalog", help="install a permanent catalog snapshot and pin its head") + pin_catalog.add_argument("--data-dir", required=True) + pin_catalog.add_argument("--catalog", required=True, metavar="model-catalog.json") + pin_catalog.add_argument("--head-cid", required=True) + pin_catalog.add_argument("--publisher-did", dest="publisher_dids", action="append", required=True, metavar="DID") + pin_catalog.add_argument("--package-cid", required=True, type=package_cid_argument) + pin_catalog.add_argument("--max-cache-bytes", required=True, type=bounded_int(1)) + pin_catalog.add_argument("--max-model-memory-bytes", required=True, type=bounded_int(1)) + pin_catalog.set_defaults(run=cmd_pin_catalog) + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + try: + receipt = args.run(args) + except (Refusal, OSError) as error: + print(f"error: {error}", file=sys.stderr) + return 1 + print(json.dumps(receipt)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/model-preparation-failure-smoke.mjs b/scripts/model-preparation-failure-smoke.mjs index 46de3b933..5c14ea749 100644 --- a/scripts/model-preparation-failure-smoke.mjs +++ b/scripts/model-preparation-failure-smoke.mjs @@ -9,8 +9,8 @@ for (const capsule of ["system", "marketplace"]) { } const context = { window: {} }; vm.runInNewContext(source.replace("window.ElastosModelManagement = {", - "window.testFailure = { parseRuntime, operationRuntime, failureText }; window.ElastosModelManagement = {"), context); -const { parseRuntime, operationRuntime, failureText } = context.window.testFailure; + "window.testFailure = { parseRuntime, operationRuntime, failureText, catalogEntries, selectCatalogEntry }; window.ElastosModelManagement = {"), context); +const { parseRuntime, operationRuntime, failureText, catalogEntries, selectCatalogEntry } = context.window.testFailure; const cid = `bafybei${"a".repeat(52)}`; const preparation = { operation_id: "fixture", cid, state: "failed", total_bytes: 1024, completed_bytes: 512, cancel_requested: false, admitted: false, activation_pending: false }; @@ -37,4 +37,22 @@ for (const invalid of ["/private/provider?credential=secret", "__proto__", "cons preparation.failure_class = invalid; assert.throws(() => parseRuntime(runtime, cid), /Invalid model response/); } -console.log("PASS model preparation failure: safe classes, operation/catalog parity, unknown history, strict rejection, shared copies"); +const otherCid = `bafybei${"c".repeat(51)}e`; +const catalogRow = (id, title) => ({ + name: title, title, role: "content", source: "signed-model-catalog", installed: false, launchable: false, + cid: id, publisher_did: "did:key:zFixturePublisher", content_size_bytes: 1024, + signature_state: "catalog-signature-verified", model_runtime: { + admitted: false, kept: false, dispatch_ready: false, offer_id: null, preparation: null, + }, +}); +const two = catalogEntries({ + schema: "elastos.capsules.catalog/v1", model_catalog_state: "verified", + capsules: [catalogRow(cid, "Qwen"), catalogRow(otherCid, "SmolLM2")], +}); +assert.equal(two.length, 2); +assert.equal(selectCatalogEntry(two, null), null); +assert.equal(selectCatalogEntry(two, cid).cid, cid); +assert.equal(selectCatalogEntry(two, otherCid).cid, otherCid); +assert.equal(selectCatalogEntry(two, `bafybei${"d".repeat(52)}`), null); +assert.equal(selectCatalogEntry(two.slice(0, 1), null).cid, cid); +console.log("PASS model preparation failure: safe classes, operation/catalog parity, unknown history, strict rejection, shared copies, explicit two-entry selection"); diff --git a/scripts/model-selection-browser-smoke.mjs b/scripts/model-selection-browser-smoke.mjs index 3cc6ff0e7..69f081af9 100644 --- a/scripts/model-selection-browser-smoke.mjs +++ b/scripts/model-selection-browser-smoke.mjs @@ -152,7 +152,7 @@ try { assert.equal(await menu.locator('[role="option"][aria-selected="true"]').count(), 0, "failed refresh leaves no selectable current model"); await menu.getByRole("button", { name: "Open Models", exact: true }).click(); const message = await page.waitForFunction(() => window.messages.find(m => m.type === "home:open-target")); - assert.deepEqual(await message.jsonValue(), { type: "home:open-target", homeToken: "fixture", target: "system", query: { settings: "models" } }); + assert.deepEqual(await message.jsonValue(), { type: "home:open-target", homeToken: "fixture", target: "marketplace", query: { category: "models" } }); assert.equal(await draft.inputValue(), "Keep my draft"); assert.deepEqual(pageErrors, []); await page.close(); diff --git a/scripts/model-selection.test.mjs b/scripts/model-selection.test.mjs index 7bbc282f1..101cebd05 100644 --- a/scripts/model-selection.test.mjs +++ b/scripts/model-selection.test.mjs @@ -22,6 +22,18 @@ test("CID intent requires exact current unique ready mapping; service intent has assert.equal(selectedModelOffer([hosted], hosted.id, cid, []), null); }); +test("catalog accepts distinct signed entries and exact CID selection", () => { + const otherCid = `bafybei${"c".repeat(51)}e`; + const other = { ...row, cid: otherCid, title: "Other model", + model_runtime: { admitted: true, dispatch_ready: true, offer_id: "other-offer-id" } }; + const models = catalogModels(catalog([row, other])); + assert.equal(models.length, 2); + assert.equal(selectedModelOffer([offer], offer.id, cid, models), offer); + const otherOffer = { id: "other-offer-id" }; + assert.equal(selectedModelOffer([otherOffer], otherOffer.id, otherCid, models), otherOffer); + assert.equal(selectedModelOffer([offer], offer.id, otherCid, models), null); +}); + test("catalog rejects duplicate, malformed and noncanonical content identity", () => { assert.throws(() => catalogModels(catalog([row, row]))); for (const value of [cid.toUpperCase(), `${cid} `, `bafybei${"a".repeat(51)}b`, "https://model.invalid"]) { diff --git a/scripts/pinned-model-consumer-proof.sh b/scripts/pinned-model-consumer-proof.sh new file mode 100755 index 000000000..905dfcce3 --- /dev/null +++ b/scripts/pinned-model-consumer-proof.sh @@ -0,0 +1,436 @@ +#!/usr/bin/env bash +set -euo pipefail +umask 077 + +# Isolated installed-consumer proof for a pinned small model. +# +# The harness installs the Runtime-verified local model engine into a +# disposable data root through the ordinary `elastos setup` path, verifies the +# caller-owned model and license bytes against their pinned identity, and then +# runs the opt-in real-engine proofs that the elastos-server test suite owns: +# native startup/reply/cancel/restart, and cold loopback Content admission +# followed by reply, restart and zero-download reuse. Every process, port and +# temporary path belongs to this run; the receipt binds the source tree, the +# built binaries, the installed engine receipt and the proof outcomes. +# +# This proves engine activation on the host it runs on. It is not Marketplace +# delivery evidence: the model bytes arrive from a caller-supplied file, the +# catalog is signed with an isolated test key, and no public holder is used. + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +usage() { + cat <<'EOF' +Usage: + scripts/pinned-model-consumer-proof.sh [--fetch] [--native-only] + scripts/pinned-model-consumer-proof.sh --check-inputs + scripts/pinned-model-consumer-proof.sh --self-test + +Arguments: + New or existing directory owned by this proof. It receives + home/ (isolated HOME with the installed engine), kubo/, + inputs/ (with --fetch) and proof/ (logs and receipt.json). + --fetch Download the pinned model and license into /inputs + when ELASTOS_PROOF_MODEL_PATH / _LICENSE_PATH are unset. + --native-only Skip the cold Content proof (no Kubo or ipfs-provider). + --check-inputs Verify pinned inputs and print the plan; run nothing else. + --self-test Check the test-output verifier against captured cargo output + shapes (zero matches, unselected ignored test, wrong test, + one real pass); run nothing else. + +Environment: + ELASTOS_PROOF_MODEL Pinned model id. Default and only value: smollm2 + ELASTOS_PROOF_MODEL_PATH Read-only pinned GGUF file (owner-only write) + ELASTOS_PROOF_LICENSE_PATH Read-only license text with the pinned digest + ELASTOS_PROOF_ELASTOS_BIN Prebuilt `elastos` binary (default: cargo build) + ELASTOS_PROOF_MODEL_PROVIDER Prebuilt model-provider (default: cargo build) + ELASTOS_PROOF_IPFS_PROVIDER Prebuilt ipfs-provider (default: cargo build) + ELASTOS_PROOF_ACCOMMODATIONS Named temporary source accommodations recorded + in the receipt; required when the tree is dirty + ELASTOS_CARGO_BIN cargo executable. Default: cargo +EOF +} + +die() { + echo "[pinned-model-proof] $*" >&2 + exit 1 +} + +need_cmd() { + command -v "$1" >/dev/null 2>&1 || die "missing required command: $1" +} + +sha256_file() { + if command -v sha256sum >/dev/null 2>&1; then + sha256sum "$1" | awk '{print $1}' + else + shasum -a 256 "$1" | awk '{print $1}' + fi +} + +# Pinned model table. Bytes and digests are the same on every host; the +# engine executable identity is per platform and lives in the Rust proof. +pinned_field() { + local model="$1" field="$2" + case "${model}:${field}" in + smollm2:file) echo "SmolLM2-135M-Instruct-Q8_0.gguf" ;; + smollm2:url) echo "https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/9e6855bc4be717fca1ef21360a1db4b29d5c559a/SmolLM2-135M-Instruct-Q8_0.gguf" ;; + smollm2:bytes) echo "144811072" ;; + smollm2:sha256) echo "c4a3dd037301b6ecea31d6da37f5cd793ead920dd5ddfe6d589294628d6ce66a" ;; + smollm2:license_url) echo "https://www.apache.org/licenses/LICENSE-2.0.txt" ;; + smollm2:license_bytes) echo "11358" ;; + smollm2:license_sha256) echo "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30" ;; + smollm2:env_prefix) echo "SMOLLM2" ;; + smollm2:native_test) echo "model_native_smollm2_exact_profile_lifecycle" ;; + smollm2:cold_test) echo "model_preparation_real_smollm2_cold_reply_restart" ;; + *) die "unknown pinned model field ${model}:${field}" ;; + esac +} + +detect_platform() { + case "$(uname -s)-$(uname -m)" in + Linux-x86_64) echo "linux-amd64" ;; + Darwin-arm64) echo "darwin-arm64" ;; + *) die "no Runtime-owned model host profile for $(uname -s)-$(uname -m)" ;; + esac +} + +verify_pinned_file() { + local label="$1" path="$2" bytes="$3" sha="$4" + [[ -f "$path" && ! -L "$path" ]] || die "$label is not a regular file: $path" + local actual_bytes + actual_bytes="$(wc -c <"$path" | tr -d ' ')" + [[ "$actual_bytes" == "$bytes" ]] || die "$label size ${actual_bytes} differs from pinned ${bytes}" + local actual_sha + actual_sha="$(sha256_file "$path")" + [[ "$actual_sha" == "$sha" ]] || die "$label sha256 ${actual_sha} differs from pinned ${sha}" + echo "[pinned-model-proof] $label verified: ${bytes} bytes sha256 ${sha}" +} + +fetch_pinned() { + local url="$1" dest="$2" + need_cmd curl + if [[ ! -f "$dest" ]]; then + echo "[pinned-model-proof] downloading $(basename "$dest")" + curl -fsSL --retry 3 -o "${dest}.partial" "$url" + mv -f -- "${dest}.partial" "$dest" + fi + chmod 0400 "$dest" +} + +PROOF_MODULE="api::capsule_inventory::preparation::tests::process_proof" + +# A cargo exit status of zero does not prove that the intended test ran: a +# filter that matches nothing runs zero tests and still exits zero, and an +# ignored test that is not selected prints "ignored" and exits zero. The log +# must show exactly the intended test executing and passing, once, and the +# summary must count exactly that one pass. Prints the refusal reason. +verify_proof_output() { + local log="$1" test_id="$2" + [[ -s "$log" ]] || { echo "empty test output"; return 1; } + if grep -qF "test ${test_id} ... ignored" "$log"; then + echo "the intended test was ignored, not executed" + return 1 + fi + if grep -qF "test ${test_id} ... FAILED" "$log"; then + echo "the intended test failed" + return 1 + fi + local executed + executed="$(grep -cxF "test ${test_id} ... ok" "$log" || true)" + if [[ "$executed" != "1" ]]; then + echo "expected exactly one passing execution of ${test_id}, found ${executed}" + return 1 + fi + if ! grep -qE '^test result: ok\. 1 passed; 0 failed; 0 ignored; 0 measured; [0-9]+ filtered out' "$log"; then + echo "the summary does not count exactly one passing test" + return 1 + fi +} + +# Regression for the verifier on captured cargo output shapes: a zero-match +# filter and an unselected ignored test are refused, a wrong test is refused, +# and one real passing execution is accepted. +verifier_self_test() { + local intended="${PROOF_MODULE}::model_native_smollm2_exact_profile_lifecycle" + local scratch + scratch="$(mktemp -d)" + cat >"${scratch}/zero-match.log" <<'EOF' + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.31s + Running unittests src/lib.rs (/build/deps/elastos_server-2795cde28a08a7f8) + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1903 filtered out; finished in 0.00s + +EOF + cat >"${scratch}/not-selected.log" <"${scratch}/wrong-test.log" <"${scratch}/one-pass.log" <&2 + failures=$((failures + 1)) + else + echo "[pinned-model-proof] self-test: ${refused} refused (${reason})" + fi + done + if reason="$(verify_proof_output "${scratch}/one-pass.log" "$intended")"; then + echo "[pinned-model-proof] self-test: one real passing execution accepted" + else + echo "[pinned-model-proof] self-test: real passing output refused (${reason})" >&2 + failures=$((failures + 1)) + fi + rm -rf -- "$scratch" + [[ "$failures" -eq 0 ]] || die "verifier self-test failed (${failures})" +} + +PROOF_ROOT="" +FETCH=0 +NATIVE_ONLY=0 +CHECK_INPUTS=0 +for arg in "$@"; do + case "$arg" in + --help | -h) usage; exit 0 ;; + --self-test) verifier_self_test; exit 0 ;; + --fetch) FETCH=1 ;; + --native-only) NATIVE_ONLY=1 ;; + --check-inputs) CHECK_INPUTS=1 ;; + --*) die "unknown option: $arg" ;; + *) + [[ -z "$PROOF_ROOT" ]] || die "only one proof root is accepted" + PROOF_ROOT="$arg" + ;; + esac +done +[[ -n "$PROOF_ROOT" ]] || { usage >&2; exit 2; } + +need_cmd python3 +need_cmd tar +MODEL="${ELASTOS_PROOF_MODEL:-smollm2}" +PLATFORM="$(detect_platform)" +CARGO_BIN="${ELASTOS_CARGO_BIN:-cargo}" + +mkdir -p "$PROOF_ROOT" +PROOF_ROOT="$(cd "$PROOF_ROOT" && pwd -P)" +case "$PROOF_ROOT" in + /) die "proof root cannot be the filesystem root" ;; +esac +mkdir -p "${PROOF_ROOT}/proof" "${PROOF_ROOT}/inputs" + +MODEL_PATH="${ELASTOS_PROOF_MODEL_PATH:-}" +LICENSE_PATH="${ELASTOS_PROOF_LICENSE_PATH:-}" +if [[ "$FETCH" == "1" ]]; then + [[ -n "$MODEL_PATH" ]] || { + MODEL_PATH="${PROOF_ROOT}/inputs/$(pinned_field "$MODEL" file)" + fetch_pinned "$(pinned_field "$MODEL" url)" "$MODEL_PATH" + } + [[ -n "$LICENSE_PATH" ]] || { + LICENSE_PATH="${PROOF_ROOT}/inputs/LICENSE" + fetch_pinned "$(pinned_field "$MODEL" license_url)" "$LICENSE_PATH" + } +fi +[[ -n "$MODEL_PATH" && -n "$LICENSE_PATH" ]] \ + || die "set ELASTOS_PROOF_MODEL_PATH and ELASTOS_PROOF_LICENSE_PATH, or pass --fetch" +MODEL_PATH="$(cd "$(dirname "$MODEL_PATH")" && pwd -P)/$(basename "$MODEL_PATH")" +LICENSE_PATH="$(cd "$(dirname "$LICENSE_PATH")" && pwd -P)/$(basename "$LICENSE_PATH")" +verify_pinned_file "model" "$MODEL_PATH" \ + "$(pinned_field "$MODEL" bytes)" "$(pinned_field "$MODEL" sha256)" +verify_pinned_file "license" "$LICENSE_PATH" \ + "$(pinned_field "$MODEL" license_bytes)" "$(pinned_field "$MODEL" license_sha256)" + +HOME_ROOT="${PROOF_ROOT}/home" +DATA_DIR="${HOME_ROOT}/xdg/elastos" +if [[ "$PLATFORM" == "darwin-arm64" ]]; then + DATA_DIR="${HOME_ROOT}/Library/Application Support/elastos" +fi + +if [[ "$CHECK_INPUTS" == "1" ]]; then + cat </dev/null 2>&1 \ + || die "cargo build failed for $bin ($manifest)" + local target_dir + target_dir="$(dirname "$manifest")/target/debug" + [[ -x "${target_dir}/${bin}" ]] || die "built binary missing: ${target_dir}/${bin}" + printf '%s\n' "$(cd "$target_dir" && pwd -P)/${bin}" +} + +echo "[pinned-model-proof] resolving binaries" +ELASTOS_BIN="$(build_or_reuse "${ELASTOS_PROOF_ELASTOS_BIN:-}" "${ROOT}/elastos/Cargo.toml" elastos)" +MODEL_PROVIDER="$(build_or_reuse "${ELASTOS_PROOF_MODEL_PROVIDER:-}" "${ROOT}/capsules/model-provider/Cargo.toml" model-provider)" +IPFS_PROVIDER="" +KUBO_BIN="" +if [[ "$NATIVE_ONLY" != "1" ]]; then + IPFS_PROVIDER="$(build_or_reuse "${ELASTOS_PROOF_IPFS_PROVIDER:-}" "${ROOT}/capsules/ipfs-provider/Cargo.toml" ipfs-provider)" + KUBO_BIN="${PROOF_ROOT}/kubo/bin/kubo" + if [[ ! -x "$KUBO_BIN" ]]; then + "${ROOT}/scripts/seed-kubo-cache.sh" "${PROOF_ROOT}/kubo/cache" "${PROOF_ROOT}/kubo" "$PLATFORM" + fi +fi + +# The ordinary installer owns the engine bundle and its v2 receipt. HOME and +# XDG_DATA_HOME select the isolated root; PATH keeps only system tools. +echo "[pinned-model-proof] installing the local model engine through elastos setup" +mkdir -p "${HOME_ROOT}" +env -i HOME="$HOME_ROOT" XDG_DATA_HOME="${HOME_ROOT}/xdg" PATH="/usr/bin:/bin" \ + ELASTOS_QUIET_RUNTIME_NOTICES=1 \ + "$ELASTOS_BIN" setup --profile operator --with llama-server \ + --without shell,localhost-provider,did-provider >"${PROOF_ROOT}/proof/setup.log" 2>&1 \ + || die "elastos setup failed; see ${PROOF_ROOT}/proof/setup.log" +BUNDLE="$(python3 - "${DATA_DIR}/components.json" "$PLATFORM" <<'PY' +import json, sys +manifest = json.load(open(sys.argv[1])) +info = manifest["external"]["llama-server"]["platforms"][sys.argv[2]] +print(info["install_path"]) +PY +)" +RECEIPT="${DATA_DIR}/${BUNDLE}/.elastos-engine.json" +[[ -f "$RECEIPT" ]] || die "engine receipt missing: $RECEIPT" +ENGINE_BIN="${DATA_DIR}/${BUNDLE}/llama-server" +[[ -x "$ENGINE_BIN" ]] || die "engine executable missing: $ENGINE_BIN" + +PREFIX="$(pinned_field "$MODEL" env_prefix)" +PROOF_ENV=( + "ELASTOS_TEST_${PREFIX}_PATH=${MODEL_PATH}" + "ELASTOS_TEST_${PREFIX}_LICENSE_PATH=${LICENSE_PATH}" + "ELASTOS_TEST_${PREFIX}_ENGINE_DATA=${DATA_DIR}" + "ELASTOS_TEST_MODEL_PROVIDER_PATH=${MODEL_PROVIDER}" +) +[[ -z "$IPFS_PROVIDER" ]] || PROOF_ENV+=("ELASTOS_TEST_IPFS_PROVIDER_PATH=${IPFS_PROVIDER}") +[[ -z "$KUBO_BIN" ]] || PROOF_ENV+=("ELASTOS_TEST_KUBO_PATH=${KUBO_BIN}") +run_proof() { + local test_name="$1" log="${PROOF_ROOT}/proof/${1}.log" + local test_id="${PROOF_MODULE}::${test_name}" + echo "[pinned-model-proof] running ${test_id}" + local status=0 + ( + cd "${ROOT}/elastos" + env "${PROOF_ENV[@]}" \ + "$CARGO_BIN" test --locked -p elastos-server --lib -- --ignored --nocapture \ + --exact "$test_id" + ) >"$log" 2>&1 || status=$? + if [[ "$status" -ne 0 ]]; then + echo "[pinned-model-proof] ${test_name} FAILED (exit ${status}); see ${log}" >&2 + return "$status" + fi + local reason + if ! reason="$(verify_proof_output "$log" "$test_id")"; then + echo "[pinned-model-proof] ${test_name} NOT PROVEN: ${reason}; see ${log}" >&2 + return 1 + fi + echo "[pinned-model-proof] ${test_name} passed" +} + +# The verifier that decides "passed" proves itself before any proof runs. +verifier_self_test +RESULT_NATIVE="failed" +RESULT_COLD="skipped" +run_proof "$(pinned_field "$MODEL" native_test)" && RESULT_NATIVE="passed" +if [[ "$NATIVE_ONLY" != "1" ]]; then + RESULT_COLD="failed" + run_proof "$(pinned_field "$MODEL" cold_test)" && RESULT_COLD="passed" +fi + +SOURCE_COMMIT="$(git -C "$ROOT" rev-parse HEAD)" +SOURCE_TREE="$(git -C "$ROOT" rev-parse 'HEAD^{tree}')" +SOURCE_DIRTY="False" +[[ -z "$(git -C "$ROOT" status --porcelain)" ]] || SOURCE_DIRTY="True" +# A dirty tree means the proof ran against source that no commit identifies. +# The operator names every temporary accommodation so the receipt says which +# behavior came from the reviewed commit and which did not. +ACCOMMODATIONS="${ELASTOS_PROOF_ACCOMMODATIONS:-}" +if [[ "$SOURCE_DIRTY" == "True" && -z "$ACCOMMODATIONS" ]]; then + die "source tree is dirty; set ELASTOS_PROOF_ACCOMMODATIONS to name the temporary changes" +fi +python3 - "${PROOF_ROOT}/proof/receipt.json" "${ACCOMMODATIONS:-none}" < Date: Wed, 16 Sep 2026 20:17:46 -0500 Subject: [PATCH 336/535] fix(installer): stage the signed catalogue before replacing the binary A failed catalogue fetch must leave the current installation in place, including the running process and the prior catalogue snapshot. --- scripts/install-bootstrap-test.py | 88 +++++++++++++++++++++++++++++-- scripts/install.sh | 46 ++++++++++++++-- 2 files changed, 128 insertions(+), 6 deletions(-) diff --git a/scripts/install-bootstrap-test.py b/scripts/install-bootstrap-test.py index 29fad3707..1da4cde67 100644 --- a/scripts/install-bootstrap-test.py +++ b/scripts/install-bootstrap-test.py @@ -7,6 +7,7 @@ """ import argparse +import base64 import copy import hashlib import json @@ -107,6 +108,21 @@ def encode_envelope(envelope): b'"ticket":"fixture-ticket","node_id":"fixture-node"}\n') +CATALOG = b'{"payload":{"schema":"elastos.model.catalog/v1","entries":[]},"note":"fixture catalog"}\n' + + +def catalog_head_cid(data): + """Raw SHA-256 CIDv1 of the catalogue bytes, the form components.json pins.""" + return "b" + base64.b32encode(b"\x01\x55\x12\x20" + hashlib.sha256(data).digest()).decode().lower().rstrip("=") + + +def pinned_components(catalog=CATALOG): + manifest = json.loads(COMPONENTS) + manifest["model_catalog"] = {"head_cid": catalog_head_cid(catalog), + "publisher_dids": ["did:key:z6MkfixturePublisher"]} + return json.dumps(manifest, sort_keys=True).encode() + b"\n" + + def installable_fixture(runtime=RUNTIME_STUB, components=COMPONENTS, version="0.7.1"): """Deterministic signed head/release advertising the fixture Runtime stub and manifest.""" release_payload = { @@ -349,10 +365,11 @@ class InstallerSandbox: and any request for a response that is absent, fails and is logged. """ - def __init__(self, head, release, did, system="Linux", machine="x86_64"): + def __init__(self, head, release, did, system="Linux", machine="x86_64", catalog_cid=""): self.directory = tempfile.TemporaryDirectory(prefix="installer-sandbox-") self.root = Path(self.directory.name) self.did, self.system, self.machine = did, system, machine + self.catalog_cid = catalog_cid self.home = self.root / "home" self.data = self.home / "xdg-data/elastos" self.binary = self.home / ".local/bin/elastos" @@ -381,6 +398,7 @@ def __init__(self, head, release, did, system="Linux", machine="x86_64"): */release.json|*/ipfs/"$FIXTURE_RELEASE_CID") response=release.json;; */artifacts/elastos-*|*/ipfs/"$FIXTURE_BINARY_CID") response=binary;; */artifacts/components-*.json|*/ipfs/"$FIXTURE_COMPONENTS_CID") response=components;; + */artifacts/model-catalog.json|*/ipfs/"$FIXTURE_CATALOG_CID") response=catalog;; */.well-known/elastos/carrier-bootstrap.json*) response=bootstrap;; *) echo unexpected-request-blocked >&2; exit 93;; esac @@ -412,9 +430,10 @@ def run(self, *options, transport="publisher"): export ELASTOS_SOURCE_CONNECT_TICKET="" ELASTOS_PUBLISHER_NODE_ID="" ELASTOS_INSTALL_ONLY="" export ELASTOS_TEST_CALLS="$1/calls" MOCK_SYSTEM="$4" MOCK_MACHINE="$5" export FIXTURE_HEAD_CID="$6" FIXTURE_RELEASE_CID="$7" FIXTURE_BINARY_CID="$8" FIXTURE_COMPONENTS_CID="$9" -exec "$2" --noprofile --norc "$3" "${@:10}" +export FIXTURE_CATALOG_CID="${10}" +exec "$2" --noprofile --norc "$3" "${@:11}" ''', self.root, OPTIONS.bash, INSTALLER, self.system, self.machine, self.head_cid, self.release_cid, - self.binary_cid, self.components_cid, "--maintainer-did", self.did, *options) + self.binary_cid, self.components_cid, self.catalog_cid, "--maintainer-did", self.did, *options) return result, self.requests()[seen:] def runtime_calls(self): @@ -657,6 +676,69 @@ def assert_no_installation_effects(self, sandbox, before, result): self.assertEqual(sandbox.home_state(), before) self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + def test_pinned_catalogue_is_verified_before_any_installation_change(self): + # A release whose components.json pins a signed model catalogue advertises + # three files. A missing or mismatched catalogue is refused like a bad + # binary: before Runtime processes are stopped and before the current + # binary, components.json or catalogue change. + components = pinned_components() + catalog_cid = catalog_head_cid(CATALOG) + did, head, release = installable_fixture(components=components) + previous_catalog = b'{"note":"previous catalog"}\n' + catalog_url = {"publisher": "https://test.invalid/artifacts/model-catalog.json", + "cid": f"https://test.invalid/ipfs/{catalog_cid}"} + for transport in ["publisher", "cid"]: + with InstallerSandbox(head, release, did, catalog_cid=catalog_cid) as sandbox: + self.existing_installation(sandbox) + (sandbox.data / "model-catalog.json").write_bytes(previous_catalog) + (sandbox.data / "model-catalog.json").chmod(0o600) + sandbox.respond("components", components) + sandbox.respond("binary", RUNTIME_STUB) + sandbox.respond("bootstrap", BOOTSTRAP) + before = sandbox.home_state() + expected = self.REQUESTS[transport][:4] + [catalog_url[transport]] + for name, served in [("missing catalogue", None), + ("catalogue differs from pin", CATALOG.replace(b"fixture", b"altered"))]: + (sandbox.responses / "catalog").unlink(missing_ok=True) + if served is not None: + sandbox.respond("catalog", served) + result, requests = sandbox.run("--install-only", transport=transport) + with self.subTest(transport=transport, case=name): + self.assert_no_installation_effects(sandbox, before, result) + self.assertEqual((sandbox.data / "model-catalog.json").read_bytes(), previous_catalog) + self.assertIn("Verifying components.json SHA-256", result.stdout) + self.assertNotIn("Installing binary to", result.stdout) + if served is None: + self.assertRegex(result.stderr, "Failed to (download model catalog|fetch CID)") + else: + self.assertIn("does not match pin", result.stderr) + self.assertIn("the current installation was preserved", result.stderr) + self.assertEqual(requests, expected) + sandbox.respond("catalog", CATALOG) + sandbox.calls.unlink(missing_ok=True) + result, requests = sandbox.run("--install-only", transport=transport) + with self.subTest(transport=transport, case="exact catalogue"): + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("Runtime installed:", result.stdout) + bootstrap = self.REQUESTS[transport][4:] + self.assertEqual(requests, expected + bootstrap) + self.assertLess(result.stdout.index("Verifying model catalog"), + result.stdout.index("Stopping verified Runtime processes")) + installed = sandbox.data / "model-catalog.json" + self.assertEqual(installed.read_bytes(), CATALOG) + self.assertEqual(installed.stat().st_mode & 0o777, 0o600) + self.assertEqual(sandbox.binary.read_bytes(), RUNTIME_STUB) + self.assertEqual((sandbox.data / "components.json").read_bytes(), components) + self.assertEqual(sandbox.runtime_calls()[0], "--version") + after = sandbox.home_state() + changed = {key for key in set(before) | set(after) if before.get(key) != after.get(key)} + self.assertEqual(changed, { + ".local/bin/elastos", "xdg-data/elastos/components.json", "xdg-data/elastos/model-catalog.json", + "xdg-data/elastos/sources.json", + "xdg-data/elastos/ElastOS/SystemServices/Publisher/release-head.json", + "xdg-data/elastos/ElastOS/SystemServices/Publisher/release.json"}) + self.assertEqual(list((sandbox.root / "tmp").iterdir()), []) + def test_corrupt_artifacts_fail_before_changes_then_clean_rerun_installs(self): did, head, release = installable_fixture() cases = [ diff --git a/scripts/install.sh b/scripts/install.sh index 625223b06..8878add95 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -21,9 +21,10 @@ # Trust anchors can be provided via env vars or CLI flags. In the canonical # bootstrap flow, they should already be stamped into install.sh. # -# Downloads exactly 2 files: +# Downloads the signed Runtime, components.json, and the pinned model catalog: # 1. elastos binary → ~/.local/bin/elastos # 2. components.json → the platform's ElastOS application-data directory +# 3. model-catalog.json → the same data directory, when components.json pins one # # After bootstrap, setup installs the Home profile and opens browser Home. # Use --install-only for automated provisioning or other profiles. @@ -34,8 +35,9 @@ # 3. Follow latest_release_cid to release.json # 4. Verify release signature # 5. Download binary + components.json, verify SHA-256 -# 6. Install to ~/.local/bin/elastos + the platform's ElastOS data directory -# 7. Save trusted-source Carrier metadata for later `setup` and `update` +# 6. Download the pinned model catalog when components.json names one, verify its head +# 7. Install to ~/.local/bin/elastos + the platform's ElastOS data directory +# 8. Save trusted-source Carrier metadata for later `setup` and `update` # # Fails closed if trust anchors or signature verification fail, unless the # operator explicitly selects --allow-unsigned. @@ -140,6 +142,7 @@ show_help() { echo -e "${BOLD}What gets installed:${NC}" echo " ~/.local/bin/elastos Runtime binary" echo " \${XDG_DATA_HOME:-~/.local/share}/elastos/components.json Capsule registry" + echo " \${XDG_DATA_HOME:-~/.local/share}/elastos/model-catalog.json Signed model catalog, when pinned" echo " macOS registry: ~/Library/Application Support/elastos/components.json" echo "" echo -e "${BOLD}After installation:${NC}" @@ -970,6 +973,37 @@ fi info "Verifying components.json SHA-256..." sha256_check "${TMPDIR}/components.json" "$COMPONENTS_SHA256" +# ── Download + verify the pinned model catalog ─────────────────────── +# The verified components.json names the catalog head. Every advertised file +# is staged and verified here; nothing running or installed changes before. + +CATALOG_HEAD=$(json_get "${TMPDIR}/components.json" '(d.get("model_catalog") or {}).get("head_cid")') +if [[ -n "$CATALOG_HEAD" ]]; then + if [[ -n "$PUBLISHER_GATEWAY" ]]; then + info "Downloading model catalog from bootstrap publisher URL" + curl -fsSL --max-time 30 -o "${TMPDIR}/model-catalog.json" "${PG}/artifacts/model-catalog.json" \ + || die "Failed to download model catalog from ${PG}/artifacts/model-catalog.json; the current installation was preserved" + else + info "Downloading model catalog by CID: ${CATALOG_HEAD} (bootstrap mode)" + ipfs_fetch "$CATALOG_HEAD" "${TMPDIR}/model-catalog.json" + fi + info "Verifying model catalog head ${CATALOG_HEAD}..." + CATALOG_HEAD="$CATALOG_HEAD" python3 - "${TMPDIR}/model-catalog.json" <<'PY' \ + || die "Downloaded model catalog does not match the pin in components.json; the current installation was preserved" +import base64 +import hashlib +import os +import pathlib +import sys + +data = pathlib.Path(sys.argv[1]).read_bytes() +expected = os.environ["CATALOG_HEAD"] +head = "b" + base64.b32encode(b"\x01\x55\x12\x20" + hashlib.sha256(data).digest()).decode("ascii").lower().rstrip("=") +if head != expected: + raise SystemExit(f"model catalog head {head} does not match pin {expected}") +PY +fi + # ── Install (2 files) ──────────────────────────────────────────────── DATA_DIR="$(installer_data_dir "$HOME" "${XDG_DATA_HOME:-}")" @@ -1030,6 +1064,12 @@ fi info "Installing components.json to ${DATA_DIR}/..." cp "${TMPDIR}/components.json" "${DATA_DIR}/components.json" +if [[ -n "$CATALOG_HEAD" ]]; then + info "Installing signed model catalog ${CATALOG_HEAD} to ${DATA_DIR}/..." + (umask 077; cp "${TMPDIR}/model-catalog.json" "${DATA_DIR}/model-catalog.json") + chmod 600 "${DATA_DIR}/model-catalog.json" +fi + PRINCIPAL_ROOT_BACKUP_DIR="${DATA_DIR}/backups/principal-root-upgrade-$(date -u +%s)-$$" info "Verifying and upgrading configured protected roots while Runtime is stopped..." "${INSTALL_DIR}/elastos" principal-root-upgrade \ From 4f877493d07371764a4d45655d0eb6fc02a849c8 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 20:17:46 -0500 Subject: [PATCH 337/535] fix(publisher): reuse the host Cargo cache for native Mac preparation Native Apple Silicon preparation now builds with the ordinary host target after proving rustc host matches, so D1 does not create a second cache. --- scripts/prepare-release-platform.sh | 18 +++- scripts/publish-release.sh | 89 +++++++++++++-- scripts/release-platform-input-test.py | 143 +++++++++++++++++++++++++ scripts/release-platform-input.py | 93 ++++++++++++---- 4 files changed, 308 insertions(+), 35 deletions(-) diff --git a/scripts/prepare-release-platform.sh b/scripts/prepare-release-platform.sh index fad6532d7..752bae9cb 100755 --- a/scripts/prepare-release-platform.sh +++ b/scripts/prepare-release-platform.sh @@ -134,20 +134,29 @@ for value in sys.argv[1:]: PY export CARGO_TARGET_DIR ELASTOS_RELEASE_VERSION="$VERSION" rustup target list --installed | grep -Fxq "$TARGET" || die "Required Rust target is not installed: $TARGET" +# Native macOS uses the same Cargo cache as normal source-home builds. Linux +# keeps its explicit musl target; the receipt still records the actual target. +BUILD_TARGET="$TARGET" +if [[ "$PLATFORM" == aarch64-darwin ]]; then + [[ "$(rustc -vV | sed -n 's/^host: //p')" == "$TARGET" ]] || die "Rust host differs from native Mac target" + BUILD_TARGET="" +fi +BUILD_TARGET_ARGS=() +[[ -z "$BUILD_TARGET" ]] || BUILD_TARGET_ARGS=(--target "$BUILD_TARGET") RELEASE_PREPARE_LOCKED=true RELEASE_PREPARE_SOURCE_COMMIT="$SOURCE_COMMIT" SKIP_BUILD=false ARTIFACTS_DIR="" info "Preparing ${PLATFORM} from ${SOURCE_COMMIT} (native cache: ${CARGO_TARGET_DIR})" -(cd elastos && cargo build --locked --release --target "$TARGET" -p elastos-server --bin elastos) -RUNTIME="$CARGO_TARGET_DIR/$TARGET/release/elastos" +(cd elastos && cargo build --locked --release ${BUILD_TARGET_ARGS[@]+"${BUILD_TARGET_ARGS[@]}"} -p elastos-server --bin elastos) +RUNTIME="$CARGO_TARGET_DIR/${BUILD_TARGET:+$BUILD_TARGET/}release/elastos" [[ -x "$RUNTIME" ]] || die "Built Runtime is missing: $RUNTIME" if [[ "$PLATFORM" == *-linux ]]; then scripts/audit-linux-runtime-portability.sh --platform "$PLATFORM" --binary "$RUNTIME" --label "prepared native Runtime" fi assert_runtime_binary_embeds_release_version "$RUNTIME" "$PLATFORM" "$VERSION" cp "$RUNTIME" "$STAGING/artifacts/elastos-$PLATFORM" -NATIVE_ASSETS=$(build_supported_direct_assets "$PLATFORM" "$SETUP_PLATFORM" "$TARGET") +NATIVE_ASSETS=$(build_supported_direct_assets "$PLATFORM" "$SETUP_PLATFORM" "$BUILD_TARGET") APP_ASSETS=$(build_platform_independent_direct_assets "$PLATFORM") PROVIDER_METADATA=$(build_platform_independent_provider_capsule_metadata_assets) DIRECT_ASSETS=$(merge_direct_assets "$(merge_direct_assets "$NATIVE_ASSETS" "$APP_ASSETS")" "$PROVIDER_METADATA") @@ -178,6 +187,9 @@ for asset in "$TMPDIR/supported-assets-$PLATFORM"/* \ [[ -f "$asset" ]] || continue cp "$asset" "$STAGING/artifacts/" done +if [[ -f "$SOURCE_ROOT/model-catalog.json" ]]; then + cp "$SOURCE_ROOT/model-catalog.json" "$STAGING/artifacts/model-catalog.json" +fi python3 scripts/release-platform-input.py record \ --root "$STAGING" --version "$VERSION" --platform "$PLATFORM" --target "$TARGET" \ --source-commit "$SOURCE_COMMIT" --source-tree "$SOURCE_TREE" \ diff --git a/scripts/publish-release.sh b/scripts/publish-release.sh index 32cb506df..ec266190b 100755 --- a/scripts/publish-release.sh +++ b/scripts/publish-release.sh @@ -77,6 +77,7 @@ DEFAULT_CAPSULES=( inbox assistant elacity-player + model-provider ) CAPSULES=("${DEFAULT_CAPSULES[@]}") REQUIRED_SUPPORTED_CAPSULES=( @@ -110,6 +111,7 @@ REQUIRED_SUPPORTED_CAPSULES=( inbox assistant elacity-player + model-provider ) SUPPORT_BINARY_ASSETS=( shell @@ -177,6 +179,7 @@ show_help() { echo " --allow-signer-rotation Allow signer DID to differ from the current canonical publisher signer" echo " --gateway-addr Gateway listen addr for auto-public URL (default: 127.0.0.1:8090)" echo " --platform-input PLATFORM=DIR Import each of the three reviewed native inputs (repeat three times)" + echo " --preview-platform PLATFORM Canary preview from exactly one native input for PLATFORM, published from that host" echo " --cross ARCH Also cross-compile for ARCH (e.g., aarch64). Creates multi-platform release." echo " --public-timeout Seconds to wait for trycloudflare URL (default: 60)" echo " --help Show this help" @@ -186,7 +189,7 @@ show_help() { echo -e "${BOLD}Output:${NC}" echo " Publishes per-capsule artifacts + runtime binary as raw CIDs." echo " Creates signed release.json/release-head.json plus content-object sidecars." - echo " Installer downloads only: binary + components.json (2 files)." + echo " Installer downloads the Runtime, components.json, and the pinned model catalog." echo " Use --cross aarch64 when publishing from x86_64 for Jetson installs." echo "" echo -e "${BOLD}Trust model:${NC}" @@ -282,10 +285,15 @@ canonical_publisher_gateway() { inspect_signer_did() { if [[ -n "${PREPARED_INPUT_ROOT:-}" ]]; then + # The shell has already admitted the native inputs. The prepared Runtime's + # dry run only reports the key's signer; a preview passes no inputs because + # the Runtime plan admits three platforms and this run selects one. local value args=() - for value in "${PLATFORM_INPUTS[@]}"; do args+=(--platform-input "$value"); done + if [[ -z "$PREVIEW_PLATFORM" ]]; then + for value in "${PLATFORM_INPUTS[@]}"; do args+=(--platform-input "$value"); done + fi "$ELASTOS" publish-release --version "$VERSION" --channel "$CHANNEL" \ - --key "$KEY_PATH" --dry-run "${args[@]}" \ + --key "$KEY_PATH" --dry-run ${args[@]+"${args[@]}"} \ | sed -n 's/^ Signer:[[:space:]]*//p' | head -n1 return fi @@ -975,6 +983,9 @@ stage_release_artifacts() { base=$(basename "$f" .capsule.tar.gz) cp -f "$f" "${artifact_dir}/${base}-${PLATFORM}.capsule.tar.gz" done + if [[ -f model-catalog.json ]]; then + cp -f model-catalog.json "${artifact_dir}/model-catalog.json" + fi if [[ -n "${CROSS_PLATFORM:-}" ]]; then CROSS_SRC="${TMPDIR}/artifacts-${CROSS_ARCH}" [ -d "$CROSS_SRC" ] || CROSS_SRC="artifacts-${CROSS_ARCH}" @@ -1099,16 +1110,18 @@ merge_direct_assets() { generate_components_json() { local capsule_entries="$1" local direct_assets="$2" - local external profiles + local external profiles catalog external=$(jq '.external' components.json) profiles=$(jq '.profiles' components.json) + catalog=$(jq -c '.model_catalog // null' components.json) jq -n \ --arg schema "elastos.components/v1" \ --argjson capsules "$capsule_entries" \ --argjson external "$external" \ --argjson profiles "$profiles" \ --argjson direct "$direct_assets" \ - '{schema: $schema, capsules: $capsules, external: ($external * $direct.external), profiles: $profiles}' + --argjson catalog "$catalog" \ + '{schema: $schema, capsules: $capsules, external: ($external * $direct.external), profiles: $profiles} + (if $catalog == null then {} else {model_catalog: $catalog} end)' } runtime_tunnel_url() { @@ -1142,13 +1155,13 @@ stage_platform_inputs() { local value args=() for value in "$@"; do args+=(--input "$value"); done python3 scripts/release-platform-input.py stage-inputs \ - --version "$VERSION" --output "$output" "${args[@]}" + --version "$VERSION" --output "$output" "${args[@]}" ${PREVIEW_ARGS[@]+"${PREVIEW_ARGS[@]}"} } publish_prepared_platform_inputs() { local root="$1" local path cid components_cid components_sha components_size - python3 scripts/release-platform-input.py verify-staged "$root" || return + python3 scripts/release-platform-input.py verify-staged "$root" ${PREVIEW_ARGS[@]+"${PREVIEW_ARGS[@]}"} || return : > "${TMPDIR}/input-cids.jsonl" while IFS= read -r path; do cid=$(ipfs_add "$root/artifacts/$path") || return @@ -1156,7 +1169,7 @@ publish_prepared_platform_inputs() { done < <(jq -r '.files | keys[]' "$root/assembly.json") jq -s 'add' "${TMPDIR}/input-cids.jsonl" > "${TMPDIR}/input-cids.json" || return python3 scripts/release-platform-input.py attach-cids "$root" \ - --cids "${TMPDIR}/input-cids.json" || return + --cids "${TMPDIR}/input-cids.json" ${PREVIEW_ARGS[@]+"${PREVIEW_ARGS[@]}"} || return PREPARED_ARTIFACTS_DIR="$root/artifacts" components_cid=$(ipfs_add "$PREPARED_ARTIFACTS_DIR/components-${PLATFORM}.json") || return components_sha=$(sha256 "$PREPARED_ARTIFACTS_DIR/components-${PLATFORM}.json") || return @@ -1287,6 +1300,30 @@ try: if errors: raise ValueError("; ".join(errors)) extra = sorted(set(present) - referenced) + pin = None + for name in present: + if name.startswith("components-") and name.endswith(".json"): + try: + candidate = json.loads(present[name].read_bytes()).get("model_catalog") + except ValueError as exc: + raise ValueError(f"{name} is not valid JSON: {exc}") + if pin is None: + pin = candidate + elif pin != candidate: + raise ValueError("prepared components disagree on model_catalog") + if isinstance(pin, dict): + head = pin.get("head_cid") + if not isinstance(head, str) or not head: + raise ValueError("model_catalog.head_cid is required") + if "model-catalog.json" not in present: + raise ValueError("advertised model catalog pin is missing model-catalog.json") + data = present["model-catalog.json"].read_bytes() + digest = hashlib.sha256(data).digest() + actual = "b" + __import__("base64").b32encode(b"\x01\x55\x12\x20" + digest).decode("ascii").lower().rstrip("=") + if actual != head: + raise ValueError(f"model-catalog.json head {actual} does not match pin {head}") + referenced.add("model-catalog.json") + extra = sorted(set(present) - referenced) if extra: raise ValueError(f"prepared artifacts are not advertised by this release: {extra}") except (OSError, ValueError, TypeError, AttributeError) as exc: @@ -1423,6 +1460,7 @@ GATEWAY_ADDR="127.0.0.1:8090" PUBLIC_URL_TIMEOUT=60 CROSS_ARCH="" PLATFORM_INPUTS=() +PREVIEW_PLATFORM="" CAPSULES_EXPLICIT=false STATE_DIR="${ELASTOS_PUBLISH_STATE_DIR:-.}" @@ -1463,6 +1501,9 @@ while [[ $# -gt 0 ]]; do [[ "$2" == *=* && -n "$input_path" ]] || die "Usage: --platform-input PLATFORM=DIR" [[ "$input_path" == /* ]] || input_path="$PUBLISH_CALLER_DIR/$input_path" PLATFORM_INPUTS+=("${input_name}=${input_path}"); shift 2 ;; + --preview-platform) + [[ -z "${2:-}" ]] && die "Usage: --preview-platform PLATFORM" + PREVIEW_PLATFORM="$2"; shift 2 ;; --capsules) CAPSULES_EXPLICIT=true [[ -z "${2:-}" ]] && die "Usage: --capsules name1,name2,..." @@ -1491,18 +1532,30 @@ if [[ ${#PLATFORM_INPUTS[@]} -gt 0 ]]; then [[ "$SKIP_BUILD" == false && "$SKIP_ROOTFS" == false && -z "$CROSS_ARCH" && "$CAPSULES_EXPLICIT" == false ]] \ || die "--platform-input conflicts with --skip-build, --skip-rootfs, --cross and --capsules" fi +# A preview publishes one explicitly named native input on the canary channel +# from that platform's own host. Default admission keeps all three platforms. +PREVIEW_ARGS=() +if [[ -n "$PREVIEW_PLATFORM" ]]; then + [[ "$CHANNEL" == canary ]] || die "--preview-platform requires --channel canary" + [[ ${#PLATFORM_INPUTS[@]} -eq 1 && "${PLATFORM_INPUTS[0]%%=*}" == "$PREVIEW_PLATFORM" ]] \ + || die "--preview-platform ${PREVIEW_PLATFORM} requires exactly one --platform-input ${PREVIEW_PLATFORM}=DIR" + PREVIEW_ARGS=(--preview-platform "$PREVIEW_PLATFORM") +fi TMPDIR=$(mktemp -d) trap 'rm -rf "$TMPDIR"' EXIT PREPARED_INPUT_ROOT="" if [[ ${#PLATFORM_INPUTS[@]} -gt 0 ]]; then PREPARED_INPUT_ROOT="${TMPDIR}/native-inputs" - stage_platform_inputs "$PREPARED_INPUT_ROOT" "${PLATFORM_INPUTS[@]}" case "$(uname -s):$(uname -m)" in Linux:x86_64) PLATFORM=x86_64-linux; SETUP_PLATFORM=linux-amd64 ;; Linux:aarch64|Linux:arm64) PLATFORM=aarch64-linux; SETUP_PLATFORM=linux-arm64 ;; Darwin:arm64|Darwin:aarch64) PLATFORM=aarch64-darwin; SETUP_PLATFORM=darwin-arm64 ;; *) die "Prepared input publication requires a supported native coordinator" ;; esac + if [[ -n "$PREVIEW_PLATFORM" && "$PLATFORM" != "$PREVIEW_PLATFORM" ]]; then + die "--preview-platform ${PREVIEW_PLATFORM} must be published from a ${PREVIEW_PLATFORM} host (this host is ${PLATFORM})" + fi + stage_platform_inputs "$PREPARED_INPUT_ROOT" "${PLATFORM_INPUTS[@]}" ELASTOS="$PREPARED_INPUT_ROOT/artifacts/elastos-${PLATFORM}" HOST_DATA_DIR="$(default_elastos_data_dir)" if [[ -z "$IPFS_PROVIDER_BIN" ]]; then @@ -2205,6 +2258,22 @@ COMPONENTS_JSON=$(generate_components_json "$CAPSULE_ENTRIES" "$HOST_DIRECT_ASSE echo "$COMPONENTS_JSON" > "${TMPDIR}/components.json" validate_generated_components_json "${TMPDIR}/components.json" "$SETUP_PLATFORM" +python3 - "${TMPDIR}/components.json" model-catalog.json <<'PY' +import base64 +import hashlib +import json +import pathlib +import sys + +manifest = json.loads(pathlib.Path(sys.argv[1]).read_bytes()) +pin = manifest.get("model_catalog") +if not isinstance(pin, dict) or not isinstance(pin.get("head_cid"), str) or not pin["head_cid"]: + raise SystemExit("generated components.json dropped model_catalog.head_cid") +data = pathlib.Path(sys.argv[2]).read_bytes() +head = "b" + base64.b32encode(b"\x01\x55\x12\x20" + hashlib.sha256(data).digest()).decode("ascii").lower().rstrip("=") +if head != pin["head_cid"]: + raise SystemExit(f"model-catalog.json head {head} does not match pin {pin['head_cid']}") +PY COMPONENTS_SHA256=$(sha256 "${TMPDIR}/components.json") COMPONENTS_SIZE=$(file_size "${TMPDIR}/components.json") @@ -2924,7 +2993,7 @@ if [[ -n "$PREPARED_INPUT_ROOT" ]]; then else echo -e "${DIM} Capsule artifacts published: ${TOTAL_ARTIFACTS} (${#CAPSULES[@]} capsules × $([ -n "$CROSS_BINARY_CID" ] && echo "2 platforms" || echo "1 platform"))${NC}" fi -echo -e "${DIM} Installer downloads: binary + components.json (2 files, platform-specific)${NC}" +echo -e "${DIM} Installer downloads: Runtime, components.json, and the pinned model catalog${NC}" echo -e "${DIM} Native setup assets are stamped per platform in components.json${NC}" echo -e "${DIM} Browser/static/WASM capsule assets are stamped once under '*' in components.json${NC}" echo -e "${DIM} Capsules downloaded on-demand by supervisor${NC}" diff --git a/scripts/release-platform-input-test.py b/scripts/release-platform-input-test.py index 6af999006..79efb0849 100644 --- a/scripts/release-platform-input-test.py +++ b/scripts/release-platform-input-test.py @@ -7,6 +7,7 @@ import io import json import os +import platform import shutil from pathlib import Path import struct @@ -269,6 +270,148 @@ def test_actual_publisher_import_success_and_effect_boundaries(self): self.assertNotIn("shell_sha256", release) self.assertTrue((prepared / "later-effect").exists()) + def pin_catalogue(self): + """Pin a signed catalogue snapshot in the template and every bundle, as the candidate source does.""" + catalog = b'{"payload":{"schema":"elastos.model.catalog/v1","entries":[]},"signature":"00","signer_did":"did:key:z6Mkfixture"}\n' + pin = {"head_cid": inputs.catalog_head_cid(catalog), "publisher_dids": ["did:key:z6Mkfixture"]} + self.template["model_catalog"] = pin + self.write_json(inputs.SOURCE_ROOT / "components.json", self.template) + for root in self.bundles.values(): + manifest = json.loads((root / "components.json").read_text()) + manifest["model_catalog"] = pin + self.write_json(root / "components.json", manifest) + self.write_json(root / "components-template.json", self.template) + (root / "artifacts/model-catalog.json").write_bytes(catalog) + self.refresh(root) + return pin + + def test_default_admission_rejects_a_single_platform_input(self): + darwin = [f"aarch64-darwin={self.bundles['aarch64-darwin']}"] + stage = self.root / "publication" + with self.assertRaisesRegex(ValueError, "all three platforms"): + inputs.validate_inputs(darwin) + with self.assertRaisesRegex(ValueError, "all three platforms"): + inputs.stage_inputs(darwin, "0.7.1", stage) + self.assertFalse(stage.exists()) + + def test_mac_preview_admits_one_input_and_keeps_catalogue_and_selected_platform(self): + pin = self.pin_catalogue() + darwin = [f"aarch64-darwin={self.bundles['aarch64-darwin']}"] + linux = [f"x86_64-linux={self.bundles['x86_64-linux']}"] + self.assertEqual(set(inputs.validate_inputs(darwin, "0.7.1", "aarch64-darwin")), {"aarch64-darwin"}) + for values, preview, message in [ + (darwin, "x86_64-linux", "outside the selected publication"), + (darwin + linux, "aarch64-darwin", "outside the selected publication"), + ([], "aarch64-darwin", "exactly one aarch64-darwin input"), + (darwin, "riscv64-linux", "not a release platform")]: + with self.subTest(preview=preview, count=len(values)): + with self.assertRaisesRegex(ValueError, message): + inputs.validate_inputs(values, "0.7.1", preview) + stage = self.root / "publication" + record = inputs.stage_inputs(darwin, "0.7.1", stage, "aarch64-darwin") + self.assertEqual(record["platforms"], ["aarch64-darwin"]) + self.assertEqual(set(record["files"]), {"elastos-aarch64-darwin", "shell-darwin-arm64", "home.tar.gz", + "shell-metadata.tar.gz", "model-catalog.json"}) + self.assertEqual(json.loads((stage / "assembly.json").read_text())["platforms"], ["aarch64-darwin"]) + inputs.verify_staged_inputs(stage, preview_platform="aarch64-darwin") + with self.assertRaisesRegex(ValueError, "all three platforms"): + inputs.verify_staged_inputs(stage) + with self.assertRaisesRegex(ValueError, "not the x86_64-linux preview"): + inputs.verify_staged_inputs(stage, preview_platform="x86_64-linux") + merged = json.loads((stage / "components.json").read_text()) + self.assertEqual(merged["model_catalog"], pin) + self.assertEqual(set(merged["external"]["shell"]["platforms"]), {"darwin-arm64"}) + cids = self.root / "cids.json" + self.write_json(cids, {name: "bafy" + entry["sha256"] for name, entry in record["files"].items()}) + with self.assertRaisesRegex(ValueError, "all three platforms"): + inputs.attach_input_cids(stage, cids) + inputs.attach_input_cids(stage, cids, "aarch64-darwin") + generated = sorted(path.name for path in (stage / "artifacts").glob("components-*.json")) + self.assertEqual(generated, ["components-aarch64-darwin.json"]) + final = json.loads((stage / "artifacts/components-aarch64-darwin.json").read_text()) + self.assertEqual(final["model_catalog"], pin) + self.assertTrue(final["external"]["shell"]["platforms"]["darwin-arm64"]["cid"].startswith("bafy")) + self.assertEqual((stage / "artifacts/model-catalog.json").read_bytes(), + (self.bundles["aarch64-darwin"] / "artifacts/model-catalog.json").read_bytes()) + # A tampered input is refused at admission, before any staging output exists. + binary_path = self.bundles["aarch64-darwin"] / "artifacts/elastos-aarch64-darwin" + binary_path.write_bytes(binary_path.read_bytes()[:-1] + b"!") + with self.assertRaisesRegex(ValueError, "differs from receipt"): + inputs.stage_inputs(darwin, "0.7.1", self.root / "publication-2", "aarch64-darwin") + self.assertFalse((self.root / "publication-2").exists()) + + def test_actual_publisher_preview_publishes_only_the_selected_platform(self): + pin = self.pin_catalogue() + source = self.actual_source_fixture() + publisher = (source / "scripts/publish-release.sh").read_text() + payload = "RELEASE_PAYLOAD=" + publisher.split("\nRELEASE_PAYLOAD=", 1)[1].split( + '\ninfo "Signing release payload', 1)[0] + work = self.root / "publisher-preview" + work.mkdir() + body = r'''source "$1" +VERSION=0.7.1 +CHANNEL=canary +TMPDIR="$2/work" +mkdir "$TMPDIR" +PLATFORM=aarch64-darwin +PREVIEW_PLATFORM=aarch64-darwin +PREVIEW_ARGS=(--preview-platform aarch64-darwin) +shift 2 +stage_platform_inputs "$TMPDIR/staged" "$@" +ipfs_add() { + printf '%s\n' "$(basename "$1")" >> "$TMPDIR/uploads" + printf 'bafy%s\n' "$(sha256 "$1")" +} +publish_prepared_platform_inputs "$TMPDIR/staged" +printf '%s\n' "$PLATFORMS_JSON" > "$TMPDIR/platforms.json" +PREV_RELEASE_CID=null +''' + payload + r''' +printf '%s\n' "$RELEASE_PAYLOAD" > "$TMPDIR/release-payload.json" +''' + result = subprocess.run(["/bin/bash", "-euc", body, "publisher-fixture", + str(source / "scripts/publish-release.sh"), str(work), + f"aarch64-darwin={self.bundles['aarch64-darwin']}"], + capture_output=True, text=True) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + prepared = work / "work" + platforms = json.loads((prepared / "platforms.json").read_text()) + self.assertEqual(set(platforms), {"aarch64-darwin"}) + self.assertEqual(sorted((prepared / "uploads").read_text().splitlines()), + ["components-aarch64-darwin.json", "elastos-aarch64-darwin", "home.tar.gz", + "model-catalog.json", "shell-darwin-arm64", "shell-metadata.tar.gz"]) + release = json.loads((prepared / "release-payload.json").read_text()) + self.assertEqual((release["channel"], set(release["platforms"])), ("canary", {"aarch64-darwin"})) + self.assertEqual(json.loads((prepared / "components.json").read_text())["model_catalog"], pin) + self.assertEqual(platforms["aarch64-darwin"]["binary"]["sha256"], + inputs.digest(self.bundles["aarch64-darwin"] / "artifacts/elastos-aarch64-darwin")) + + def test_actual_publisher_preview_rejects_channel_input_count_and_host(self): + source = self.actual_source_fixture() + darwin = f"aarch64-darwin={os.path.relpath(self.bundles['aarch64-darwin'], self.root)}" + linux = f"x86_64-linux={os.path.relpath(self.bundles['x86_64-linux'], self.root)}" + base = ["/bin/bash", str(source / "scripts/publish-release.sh"), "--version", "0.7.1", + "--key", str(self.root / "missing-key"), "--preview-platform", "aarch64-darwin"] + state = self.root / "publisher-state" + cases = [ + ("stable channel", ["--channel", "stable", "--platform-input", darwin], "requires --channel canary"), + ("no input", ["--channel", "canary"], "exactly one --platform-input aarch64-darwin=DIR"), + ("two inputs", ["--channel", "canary", "--platform-input", darwin, "--platform-input", linux], + "exactly one --platform-input aarch64-darwin=DIR"), + ("other platform input", ["--channel", "canary", "--platform-input", linux], + "exactly one --platform-input aarch64-darwin=DIR"), + ] + if platform.system() != "Darwin": + cases.append(("wrong host", ["--channel", "canary", "--platform-input", darwin], + "must be published from a aarch64-darwin host")) + for name, options, message in cases: + with self.subTest(case=name): + result = subprocess.run(base + options, cwd=self.root, + env={**os.environ, "ELASTOS_PUBLISH_STATE_DIR": str(state)}, + capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn(message, result.stderr) + self.assertFalse(state.exists()) + def test_actual_publisher_rejects_input_before_state_or_key_inspection(self): source = self.actual_source_fixture() state = self.root / "publisher-state" diff --git a/scripts/release-platform-input.py b/scripts/release-platform-input.py index 5f8b1d2a8..62828b759 100644 --- a/scripts/release-platform-input.py +++ b/scripts/release-platform-input.py @@ -2,6 +2,7 @@ """Record and verify unsigned native build inputs. This tool never publishes.""" import argparse +import base64 import copy from datetime import datetime, timezone import hashlib @@ -60,6 +61,23 @@ def regular_file(root, relative): return path +def catalog_head_cid(data): + return "b" + base64.b32encode(b"\x01\x55\x12\x20" + hashlib.sha256(data).digest()).decode("ascii").lower().rstrip("=") + + +def admit_model_catalog_artifact(manifest, artifact_root, referenced): + pin = manifest.get("model_catalog") + if pin is None: + return + if not isinstance(pin, dict) or not isinstance(pin.get("head_cid"), str) or not pin["head_cid"]: + raise ValueError("model_catalog.head_cid is required") + data = regular_file(artifact_root, "model-catalog.json").read_bytes() + actual = catalog_head_cid(data) + if actual != pin["head_cid"]: + raise ValueError(f"model-catalog.json head {actual} does not match pin {pin['head_cid']}") + referenced.add("model-catalog.json") + + def file_record(path): return {"sha256": digest(path), "size": path.stat().st_size, "executable": bool(path.stat().st_mode & 0o111)} @@ -301,6 +319,7 @@ def check_contents(root, platform, omissions): check_binary(regular_file(root / "artifacts", f"elastos-{platform}"), platform) actual = {str(path.relative_to(root / "artifacts")) for path in (root / "artifacts").rglob("*") if not path.is_dir() or path.is_symlink()} + admit_model_catalog_artifact(manifest, root / "artifacts", referenced) if actual != referenced: raise ValueError(f"artifact inventory mismatch: {sorted(actual ^ referenced)}") return manifest, template @@ -397,12 +416,24 @@ def verify(root): return receipt -def validate_inputs(values, version=None): +def selected_platforms(preview_platform=None): + """The platform set one publication admits: every release platform, or one explicit preview.""" + if preview_platform is None: + return set(PLATFORMS) + if preview_platform not in PLATFORMS: + raise ValueError(f"preview platform is not a release platform: {preview_platform}") + return {preview_platform} + + +def validate_inputs(values, version=None, preview_platform=None): + selected = selected_platforms(preview_platform) inputs = {} for value in values: name, sep, path = value.partition("=") if not sep or name not in PLATFORMS or name in inputs: raise ValueError("inputs require each full release platform exactly once") + if name not in selected: + raise ValueError(f"input platform is outside the selected publication: {name}") receipt = verify(Path(path)) if receipt["platform"] != name: raise ValueError("input label differs from receipt platform") @@ -420,8 +451,10 @@ def validate_inputs(values, version=None): elif not any(info.get(key) for key in ("release_path", "url")): raise ValueError(f"{name}: required Home component has no prepared delivery path: {component_name}") inputs[name] = receipt - if set(inputs) != set(PLATFORMS): - raise ValueError("candidate input requires all three platforms") + if set(inputs) != selected: + if preview_platform is None: + raise ValueError("candidate input requires all three platforms") + raise ValueError(f"preview input requires exactly one {preview_platform} input") first = next(iter(inputs.values())) if version is not None and first["version"] != version: raise ValueError("platform input version differs from requested release") @@ -482,11 +515,20 @@ def merged_input_components(values, receipts): if {k: v for k, v in target.items() if k != "platforms"} != contract: raise ValueError(f"conflicting provider metadata contract: {name}") target.setdefault("platforms", {})[key] = copy.deepcopy(info) + # A publication advertises only the platforms it admitted. Template + # descriptors for the other release platforms would name artifacts this + # publication never stages, so they leave the merged manifest here. + unselected = {PLATFORMS[platform][0] for platform in PLATFORMS if platform not in receipts} + for component in merged["external"].values(): + for entry in (component, component.get("capsule_metadata")): + if isinstance(entry, dict): + for setup in unselected: + entry.get("platforms", {}).pop(setup, None) return merged -def stage_inputs(values, version, output): - receipts = validate_inputs(values, version) +def stage_inputs(values, version, output, preview_platform=None): + receipts = validate_inputs(values, version, preview_platform) if output.exists() or output.is_symlink(): raise ValueError("publication input staging output already exists") merged = merged_input_components(values, receipts) @@ -516,24 +558,26 @@ def stage_inputs(values, version, output): (stage / "components.json").write_text(json.dumps(merged, indent=2) + "\n") first = next(iter(receipts.values())) record = {"version": version, "source": first["source"], - "platforms": list(PLATFORMS), "files": files, + "platforms": sorted(receipts), "files": files, "components": file_record(stage / "components.json")} (stage / "assembly.json").write_text(json.dumps(record, indent=2) + "\n") - verify_staged_inputs(stage) + verify_staged_inputs(stage, preview_platform=preview_platform) stage.rename(output) return record -def verify_staged_inputs(stage, allow_generated=False): +def verify_staged_inputs(stage, allow_generated=False, preview_platform=None): record = json.loads(regular_file(stage, "assembly.json").read_text()) source = record["source"] if source_identity(source["commit"], source["tree"]) != source: raise ValueError("publication staging source differs from candidate checkout") - if set(record["platforms"]) != set(PLATFORMS): - raise ValueError("publication staging requires all three platforms") + if set(record["platforms"]) != selected_platforms(preview_platform): + if preview_platform is None: + raise ValueError("publication staging requires all three platforms") + raise ValueError(f"publication staging is not the {preview_platform} preview") actual = {str(path.relative_to(stage / "artifacts")) for path in (stage / "artifacts").rglob("*") if not path.is_dir() or path.is_symlink()} - generated = {f"components-{platform}.json" for platform in PLATFORMS} if allow_generated else set() + generated = {f"components-{platform}.json" for platform in record["platforms"]} if allow_generated else set() if not set(record["files"]) <= actual or actual - set(record["files"]) - generated: raise ValueError("staged artifact inventory differs from admitted inputs") for name, expected in record["files"].items(): @@ -543,7 +587,8 @@ def verify_staged_inputs(stage, allow_generated=False): if file_record(components) != record["components"]: raise ValueError("staged components changed after admission") manifest = json.loads(components.read_text()) - for platform, (setup, _, _) in PLATFORMS.items(): + for platform in record["platforms"]: + setup = PLATFORMS[platform][0] check_binary(regular_file(stage / "artifacts", f"elastos-{platform}"), platform) errors = integrity.audit_manifest(manifest, [setup]) errors += integrity.audit_release_artifacts(manifest, [setup], stage / "artifacts") @@ -552,8 +597,8 @@ def verify_staged_inputs(stage, allow_generated=False): return record -def attach_input_cids(stage, cids_path): - record = verify_staged_inputs(stage, allow_generated=True) +def attach_input_cids(stage, cids_path, preview_platform=None): + record = verify_staged_inputs(stage, allow_generated=True, preview_platform=preview_platform) cids = json.loads(cids_path.read_text()) if set(cids) != set(record["files"]) or any( not isinstance(cid, str) or not re.fullmatch(r"[A-Za-z0-9]+", cid) for cid in cids.values()): @@ -564,10 +609,11 @@ def attach_input_cids(stage, cids_path): for info in entry.get("platforms", {}).values(): if info.get("release_path"): info["cid"] = cids[info["release_path"]] - # Each platform gets the same complete manifest. Descriptors were replaced - # as units during staging; pinned external URL records remain unchanged. + # Each selected platform gets the same complete manifest. Descriptors were + # replaced as units during staging; pinned external URL records remain unchanged. output_bytes = (json.dumps(manifest, indent=2) + "\n").encode() - for platform, (setup, _, _) in PLATFORMS.items(): + for platform in record["platforms"]: + setup = PLATFORMS[platform][0] existing = stage / "artifacts" / f"components-{platform}.json" if existing.exists() or existing.is_symlink(): if regular_file(stage / "artifacts", existing.name).read_bytes() != output_bytes: @@ -576,7 +622,7 @@ def attach_input_cids(stage, cids_path): errors += integrity.audit_release_artifacts(manifest, [setup], stage / "artifacts") if errors: raise ValueError("; ".join(errors)) - for platform in PLATFORMS: + for platform in record["platforms"]: output = stage / "artifacts" / f"components-{platform}.json" if not output.exists(): output.write_bytes(output_bytes) @@ -606,6 +652,9 @@ def main(): attach = commands.add_parser("attach-cids") attach.add_argument("root", type=Path) attach.add_argument("--cids", required=True, type=Path) + for command in (combined, stage, staged, attach): + command.add_argument("--preview-platform", choices=PLATFORMS, + help="admit exactly this one native input instead of all release platforms") args = parser.parse_args() try: if args.command == "record": @@ -613,13 +662,13 @@ def main(): elif args.command == "verify": print(json.dumps(verify(args.root), sort_keys=True)) elif args.command == "stage-inputs": - stage_inputs(args.input, args.version, args.output) + stage_inputs(args.input, args.version, args.output, args.preview_platform) elif args.command == "verify-staged": - verify_staged_inputs(args.root) + verify_staged_inputs(args.root, preview_platform=args.preview_platform) elif args.command == "attach-cids": - attach_input_cids(args.root, args.cids) + attach_input_cids(args.root, args.cids, args.preview_platform) else: - receipts = validate_inputs(args.input, args.version) + receipts = validate_inputs(args.input, args.version, args.preview_platform) print(f"Verified source and local bytes for {len(receipts)} platform inputs; publication and installed acceptance remain separate.") except (ValueError, OSError, KeyError, TypeError, tarfile.TarError, subprocess.CalledProcessError) as exc: parser.exit(1, f"Error: {exc}\n") From 7aaded6555d68a8ef73b09a2ad535a2d77256cf3 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 20:18:34 -0500 Subject: [PATCH 338/535] fix(browser): keep paused remote-display, restore, and engine work on this branch D1 ships from feat/remote-services without dropping Browser ancestry. Product Browser acceptance stays open. --- capsules/browser-engine-adapter/src/main.rs | 16 +- capsules/browser-engine-adapter/src/tests.rs | 194 ++- .../browser-engine-adapter/src/transport.rs | 27 +- .../browser/browser/browser-page-cleanup.js | 6 + .../browser/browser-page-cleanup.test.mjs | 17 + .../browser/browser/browser-remote-display.js | 109 +- .../browser/browser/browser-restore-boot.js | 299 +++++ capsules/browser/browser/browser-webrtc.js | 57 + .../browser/browser/browser-webrtc.test.mjs | 47 + capsules/browser/browser/browser.js | 318 ++++- capsules/browser/browser/index.html | 17 +- docs/BROWSER_CAPSULE.md | 48 + .../src/api/browser_capsules.rs | 211 +++- .../elastos-server/src/api/gateway_browser.rs | 894 ++++++++++--- .../src/api/gateway_browser_engine.rs | 104 +- .../src/api/gateway_browser_remote.rs | 356 +++++- .../src/api/gateway_browser_route_tests.rs | 539 +++++++- .../src/api/gateway_browser_sessions.rs | 21 +- .../src/api/gateway_browser_wallet.rs | 93 +- .../src/api/gateway_browser_wallet_bridge.rs | 161 ++- .../gateway_tests/browser_reconciliation.rs | 89 ++ .../src/api/gateway_tests/remote_engine.rs | 168 ++- .../api/gateway_tests/support_providers.rs | 53 +- .../src/carrier_engine_binding.rs | 9 +- .../src/carrier_engine_media.rs | 767 +++++++++++- .../src/bin/browser-vz-engine-supervisor.rs | 899 +++++++++++++- elastos/crates/elastos-vz/src/ffi/delegate.rs | 53 + .../crates/elastos-vz/src/ffi/lifecycle.rs | 23 +- elastos/crates/elastos-vz/src/provider.rs | 38 + elastos/crates/elastos-vz/src/vm.rs | 17 + scripts/browser-display-attachment.test.mjs | 152 ++- ...browser-display-metrics-freshness.test.mjs | 13 +- ...wser-display-signaling-generation.test.mjs | 65 +- scripts/browser-entropy-check.mjs | 210 +++- scripts/browser-profile-disk-flush.test.mjs | 194 +++ scripts/browser-selkies-control-service.mjs | 616 +++++++-- scripts/browser-startup-readiness.test.mjs | 17 +- scripts/browser-viewer-metrics-read.test.mjs | 22 + scripts/browser-viewer-restore.test.mjs | 339 ++++- ...ser-vm-control-service-settlement-smoke.sh | 1101 +++++++++++++++++ scripts/browser-vm-control-service.mjs | 990 +++++++++++++-- .../browser-vm-control-stdout-ready.test.mjs | 27 + .../browser-wallet-account-consent-smoke.mjs | 50 + .../browser-window-close-handshake.test.mjs | 9 +- .../build/browser-webrtc-send-diagnostic.py | 172 +++ .../browser-webrtc-send-diagnostic.test.py | 101 ++ scripts/build/stage-browser-vm-target.sh | 84 ++ scripts/lib/browser-journey-harness.test.mjs | 159 ++- .../lib/browser-journey-turn-interruption.mjs | 44 +- ...browser-journey-turn-interruption.test.mjs | 82 ++ scripts/lib/browser-journey-viewer-reload.mjs | 423 ++++++- .../browser-journey-viewer-reload.test.mjs | 202 ++- scripts/lib/browser-vm-webrtc-apply-gate.mjs | 99 ++ .../lib/browser-vm-webrtc-apply-gate.test.mjs | 116 ++ 54 files changed, 10268 insertions(+), 669 deletions(-) create mode 100644 capsules/browser/browser/browser-restore-boot.js create mode 100644 scripts/browser-profile-disk-flush.test.mjs create mode 100644 scripts/browser-vm-control-stdout-ready.test.mjs create mode 100644 scripts/build/browser-webrtc-send-diagnostic.py create mode 100644 scripts/build/browser-webrtc-send-diagnostic.test.py create mode 100644 scripts/lib/browser-vm-webrtc-apply-gate.mjs create mode 100644 scripts/lib/browser-vm-webrtc-apply-gate.test.mjs diff --git a/capsules/browser-engine-adapter/src/main.rs b/capsules/browser-engine-adapter/src/main.rs index 14bd62f6d..6b28aec1b 100644 --- a/capsules/browser-engine-adapter/src/main.rs +++ b/capsules/browser-engine-adapter/src/main.rs @@ -674,14 +674,22 @@ fn engine_terminal_cleanup_result( "Browser supervisor did not return an exact typed terminal cleanup receipt".to_string(), ); } - Ok(json!({ + let mut terminal = json!({ "schema": BROWSER_ENGINE_CLEANUP_RESULT_SCHEMA, "page_id": binding.page_id, "generation": binding.generation, "binding": binding, "terminal": true, "effects": supervisor_receipt["effects"], - })) + }); + if let Some(durability) = supervisor_receipt + .get("profile_durability") + .and_then(Value::as_str) + .filter(|value| matches!(*value, "failed" | "unknown" | "proved")) + { + terminal["profile_durability"] = json!(durability); + } + Ok(terminal) } struct LaunchContext<'a> { @@ -1271,10 +1279,6 @@ impl BrowserEngineAdapter { if selected_adapter.kind != AdapterKind::ChromiumMicrovm || page_acquired || vm_acquired - != settlement - .pointer("/effects/vm") - .and_then(Value::as_bool) - .unwrap_or(false) || launch .and_then(|value| value.get("adapter")) .and_then(Value::as_str) diff --git a/capsules/browser-engine-adapter/src/tests.rs b/capsules/browser-engine-adapter/src/tests.rs index 63ec4c894..de62264ce 100644 --- a/capsules/browser-engine-adapter/src/tests.rs +++ b/capsules/browser-engine-adapter/src/tests.rs @@ -704,7 +704,7 @@ fn reconcile_terminal_vz_settlement(mutate: impl FnOnce(&mut Value)) -> Value { "transport_authority": transport.authority, "effects": { "page_acquired": false, - "vm_acquired": true, + "vm_acquired": false, }, "launch_settlement_result": settlement, }); @@ -2452,13 +2452,41 @@ fn launch_reconciliation_adopts_exact_terminal_vz_settlement_without_page_bindin assert_eq!(response["status"], "ok"); assert_eq!(response["data"]["state"], "terminal_post_effect_cleanup"); assert_eq!(response["data"]["effects"]["page_acquired"], false); - assert_eq!(response["data"]["effects"]["vm_acquired"], true); + assert_eq!(response["data"]["effects"]["vm_acquired"], false); assert_eq!( response["data"]["terminal_cleanup_receipt"]["schema"], "elastos.browser.vz-launch-settlement/v1" ); } +#[test] +fn launch_reconciliation_adopts_terminal_vz_settlement_with_failed_profile_durability() { + let response = reconcile_terminal_vz_settlement(|value| { + value["launch_settlement_result"]["profile_durability"] = json!("failed"); + value["control_service"]["config_fingerprint"] = json!("b".repeat(64)); + }); + + assert_eq!(response["status"], "ok"); + assert_eq!(response["data"]["state"], "terminal_post_effect_cleanup"); + assert_eq!( + response["data"]["terminal_cleanup_receipt"]["absence"]["child_absent"], + true + ); + assert_eq!( + response["data"]["terminal_cleanup_receipt"]["profile_durability"], + "failed" + ); +} + +#[test] +fn launch_reconciliation_rejects_invalid_profile_durability() { + let response = reconcile_terminal_vz_settlement(|value| { + value["launch_settlement_result"]["profile_durability"] = json!("maybe"); + }); + assert_eq!(response["status"], "ok"); + assert_eq!(response["data"]["state"], "cleanup_pending"); +} + #[test] fn launch_reconciliation_rejects_malformed_terminal_vz_settlements() { let substitutions = [ @@ -2696,6 +2724,51 @@ fn exact_typed_already_absent_supervisor_proof_is_terminal() { assert_eq!(terminal["terminal"], true); } +#[test] +fn ordinary_close_builder_keeps_failed_profile_durability() { + let session = PageControlSession { + display_attachment: BrowserDisplayAttachment::default(), + generation: "sha256:typed-durability".to_string(), + stream_id: "stream:typed-durability".to_string(), + socket_path: "/tmp/elastos-browser-typed-durability.sock".to_string(), + shutdown_socket_path: None, + adapter_id: "hosted-product".to_string(), + principal_id: Some("person:local:test".to_string()), + engine: AdapterKind::SelkiesGstreamer, + display_mode: BrowserDisplayMode::WebrtcRemoteDisplay, + guarantee_level: BrowserGuaranteeLevel::OperatorRbi, + isolated_session: false, + isolation_session_dir: None, + isolation_kind: None, + control_service: None, + process: None, + transport_authority: None, + transport_receipt: None, + }; + let binding = engine_cleanup_binding("page:typed-durability", &session); + let receipt = json!({ + "schema": BROWSER_SUPERVISOR_CLEANUP_RESULT_SCHEMA, + "page_id": binding.page_id, + "generation": binding.generation, + "binding": binding, + "terminal": true, + "profile_durability": "failed", + "effects": { + "page_absent": true, + "child_absent": true, + "vm_absent": true, + "route_absent": true, + "socket_absent": true + } + }); + + let terminal = engine_terminal_cleanup_result(&binding, receipt).unwrap(); + assert_eq!(terminal["schema"], BROWSER_ENGINE_CLEANUP_RESULT_SCHEMA); + assert_eq!(terminal["terminal"], true); + assert_eq!(terminal["effects"]["child_absent"], true); + assert_eq!(terminal["profile_durability"], "failed"); +} + #[test] fn page_operations_reject_mismatched_principal() { let mut provider = BrowserEngineAdapter::new(); @@ -3004,6 +3077,123 @@ fn isolated_close_uses_target_shutdown_contract() { handle.join().unwrap(); } +#[test] +fn ordinary_isolated_close_and_replay_keep_failed_profile_durability() { + use std::io::{Read, Write}; + use std::os::unix::net::UnixListener; + use std::thread; + + let socket_path = format!( + "/tmp/elastos-browser-isolated-durability-{}-{}.sock", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let _ = std::fs::remove_file(&socket_path); + let listener = UnixListener::bind(&socket_path).unwrap(); + let handle = thread::spawn({ + let socket_path = socket_path.clone(); + move || { + let mut shutdowns = 0; + while shutdowns < 2 { + let (mut stream, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut buffer = [0_u8; 256]; + loop { + let size = stream.read(&mut buffer).unwrap_or(0); + if size == 0 { + break; + } + request.extend_from_slice(&buffer[..size]); + if http_request_is_complete(&request) { + break; + } + } + let request = String::from_utf8_lossy(&request); + if !request.starts_with("POST /shutdown HTTP/1.1") { + let body = r#"{"schema":"elastos.browser.test-probe/v1","ok":true}"#; + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", + body.len(), + body + ) + .unwrap(); + continue; + } + let mut receipt: serde_json::Value = + serde_json::from_str(&typed_supervisor_cleanup_receipt(&request)).unwrap(); + receipt["profile_durability"] = json!("failed"); + let body = serde_json::to_string(&receipt).unwrap(); + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", + body.len(), + body + ) + .unwrap(); + shutdowns += 1; + } + let _ = std::fs::remove_file(socket_path); + } + }); + + let mut provider = BrowserEngineAdapter::new(); + provider.page_control_sessions.insert( + "page:isolated-durability".to_string(), + PageControlSession { + display_attachment: BrowserDisplayAttachment::default(), + generation: "sha256:test-generation".to_string(), + stream_id: "stream:proof:test".to_string(), + socket_path: socket_path.clone(), + shutdown_socket_path: None, + adapter_id: "hosted-product".to_string(), + principal_id: Some("person:local:test".to_string()), + engine: AdapterKind::SelkiesGstreamer, + display_mode: BrowserDisplayMode::WebrtcRemoteDisplay, + guarantee_level: BrowserGuaranteeLevel::OperatorRbi, + isolated_session: true, + isolation_session_dir: Some( + "/tmp/elastos-browser-sessions/stream_isolated-durability-test".to_string(), + ), + isolation_kind: Some("per_launch_selkies_target".to_string()), + control_service: None, + process: None, + transport_authority: None, + transport_receipt: None, + }, + ); + + let cleanup = cleanup_binding_for(&provider, "page:isolated-durability"); + let first = serde_json::to_value(provider.close_page( + "page:isolated-durability", + Some("person:local:test".to_string()), + cleanup.clone(), + )) + .unwrap(); + assert_eq!(first["status"], "ok"); + assert_eq!(first["data"]["terminal"], true); + assert_eq!(first["data"]["effects"]["child_absent"], true); + assert_eq!(first["data"]["profile_durability"], "failed"); + assert!(!provider + .page_control_sessions + .contains_key("page:isolated-durability")); + + let replay = serde_json::to_value(provider.close_page( + "page:isolated-durability", + Some("person:local:test".to_string()), + cleanup, + )) + .unwrap(); + assert_eq!(replay["status"], "ok"); + assert_eq!(replay["data"]["terminal"], true); + assert_eq!(replay["data"]["effects"]["child_absent"], true); + assert_eq!(replay["data"]["profile_durability"], "failed"); + handle.join().unwrap(); +} + #[test] fn vm_isolated_close_uses_global_shutdown_socket() { use std::io::{Read, Write}; diff --git a/capsules/browser-engine-adapter/src/transport.rs b/capsules/browser-engine-adapter/src/transport.rs index 5e09d9e9d..d9c8ae804 100644 --- a/capsules/browser-engine-adapter/src/transport.rs +++ b/capsules/browser-engine-adapter/src/transport.rs @@ -212,6 +212,23 @@ pub(super) fn validate_vz_transport_effect_receipt( Ok(()) } +fn vz_launch_settlement_keys_are_exact(settlement: &Value, keys: &[&str]) -> bool { + let Some(object) = settlement.as_object() else { + return false; + }; + keys.iter().all(|key| object.contains_key(*key)) + && object + .keys() + .all(|key| keys.contains(&key.as_str()) || key == "profile_durability") +} + +fn vz_launch_profile_durability_is_safe(settlement: &Value) -> bool { + match settlement.get("profile_durability") { + None => true, + Some(value) => matches!(value.as_str(), Some("failed" | "unknown" | "proved")), + } +} + pub(super) fn validate_vz_launch_settlement( settlement: &Value, transport: &VzTransportLaunchContext, @@ -228,9 +245,9 @@ pub(super) fn validate_vz_launch_settlement_binding( authority: &Value, ) -> Result<(), String> { validate_vz_transport_authority(authority)?; - let object = settlement - .as_object() - .ok_or_else(|| "Browser VZ launch settlement must be an object".to_string())?; + if settlement.as_object().is_none() { + return Err("Browser VZ launch settlement must be an object".to_string()); + } let keys = [ "schema", "state", @@ -283,8 +300,8 @@ pub(super) fn validate_vz_launch_settlement_binding( .filter(|value| value.len() <= 8_192 && !value.contains('\0')) .ok_or_else(|| "Browser VZ launch settlement message is invalid".to_string())?; let _ = message; - if object.len() != keys.len() - || keys.iter().any(|key| !object.contains_key(*key)) + if !vz_launch_settlement_keys_are_exact(settlement, &keys) + || !vz_launch_profile_durability_is_safe(settlement) || effects.len() != effect_keys.len() || effect_keys.iter().any(|key| !effects.contains_key(*key)) || absence.len() != absence_keys.len() diff --git a/capsules/browser/browser/browser-page-cleanup.js b/capsules/browser/browser/browser-page-cleanup.js index db7281c5b..7450bcb72 100644 --- a/capsules/browser/browser/browser-page-cleanup.js +++ b/capsules/browser/browser/browser-page-cleanup.js @@ -93,12 +93,18 @@ export function classifyRuntimePageCloseResponse(owner, response) { return pendingOutcome(owner.page_id, owner.generation, "malformed_response"); } if (response.closed === true || response.already_closed === true) { + const durability = ["failed", "unknown", "proved"].includes( + response.profile_durability, + ) + ? response.profile_durability + : undefined; return { state: "terminal", page_id: owner.page_id, generation: owner.generation, terminal_kind: response.already_closed === true ? "already_absent" : "closed", + ...(durability ? { profile_durability: durability } : {}), }; } return pendingOutcome( diff --git a/capsules/browser/browser/browser-page-cleanup.test.mjs b/capsules/browser/browser/browser-page-cleanup.test.mjs index 85a4deea3..f7df9fb2f 100644 --- a/capsules/browser/browser/browser-page-cleanup.test.mjs +++ b/capsules/browser/browser/browser-page-cleanup.test.mjs @@ -98,6 +98,23 @@ test("successful close is terminal and calls the terminal hook once", async () = assert.equal(controller.status(pageOwner).terminal, true); }); +test("failed profile durability stays on a terminal close", async () => { + const pageOwner = owner(); + const controller = createRuntimePageCleanupController({ + closePage: async () => + closeReceipt(pageOwner.page_id, { + profile_durability: "failed", + terminal_effects: { child_absent: true }, + }), + }); + + const outcome = await controller.reconcile(pageOwner); + + assert.equal(outcome.state, "terminal"); + assert.equal(outcome.terminal_kind, "closed"); + assert.equal(outcome.profile_durability, "failed"); +}); + test("Runtime-proven already-absent receipt is terminal", async () => { const pageOwner = owner(); const receiptController = createRuntimePageCleanupController({ diff --git a/capsules/browser/browser/browser-remote-display.js b/capsules/browser/browser/browser-remote-display.js index b68ca11b5..d4699bf12 100644 --- a/capsules/browser/browser/browser-remote-display.js +++ b/capsules/browser/browser/browser-remote-display.js @@ -7,7 +7,7 @@ import { sdpHasOnlyRelayCandidates, stripTrickleCandidatesFromSdp, validateRuntimeLaunchTurn, -} from "./browser-webrtc.js?v=browser-20260730a"; +} from "./browser-webrtc.js?v=browser-20260913h"; const WEBRTC_CONNECT_TIMEOUT_MS = 30000; const WEBRTC_DISCONNECT_GRACE_MS = 10000; @@ -851,29 +851,56 @@ export function createBrowserRemoteDisplay({ await nextAudioPeerConnection.setLocalDescription(answer); if (!isCurrentDisplayPeer(nextAudioPeerConnection)) return; audioAnswerSummary = summarizeSdp(nextAudioPeerConnection.localDescription?.sdp || answer.sdp); - const ack = await sendRuntimeSignal(nextAudioPeerConnection, displaySession, { - method: "POST", - body: { - type: "answer", - channel: "audio", - sdp: stripTrickleCandidatesFromSdp(nextAudioPeerConnection.localDescription.sdp), + const ack = await postAnswerWithQueuedLocalIce( + nextAudioPeerConnection, + displaySession, + { channel: "audio" }, + queuedAudioCandidates, + () => { + canSignalAudioCandidates = true; }, - }); - if (!isCurrentDisplayPeer(nextAudioPeerConnection)) return; - requireSignalAck(ack, "answer"); - await applyEngineRemoteSignalsTo(nextAudioPeerConnection, ack); - if (!isCurrentDisplayPeer(nextAudioPeerConnection)) return; - canSignalAudioCandidates = true; - for (const candidate of queuedAudioCandidates.splice(0)) { - await signalAudioCandidate(candidate); - if (!isCurrentDisplayPeer(nextAudioPeerConnection)) return; - } + signalAudioCandidate, + iceTransportPolicy === "relay", + ); + if (!ack || !isCurrentDisplayPeer(nextAudioPeerConnection)) return; pollAudioEngineCandidates(WEBRTC_ENGINE_CANDIDATE_POLL_ATTEMPTS); } catch (error) { if (isCurrentDisplayPeer(nextAudioPeerConnection)) throw error; } } + async function postAnswerWithQueuedLocalIce( + nextPeer, + displaySession, + extraBody, + queued, + enableSignal, + signalOne, + relayOnly, + ) { + if (!isCurrentDisplayPeer(nextPeer)) return null; + enableSignal(); + const flushing = queued.splice(0).map((candidate) => + signalOne(candidate).catch((error) => { + recoverPeerSignal(nextPeer, error); + }), + ); + const ack = await sendRuntimeSignal(nextPeer, displaySession, { + method: "POST", + body: { + type: "answer", + sdp: stripTrickleCandidatesFromSdp(nextPeer.localDescription.sdp), + ...extraBody, + }, + }); + if (!isCurrentDisplayPeer(nextPeer)) return null; + requireSignalAck(ack, "answer"); + await applyEngineRemoteSignalsTo(nextPeer, ack); + if (!isCurrentDisplayPeer(nextPeer)) return null; + void Promise.all(flushing); + return ack; + } + function displayForRuntimeSignaling(displaySession) { const generation = supportsDisplayGeneration() ? displaySession?.display_generation : undefined; if (generation !== undefined && (typeof generation !== "string" || !/^display:[a-f0-9]{32}$/.test(generation))) { @@ -968,9 +995,28 @@ export function createBrowserRemoteDisplay({ startStatsPolling(nextPeerConnection); startFrameWatch(nextPeerConnection); const expectsAudio = displaySession.audio === true; + let audioPeerStarted = false; + const startAudioPeerAfterVideoIce = () => { + if ( + audioPeerStarted || + !expectsAudio || + !isCurrentDisplayPeer(nextPeerConnection) + ) { + return; + } + audioPeerStarted = true; + void connectAudioPeer( + displaySession, + iceServers, + iceTransportPolicy, + ).catch((error) => { + recoverPeerSignal(nextPeerConnection, error); + }); + }; prepareAudio(expectsAudio); peerConnection = nextPeerConnection; emitMediaDiagnostic("viewer_peer_created", "video"); + startAudioPeerAfterVideoIce(); inputChannel = null; const offerer = displaySession.offerer === "engine" ? "engine" : "browser"; if (inputTransport === "datachannel") { @@ -1013,6 +1059,7 @@ export function createBrowserRemoteDisplay({ ), ...mediaDiagnosticStats(latestVideoWebrtcStats), }); + startAudioPeerAfterVideoIce(); showStatus( remoteAudioExpected ? "Remote display ready. Click the page to enable audio." @@ -1130,6 +1177,7 @@ export function createBrowserRemoteDisplay({ window.clearTimeout(disconnectTimer); disconnectTimer = 0; markVideoProgress(); + startAudioPeerAfterVideoIce(); return; } if ( @@ -1260,19 +1308,18 @@ export function createBrowserRemoteDisplay({ if (!isCurrentDisplayPeer(nextPeerConnection)) return; await nextPeerConnection.setLocalDescription(answer); if (!isCurrentDisplayPeer(nextPeerConnection)) return; - const ack = await sendRuntimeSignal(nextPeerConnection, displaySession, { - method: "POST", - body: { - type: "answer", - sdp: stripTrickleCandidatesFromSdp( - nextPeerConnection.localDescription.sdp, - ), + const ack = await postAnswerWithQueuedLocalIce( + nextPeerConnection, + displaySession, + {}, + queuedCandidates, + () => { + canSignalCandidates = true; }, - }); - if (!isCurrentDisplayPeer(nextPeerConnection)) return; - requireSignalAck(ack, "answer"); - await applyEngineRemoteSignalsTo(nextPeerConnection, ack); - if (!isCurrentDisplayPeer(nextPeerConnection)) return; + signalCandidate, + iceTransportPolicy === "relay", + ); + if (!ack || !isCurrentDisplayPeer(nextPeerConnection)) return; } else { const offer = await nextPeerConnection.createOffer(); if (!isCurrentDisplayPeer(nextPeerConnection)) return; @@ -1309,10 +1356,6 @@ export function createBrowserRemoteDisplay({ if (!isCurrentDisplayPeer(nextPeerConnection)) return; } pollEngineCandidates(WEBRTC_ENGINE_CANDIDATE_POLL_ATTEMPTS); - if (expectsAudio) { - await connectAudioPeer(displaySession, iceServers, iceTransportPolicy); - if (!isCurrentDisplayPeer(nextPeerConnection)) return; - } connectTimer = window.setTimeout(() => { if (!trackReady && getCurrentDisplayMode() === "webrtc_remote_display") { failRemoteDisplay(nextPeerConnection, "no_first_frame"); diff --git a/capsules/browser/browser/browser-restore-boot.js b/capsules/browser/browser/browser-restore-boot.js new file mode 100644 index 000000000..ae3037a97 --- /dev/null +++ b/capsules/browser/browser/browser-restore-boot.js @@ -0,0 +1,299 @@ +(function () { + var params = new URLSearchParams(location.search); + var instance = params.get("browser_instance") || ""; + var pageId = params.get("page_id") || ""; + var generationHint = params.get("display_generation") || ""; + var token = new URLSearchParams(location.hash.replace(/^#/, "")).get("home_token") || ""; + if (!instance || !token) return; + + function validRequest(value) { + return typeof value === "string" && /^[a-f0-9]{32}$/.test(value); + } + + function validGeneration(value) { + return typeof value === "string" && /^display:[a-f0-9]{32}$/.test(value); + } + + function requestIdFromGeneration(generation) { + return generation.indexOf("display:") === 0 ? generation.slice("display:".length) : ""; + } + + var retryIdentity = { page_id: "", generation: "", request_id: "" }; + + function attachTimeoutMs() { + return typeof VIEWER_DISPLAY_ATTACH_TIMEOUT_MS === "number" && VIEWER_DISPLAY_ATTACH_TIMEOUT_MS > 0 + ? VIEWER_DISPLAY_ATTACH_TIMEOUT_MS + : 8000; + } + + function reuseAttachment(pending) { + if ( + !pending || + pending.schema !== "elastos.browser.display-attachment/v1" || + !validRequest(pending.request_id) || + !validGeneration(pending.previous_display_generation) + ) { + return null; + } + if (pending.state === "pending" || pending.state === "ready") return pending; + if (pending.state === "failed" && pending.error_code === "display_attach_uncertain") return pending; + return null; + } + + function allocateRetryRequestId(targetPageId, generation) { + if ( + retryIdentity.page_id === targetPageId && + retryIdentity.generation === generation && + validRequest(retryIdentity.request_id) + ) { + return retryIdentity.request_id; + } + retryIdentity.page_id = targetPageId; + retryIdentity.generation = generation; + retryIdentity.request_id = crypto.randomUUID().replace(/-/g, ""); + return retryIdentity.request_id; + } + + function displayAttachRequest(targetPageId, generation, pending) { + var reused = reuseAttachment(pending); + if (reused) { + return { + type: "display_attach", + request_id: reused.request_id, + display_generation: reused.previous_display_generation, + }; + } + if ( + (pending && pending.state === "failed" && pending.error_code === "display_attach_failed") || + (validRequest(retryIdentity.request_id) && + retryIdentity.page_id === targetPageId && + retryIdentity.generation === generation) + ) { + return { + type: "display_attach", + request_id: allocateRetryRequestId(targetPageId, generation), + display_generation: generation, + }; + } + return { + type: "display_attach", + request_id: generation.indexOf("display:") === 0 + ? generation.slice("display:".length) + : crypto.randomUUID().replace(/-/g, ""), + display_generation: generation, + }; + } + + function postAttach(targetPageId, generation, pending) { + var request = displayAttachRequest(targetPageId, generation, pending); + var controller = typeof AbortController === "function" ? new AbortController() : null; + var timer = controller ? setTimeout(function () { controller.abort(); }, attachTimeoutMs()) : 0; + return fetch("/api/apps/browser/pages/" + encodeURIComponent(targetPageId) + "/webrtc", { + method: "POST", + headers: { "x-elastos-home-token": token, "content-type": "application/json" }, + body: JSON.stringify(request), + signal: controller ? controller.signal : undefined, + }).then(function (response) { + if (!response.ok) throw new Error("Browser display attach failed"); + return response.json(); + }).then(function (result) { + return { request: request, result: result, page_id: targetPageId }; + }).finally(function () { + if (timer) clearTimeout(timer); + }); + } + + function offerValid(offer) { + return offer && + offer.schema === "elastos.browser.webrtc-offer/v1" && + offer.type === "offer" && + typeof offer.sdp === "string" && + offer.sdp.length > 0 && + offer.sdp.length <= 256 * 1024; + } + + function resultFromSummary(summary, targetPageId, generation) { + var recovery = summary && summary.sessions && summary.sessions.recoverable_page; + var display = recovery && recovery.engine_page && recovery.engine_page.display_session; + var pending = reuseAttachment(recovery && recovery.display_attachment); + if ( + !recovery || + recovery.page_id !== targetPageId || + !pending || + pending.state !== "ready" || + pending.previous_display_generation !== generation || + !display || + !validGeneration(display.display_generation) || + display.display_generation === generation || + !offerValid(display.initial_offer) || + !offerValid(display.audio_offer) + ) { + return null; + } + return { + request: { + type: "display_attach", + request_id: pending.request_id, + display_generation: generation, + }, + result: { + schema: "elastos.browser.display-attach-result/v1", + page_id: targetPageId, + request_id: pending.request_id, + previous_display_generation: generation, + display_generation: display.display_generation, + initial_offer: display.initial_offer, + audio_offer: display.audio_offer, + }, + page_id: targetPageId, + }; + } + + function matchingAttachment(summary, targetPageId, generation) { + var recovery = summary && summary.sessions && summary.sessions.recoverable_page; + var pending = recovery && recovery.display_attachment; + if ( + !recovery || + recovery.page_id !== targetPageId || + !pending || + pending.schema !== "elastos.browser.display-attachment/v1" || + ["pending", "ready", "failed"].indexOf(pending.state) < 0 || + !validRequest(pending.request_id) || + !validGeneration(pending.previous_display_generation) || + pending.previous_display_generation !== generation + ) { + return null; + } + return pending; + } + + var headers = { "x-elastos-home-token": token }; + function fetchSummary() { + return fetch( + "/api/apps/browser/summary?browser_instance=" + encodeURIComponent(instance) + "&remote_services=0", + { headers: headers }, + ).then(function (response) { + if (!response.ok) throw new Error("Browser summary failed"); + return response.json(); + }); + } + + function delay(ms) { + return new Promise(function (resolve) { setTimeout(resolve, ms); }); + } + + function announceReady(attached, fromSummary) { + try { + console.info(JSON.stringify({ + schema: "elastos.browser.media-diagnostic/v1", + event: "restore_boot_ready", + from_summary: fromSummary === true, + request_id: attached && attached.request ? attached.request.request_id : "", + })); + } catch (error) { + // Keep restore attach on the product path if diagnostics cannot print. + } + return attached; + } + + function attachFromKnownUrl(targetPageId, generation) { + return postAttach(targetPageId, generation, null).then(function (attached) { + return announceReady(attached, false); + }).catch(function () { + return summaryPromise.then(function (summary) { + var pending = matchingAttachment(summary, targetPageId, generation); + if (!pending) { + pending = summary && summary.sessions && summary.sessions.recoverable_page + ? summary.sessions.recoverable_page.display_attachment + : null; + } + return postAttach(targetPageId, generation, pending).then(function (attached) { + return announceReady(attached, false); + }); + }); + }); + } + + function waitForReadyAttach(targetPageId, generation) { + var deadline = Date.now() + 2500; + var missDeadline = Date.now() + 120; + var seenPending = false; + function step(summary) { + var ready = resultFromSummary(summary, targetPageId, generation); + if (ready) return announceReady(ready, true); + var pending = matchingAttachment(summary, targetPageId, generation); + if (pending) seenPending = true; + if (pending && pending.state === "ready") { + return postAttach(targetPageId, generation, pending).then(function (attached) { + return announceReady(attached, false); + }); + } + if (pending && pending.state === "failed") { + return postAttach(targetPageId, generation, pending); + } + var now = Date.now(); + if (seenPending && pending && pending.state === "pending" && now < deadline) { + return delay(40).then(function () { return fetchSummary().then(step); }); + } + if (!seenPending && now < missDeadline) { + return delay(40).then(function () { return fetchSummary().then(step); }); + } + return postAttach(targetPageId, generation, pending); + } + return summaryPromise.then(step); + } + + function attachFromSummary(summary) { + var recovery = summary && summary.sessions && summary.sessions.recoverable_page; + var display = recovery && recovery.engine_page && recovery.engine_page.display_session; + var generation = display && display.display_generation; + var pending = recovery && recovery.display_attachment; + if ( + !recovery || + recovery.state !== "active" || + typeof recovery.page_id !== "string" || + !validGeneration(generation) || + !summary.engine_adapter || + summary.engine_adapter.display_attach_supported !== true || + display.mode !== "webrtc_remote_display" + ) { + return null; + } + if ( + pending && + (pending.schema !== "elastos.browser.display-attachment/v1" || + ["pending", "ready", "failed"].indexOf(pending.state) < 0 || + !validRequest(pending.request_id) || + !validGeneration(pending.previous_display_generation)) + ) { + return null; + } + if (pending && pending.state === "pending") { + return waitForReadyAttach(recovery.page_id, pending.previous_display_generation); + } + return postAttach(recovery.page_id, generation, pending); + } + + var summaryPromise = fetchSummary(); + + var attachFromUrl = Boolean(pageId && validGeneration(generationHint)); + try { + console.info(JSON.stringify({ + schema: "elastos.browser.media-diagnostic/v1", + event: "restore_boot", + attach_from_url: attachFromUrl, + request_id: attachFromUrl ? requestIdFromGeneration(generationHint) : "", + })); + } catch (error) { + // Keep boot attach on the product path if diagnostics cannot print. + } + var attachPromise = attachFromUrl + ? attachFromKnownUrl(pageId, generationHint) + : summaryPromise.then(attachFromSummary); + + window.__elastosBrowserRestoreBoot = { + summaryPromise: summaryPromise, + attachPromise: attachPromise, + retryIdentity: retryIdentity, + }; +})(); diff --git a/capsules/browser/browser/browser-webrtc.js b/capsules/browser/browser/browser-webrtc.js index bea029ff5..90df5fbd0 100644 --- a/capsules/browser/browser/browser-webrtc.js +++ b/capsules/browser/browser/browser-webrtc.js @@ -65,6 +65,63 @@ export function sdpHasOnlyRelayCandidates(sdp) { .every((line) => iceCandidateType(line) === "relay"); } +export const ANSWER_ICE_GATHER_BUDGET_MS = 800; + +export function localAnswerSdpHasUsableCandidates(sdp, relayOnly) { + const lines = String(sdp || "") + .split(/\r?\n/) + .map((line) => line.trim()) + .filter((line) => /^a=candidate:/i.test(line)); + if (lines.length === 0) { + return false; + } + if (relayOnly) { + return lines.some((line) => iceCandidateType(line) === "relay"); + } + return true; +} + +export function waitForLocalAnswerIce( + peerConnection, + { relayOnly = false, budgetMs = ANSWER_ICE_GATHER_BUDGET_MS } = {}, +) { + const usable = () => + localAnswerSdpHasUsableCandidates(peerConnection.localDescription?.sdp, relayOnly) || + peerConnection.iceGatheringState === "complete"; + if (usable()) { + return Promise.resolve(); + } + return new Promise((resolve) => { + let done = false; + const finish = () => { + if (done) { + return; + } + done = true; + peerConnection.removeEventListener("icecandidate", onCandidate); + peerConnection.removeEventListener("icegatheringstatechange", onState); + clearTimeout(timer); + resolve(); + }; + const onCandidate = () => { + if (usable()) { + finish(); + } + }; + const onState = () => { + if (peerConnection.iceGatheringState === "complete") { + finish(); + } + }; + const timer = setTimeout(finish, budgetMs); + peerConnection.addEventListener("icecandidate", onCandidate); + peerConnection.addEventListener("icegatheringstatechange", onState); + if (usable()) { + finish(); + } + }); +} + export function normalizeDisplayIceServers(value) { if (!Array.isArray(value)) { return []; diff --git a/capsules/browser/browser/browser-webrtc.test.mjs b/capsules/browser/browser/browser-webrtc.test.mjs index 5ea70a3ba..6bc1aeac4 100644 --- a/capsules/browser/browser/browser-webrtc.test.mjs +++ b/capsules/browser/browser/browser-webrtc.test.mjs @@ -3,7 +3,9 @@ import test from "node:test"; import { iceCandidateType, + localAnswerSdpHasUsableCandidates, sdpHasOnlyRelayCandidates, + waitForLocalAnswerIce, } from "./browser-webrtc.js"; test("engine-only VZ media accepts only relay candidates", () => { @@ -24,3 +26,48 @@ test("engine-only VZ media accepts only relay candidates", () => { ); assert.equal(sdpHasOnlyRelayCandidates("v=0\r\n"), true); }); + +function createFakePeer({ sdp = "", gathering = "new" } = {}) { + const listeners = { + icecandidate: new Set(), + icegatheringstatechange: new Set(), + }; + return { + localDescription: { sdp }, + iceGatheringState: gathering, + addEventListener(type, fn) { + listeners[type].add(fn); + }, + removeEventListener(type, fn) { + listeners[type].delete(fn); + }, + emit(type) { + for (const fn of listeners[type]) { + fn(); + } + }, + }; +} + +test("relay answers wait for a relay candidate", async () => { + const relay = + "a=candidate:1 1 UDP 16777215 203.0.113.7 55001 typ relay raddr 0.0.0.0 rport 0"; + const host = "a=candidate:2 1 UDP 2122260223 192.0.2.9 49152 typ host"; + assert.equal(localAnswerSdpHasUsableCandidates(`v=0\r\n${host}\r\n`, true), false); + assert.equal(localAnswerSdpHasUsableCandidates(`v=0\r\n${relay}\r\n`, true), true); + const peer = createFakePeer(); + const waiting = waitForLocalAnswerIce(peer, { relayOnly: true, budgetMs: 1000 }); + peer.localDescription = { sdp: `v=0\r\n${host}\r\n` }; + peer.emit("icecandidate"); + peer.localDescription = { sdp: `v=0\r\n${relay}\r\n` }; + peer.emit("icecandidate"); + await waiting; +}); + +test("answer ICE wait ends when gathering completes", async () => { + const peer = createFakePeer(); + const waiting = waitForLocalAnswerIce(peer, { relayOnly: true, budgetMs: 1000 }); + peer.iceGatheringState = "complete"; + peer.emit("icegatheringstatechange"); + await waiting; +}); diff --git a/capsules/browser/browser/browser.js b/capsules/browser/browser/browser.js index 091bc823a..a0b8274b6 100644 --- a/capsules/browser/browser/browser.js +++ b/capsules/browser/browser/browser.js @@ -27,7 +27,7 @@ import { isMissingRuntimePageError, requestedDisplayMode, } from "./browser-status.js?v=browser-20260907c"; -import { createBrowserRemoteDisplay } from "./browser-remote-display.js?v=browser-20260907c"; +import { createBrowserRemoteDisplay } from "./browser-remote-display.js?v=browser-20260915d"; import { renderServiceSelection } from "./browser-service-selection.js?v=browser-20260907c"; import { createOperatorApprovalController, mountOperatorApproval } from "./browser-operator-approval.js"; @@ -117,6 +117,8 @@ let pageStatusRefreshTimers = []; let pageHeartbeatTimer = 0; let lastPageStatus = null; let unloadCleanupStarted = false; +let recoverableDisplayAttachStarted = false; +let displayAttachRetryIdentity = null; let remoteDisplay = null; let relaunchRequested = false; let browserAuthorityRenewal = null; @@ -428,8 +430,185 @@ function recoverableRuntimePage(summary = browserSummary) { }; } +function displayAttachRequestId(generation) { + return typeof generation === "string" && generation.startsWith("display:") + ? generation.slice("display:".length) + : ""; +} + +function viewerDisplayAttachTimeoutMs() { + const value = globalThis.VIEWER_DISPLAY_ATTACH_TIMEOUT_MS; + return typeof value === "number" && value > 0 ? value : 8000; +} + +function reuseDisplayAttachment(pending) { + const validRequest = value => typeof value === "string" && /^[a-f0-9]{32}$/.test(value); + const validGeneration = value => typeof value === "string" && /^display:[a-f0-9]{32}$/.test(value); + if ( + !pending || + pending.schema !== "elastos.browser.display-attachment/v1" || + !validRequest(pending.request_id) || + !validGeneration(pending.previous_display_generation) + ) { + return null; + } + if (pending.state === "pending" || pending.state === "ready") { + return pending; + } + if (pending.state === "failed" && pending.error_code === "display_attach_uncertain") { + return pending; + } + return null; +} + +function terminalDisplayAttachment(pending) { + return pending && + pending.schema === "elastos.browser.display-attachment/v1" && + pending.state === "failed" && + pending.error_code === "display_attach_failed"; +} + +function allocateDisplayAttachRetryId(pageId, generation) { + const validRequest = value => typeof value === "string" && /^[a-f0-9]{32}$/.test(value); + const boot = typeof window !== "undefined" ? window.__elastosBrowserRestoreBoot : null; + const shared = (boot && boot.retryIdentity) || displayAttachRetryIdentity; + if ( + shared && + shared.page_id === pageId && + shared.generation === generation && + validRequest(shared.request_id) + ) { + displayAttachRetryIdentity = shared; + if (boot) boot.retryIdentity = shared; + return shared.request_id; + } + const requestId = crypto.randomUUID().replace(/-/g, ""); + if (shared && typeof shared === "object") { + shared.page_id = pageId; + shared.generation = generation; + shared.request_id = requestId; + displayAttachRetryIdentity = shared; + if (boot) boot.retryIdentity = shared; + return requestId; + } + const next = { page_id: pageId, generation, request_id: requestId }; + displayAttachRetryIdentity = next; + if (boot) boot.retryIdentity = next; + return next.request_id; +} + +function markDisplayAttachRetryNeeded(pageId, generation) { + const boot = typeof window !== "undefined" ? window.__elastosBrowserRestoreBoot : null; + const shared = (boot && boot.retryIdentity) || displayAttachRetryIdentity || { + page_id: "", + generation: "", + request_id: "", + }; + shared.page_id = pageId; + shared.generation = generation; + shared.request_id = ""; + displayAttachRetryIdentity = shared; + if (boot) boot.retryIdentity = shared; +} + +function displayAttachRequestForRecovery(pageId, generation, pending) { + const reused = reuseDisplayAttachment(pending); + if (reused) { + return { + type: "display_attach", + request_id: reused.request_id, + display_generation: reused.previous_display_generation, + }; + } + const boot = typeof window !== "undefined" ? window.__elastosBrowserRestoreBoot : null; + const shared = (boot && boot.retryIdentity) || displayAttachRetryIdentity; + if ( + terminalDisplayAttachment(pending) || + (shared && shared.page_id === pageId && shared.generation === generation) + ) { + return { + type: "display_attach", + request_id: allocateDisplayAttachRetryId(pageId, generation), + display_generation: generation, + }; + } + return { + type: "display_attach", + request_id: displayAttachRequestId(generation), + display_generation: generation, + }; +} + +function startRecoverableDisplayAttach() { + if ( + recoverableDisplayAttachStarted || + homeWindowCloseInFlight || + homeWindowTerminalCloseConfirmed + ) { + return; + } + const pageId = currentPage?.page_id; + const generation = currentPage?.display_session?.display_generation; + const requestId = displayAttachRequestId(generation); + if ( + typeof pageId !== "string" || + !pageId || + !launchToken || + !/^[a-f0-9]{32}$/.test(requestId) || + currentPage?.display_session?.mode !== PRODUCT_DISPLAY_MODE || + browserSummary?.engine_adapter?.display_attach_supported !== true + ) { + return; + } + recoverableDisplayAttachStarted = true; + try { + console.info(JSON.stringify({ + schema: "elastos.browser.media-diagnostic/v1", + event: "dying_page_attach", + request_id: requestId, + })); + } catch { + // Keep unload attach on the product path if diagnostics cannot print. + } + fetch(`/api/apps/browser/pages/${encodeURIComponent(pageId)}/webrtc`, { + method: "POST", + headers: { + "x-elastos-home-token": launchToken, + "content-type": "application/json", + }, + body: JSON.stringify({ + type: "display_attach", + request_id: requestId, + display_generation: generation, + }), + keepalive: true, + }).catch(() => {}); +} + +function persistRecoverableDisplayQuery(page) { + if (typeof location === "undefined" || typeof history === "undefined") { + return; + } + const url = new URL(location.href); + const generation = page?.display_session?.display_generation; + const validGeneration = typeof generation === "string" && /^display:[a-f0-9]{32}$/.test(generation); + if (typeof page?.page_id === "string" && page.page_id && validGeneration) { + url.searchParams.set("page_id", page.page_id); + url.searchParams.set("display_generation", generation); + } else if (!page?.page_id) { + url.searchParams.delete("page_id"); + url.searchParams.delete("display_generation"); + } + const next = `${url.pathname}${url.search}${url.hash}`; + const current = `${location.pathname}${location.search}${location.hash}`; + if (next !== current) { + history.replaceState(history.state, "", next); + } +} + function publishRuntimePageForHost(page = currentPage) { window.__elastosBrowserCurrentPageId = page?.page_id || ""; + persistRecoverableDisplayQuery(page); } function currentRuntimePageOwner() { @@ -489,6 +668,18 @@ const runtimePageCleanup = createRuntimePageCleanupController({ onTerminal: (owner, _outcome, failure) => { const applied = finalizeRuntimePageClose(owner); deliverPendingHomeBrowserWindowClose(owner, _outcome); + if ( + applied && + (_outcome?.profile_durability === "failed" || + _outcome?.profile_durability === "unknown") && + !unloadCleanupStarted + ) { + showStatus( + "Runtime closed the Browser session, but the profile save failed. You can open the address again.", + { sticky: true }, + ); + return; + } if (applied && failure && !unloadCleanupStarted) { showStatus( "Runtime confirmed the failed Browser session closed. You can open the address again or choose another Browser Engine.", @@ -1750,13 +1941,33 @@ function syncEngineSelect(summary) { updateSettingsTitle(); } -async function fetchBrowserSummary() { +async function fetchBrowserSummary({ remoteServices = true } = {}) { if (browserSummaryPromise) { return browserSummaryPromise; } - const summaryPath = browserInstanceId - ? `/api/apps/browser/summary?browser_instance=${encodeURIComponent(browserInstanceId)}` - : "/api/apps/browser/summary"; + const boot = typeof window !== "undefined" ? window.__elastosBrowserRestoreBoot : null; + if (!remoteServices && boot?.summaryPromise) { + const pending = boot.summaryPromise; + boot.summaryPromise = null; + try { + const summary = await pending; + browserSummary = summary; + syncEngineSelect(summary); + syncExitSelect(summary); + return summary; + } catch { + // Fall through to a live summary request when the boot prefetch fails. + } + } + const query = new URLSearchParams(); + if (browserInstanceId) { + query.set("browser_instance", browserInstanceId); + } + if (!remoteServices) { + query.set("remote_services", "0"); + } + const encoded = query.toString(); + const summaryPath = encoded ? `/api/apps/browser/summary?${encoded}` : "/api/apps/browser/summary"; browserSummaryPromise = fetchJson(summaryPath, { method: "GET" }) .then((summary) => { browserSummary = summary; @@ -2051,6 +2262,7 @@ async function requestRuntimeOpen(value, { history = "push" } = {}) { ); } await connectRemoteDisplay(openedPage.display_session, openedPage); + publishRuntimePageForHost(currentPage); startPageStatusPolling(); startPageHeartbeat(); if (!remoteDisplay.isTrackReady()) { @@ -2347,19 +2559,41 @@ async function attachRecoveredDisplay(summary, owner) { !validRequest(pending.request_id) || !validGeneration(pending.previous_display_generation))) { throw new Error("Runtime could not check the pending Browser display attachment."); } - const request = { - type: "display_attach", - request_id: pending?.state === "pending" ? pending.request_id : crypto.randomUUID().replaceAll("-", ""), - display_generation: pending?.state === "pending" ? pending.previous_display_generation : display.display_generation, - }; + const request = displayAttachRequestForRecovery( + owner.page_id, + display.display_generation, + pending, + ); let result; - try { - result = await fetchJson(`/api/apps/browser/pages/${encodeURIComponent(owner.page_id)}/webrtc`, { - method: "POST", body: request, - }); - } catch (error) { - if (!runtimeViewerOwnerActive(owner)) return null; - throw error; + const boot = typeof window !== "undefined" ? window.__elastosBrowserRestoreBoot : null; + const bootAttach = boot?.attachPromise; + if (bootAttach) { + boot.attachPromise = null; + try { + const attached = await bootAttach; + if (attached?.page_id === owner.page_id && attached.request && attached.result) { + request.request_id = attached.request.request_id; + request.display_generation = attached.request.display_generation; + result = attached.result; + } + } catch { + result = undefined; + } + } + if (!result) { + try { + result = await fetchJson(`/api/apps/browser/pages/${encodeURIComponent(owner.page_id)}/webrtc`, { + method: "POST", + body: request, + signal: AbortSignal.timeout(viewerDisplayAttachTimeoutMs()), + }); + } catch (error) { + if (!runtimeViewerOwnerActive(owner)) return null; + if (error?.payload?.code === "display_attach_failed") { + markDisplayAttachRetryNeeded(owner.page_id, request.display_generation); + } + throw error; + } } if (!runtimeViewerOwnerActive(owner)) return null; const offerValid = offer => offer?.schema === "elastos.browser.webrtc-offer/v1" && offer.type === "offer" && @@ -2419,28 +2653,43 @@ async function restoreRuntimePageViewer(summary) { syncExitSelect(summary); showStatus("Restoring the Browser display...", { sticky: true }); startPageHeartbeat(); + let statusPromise = null; try { // Page status is diagnostic; attach replaces offers within retained Runtime authority. if (currentPage.display_session?.mode !== "webrtc_remote_display") { throw new Error("Runtime could not restore the Browser display."); } - const [status, display] = await Promise.all([ - fetchPageStatus({ history: "replace", forceAddress: true }), - attachRecoveredDisplay(summary, owner), - ]); - if (!status || !display || !runtimeViewerOwnerActive(owner)) return true; + statusPromise = fetchPageStatus({ history: "replace", forceAddress: true }); + const display = await attachRecoveredDisplay(summary, owner); + if (!display || !runtimeViewerOwnerActive(owner)) { + await statusPromise.catch(() => null); + return true; + } if (display.mode !== "webrtc_remote_display") { throw new Error("Runtime could not restore the Browser display."); } - currentPage = { ...currentPage, display_session: display }; - publishRuntimePageForHost(currentPage); currentDisplayMode = display.mode; syncDisplayInputFromSession(display); currentView = viewFromDisplaySession(display) || currentPage.view; startPageStatusPolling(); - await connectRemoteDisplay(display, currentPage); + const connecting = connectRemoteDisplay(display, { ...currentPage, display_session: display }); + let status; + try { + status = await statusPromise; + } catch (error) { + await connecting.catch(() => null); + throw error; + } + if (!status || !runtimeViewerOwnerActive(owner)) { + await connecting.catch(() => null); + return true; + } + currentPage = { ...currentPage, display_session: display }; + publishRuntimePageForHost(currentPage); + await connecting; return true; } catch (error) { + if (statusPromise) await statusPromise.catch(() => null); if (!runtimeViewerOwnerActive(owner)) return true; throw error; } finally { @@ -2449,10 +2698,16 @@ async function restoreRuntimePageViewer(summary) { } window.addEventListener("beforeunload", () => { + startRecoverableDisplayAttach(); releaseRuntimePageForUnload(); }); -window.addEventListener("pagehide", releaseRuntimePageForUnload); +window.addEventListener("pagehide", (event) => { + if (!event.persisted) { + startRecoverableDisplayAttach(); + } + releaseRuntimePageForUnload(); +}); mountOperatorApproval({ container: settingsPanel, @@ -2465,13 +2720,18 @@ mountOperatorApproval({ }), }); -const initialUrl = params.get("url") || DEFAULT_URL; +const requestedStartupUrl = params.get("url"); +const initialUrl = requestedStartupUrl || DEFAULT_URL; addressInput.value = initialUrl; setLoading(true); -fetchBrowserSummary() +fetchBrowserSummary({ remoteServices: false }) .then(async (summary) => { if (await restoreRuntimePageViewer(summary)) return; - return requestRuntimeOpen(initialUrl, { history: "replace" }); + if (!requestedStartupUrl) { + setLoading(false); + return; + } + return requestRuntimeOpen(requestedStartupUrl, { history: "replace" }); }) .catch((error) => { if (unloadCleanupStarted || homeWindowCloseInFlight || homeWindowTerminalCloseConfirmed) return; diff --git a/capsules/browser/browser/index.html b/capsules/browser/browser/index.html index 4af52b1c4..33e32b8f2 100644 --- a/capsules/browser/browser/index.html +++ b/capsules/browser/browser/index.html @@ -7,6 +7,21 @@ + + + + + + + + + + + + + + +
@@ -89,6 +104,6 @@
- + diff --git a/docs/BROWSER_CAPSULE.md b/docs/BROWSER_CAPSULE.md index d19722768..542c01624 100644 --- a/docs/BROWSER_CAPSULE.md +++ b/docs/BROWSER_CAPSULE.md @@ -7,6 +7,54 @@ The shared wire definitions and version rules are in [Browser contract](BROWSER_PROTOCOL.md). Device roles, resource requirements and qualification gates are in [Browser support](BROWSER_SUPPORT.md). +## Capsule delivery and service selection + +Browser uses the common [Digital Capsule model](CAPSULE_MODEL.md). Its entry +point and UI can remain one app capsule. Engine and Exit implementations are +provider capsules. Each installable package has a complete signed artifact +closure, including its declared dependencies and compatible platform payloads. +Native Engine helpers and the VM image belong to that verified closure; Runtime +owns admission onto the supported host substrate. A library may be a verified +member of a package without becoming another independently managed capsule. + +The Browser entry declares required interfaces and dependencies. Runtime resolves +them through the trusted catalog to exact artifact identities, verifies and +admits locally needed packages, and binds approved running services. Dependency +names alone do not identify acceptable bytes. Service selection and mutable +profiles stay outside the shared immutable Browser artifact. + +Admission also defines update and removal. An active instance retains its exact +verified package and dependencies. A replacement activates atomically. Removal +waits for consumers to release ownership; a failed close retains that ownership +for reconciliation. These rules belong to the common capsule path. + +[Content availability](CONTENT_AVAILABILITY.md) owns retention and retrieval. +Multiple nodes can retain the same capsule CID. Their availability receipts are +separate from offers to execute an Engine or provide Exit service. A retained +Engine package remains useful while its original execution host is offline, but +another host can run it only if that host has a compatible substrate and grants +the required authority. Pinning bytes does not promise running capacity. + +For local execution, the selected Runtime acquires and admits the Engine closure +on demand. For remote execution, the provider Runtime owns that preparation; +the consumer acquires only its local dependencies and attaches to the approved +service. Both paths use the same Browser entry and Runtime contracts. Content +and availability providers select the byte source; Carrier carries off-box +traffic. Browser does not implement its own downloader or call IPFS directly. + +The installer supplies a small verified bootstrap set with enough Content and +Carrier support to acquire other capsules, plus trusted catalog roots and an +upgrade path. The installer and Runtime distribution paths must agree on artifact +identity and admission. They must not require developer source-home files. + +Delivery proof includes fresh acquisition from each declared availability node +with the other source offline, verified reuse, interrupted acquisition and +tamper rejection. Execution-service unavailability is tested separately. Browser +proof then distinguishes local Engine admission from remote service use without a local +Engine image. Runtime preserves user state and reports unavailable content or +services as separate outcomes. This section defines the target contract; current +installation and replication evidence remains in state.md. + ## Decision ElastOS needs a real browser capsule, but the first production target should not diff --git a/elastos/crates/elastos-server/src/api/browser_capsules.rs b/elastos/crates/elastos-server/src/api/browser_capsules.rs index b3d0736c2..834b2997a 100644 --- a/elastos/crates/elastos-server/src/api/browser_capsules.rs +++ b/elastos/crates/elastos-server/src/api/browser_capsules.rs @@ -5,6 +5,7 @@ use axum::extract::{Path as AxumPath, RawQuery, State}; use axum::http::StatusCode; use axum::response::{IntoResponse, Redirect, Response}; use elastos_common::{CapsuleManifest, CapsuleRole, CapsuleType}; +use sha2::{Digest, Sha256}; use super::capsule_inventory::{ installed_active_capsule_dir, installed_capsules_root, list_active_capsule_manifests, @@ -16,6 +17,7 @@ use super::gateway::{ }; const BROWSER_CAPSULE_CACHE_CONTROL: &str = "no-store"; +const BROWSER_VERSIONED_ASSET_CACHE_CONTROL: &str = "private, max-age=60, must-revalidate"; const BROWSER_CAPSULE_COOP: &str = "same-origin"; const BROWSER_CAPSULE_COEP: &str = "require-corp"; const BROWSER_CAPSULE_DOCUMENT_CORP: &str = "cross-origin"; @@ -87,8 +89,16 @@ pub async fn serve_home_asset( State(state): State, headers: axum::http::HeaderMap, AxumPath(path): AxumPath, + RawQuery(query): RawQuery, ) -> Response { - serve_browser_capsule_path(&state.data_dir, &headers, HOME_CAPSULE_ID, Some(&path)).await + serve_browser_capsule_asset( + &state.data_dir, + &headers, + HOME_CAPSULE_ID, + Some(&path), + query.as_deref(), + ) + .await } pub async fn serve_browser_app_root(AxumPath(app): AxumPath) -> Response { @@ -115,6 +125,7 @@ pub async fn serve_browser_app_asset( State(state): State, headers: axum::http::HeaderMap, AxumPath((app, path)): AxumPath<(String, String)>, + RawQuery(query): RawQuery, ) -> Response { if app == "home-agent" { return if path == "index.html" { @@ -123,7 +134,14 @@ pub async fn serve_browser_app_asset( StatusCode::NOT_FOUND.into_response() }; } - serve_browser_capsule_path(&state.data_dir, &headers, &app, Some(&path)).await + serve_browser_capsule_asset( + &state.data_dir, + &headers, + &app, + Some(&path), + query.as_deref(), + ) + .await } pub(super) fn canonical_browser_capsule_route(route: &str) -> Result { @@ -142,11 +160,59 @@ pub(super) fn canonical_browser_capsule_route(route: &str) -> Result) -> bool { + let versioned = raw_query + .unwrap_or("") + .split('&') + .any(|part| part.starts_with("v=browser-")); + versioned + && matches!( + Path::new(relative_path) + .extension() + .and_then(|ext| ext.to_str()), + Some("js" | "css" | "woff" | "woff2" | "svg") + ) +} + +fn browser_capsule_cache_control(relative_path: &str, raw_query: Option<&str>) -> &'static str { + if browser_capsule_versioned_static_asset(relative_path, raw_query) { + BROWSER_VERSIONED_ASSET_CACHE_CONTROL + } else { + BROWSER_CAPSULE_CACHE_CONTROL + } +} + +fn browser_capsule_etag(bytes: &[u8]) -> String { + format!("\"{}\"", hex::encode(Sha256::digest(bytes))) +} + +fn if_none_match_contains(headers: &axum::http::HeaderMap, etag: &str) -> bool { + headers + .get(axum::http::header::IF_NONE_MATCH) + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| { + value + .split(',') + .map(str::trim) + .any(|candidate| candidate == "*" || candidate == etag) + }) +} + async fn serve_browser_capsule_path( data_dir: &Path, request_headers: &axum::http::HeaderMap, app: &str, requested_path: Option<&str>, +) -> Response { + serve_browser_capsule_asset(data_dir, request_headers, app, requested_path, None).await +} + +async fn serve_browser_capsule_asset( + data_dir: &Path, + request_headers: &axum::http::HeaderMap, + app: &str, + requested_path: Option<&str>, + raw_query: Option<&str>, ) -> Response { if ensure_wallet_connector_configured(data_dir, app).is_err() { return (StatusCode::NOT_FOUND, "Browser capsule not found").into_response(); @@ -199,11 +265,41 @@ async fn serve_browser_capsule_path( }; let is_document = relative_path == capsule.entrypoint; + let cache_control = browser_capsule_cache_control(relative_path, raw_query); + let etag = browser_capsule_versioned_static_asset(relative_path, raw_query) + .then(|| browser_capsule_etag(&bytes)); + if let Some(etag) = etag.as_deref() { + if if_none_match_contains(request_headers, etag) { + let mut response = StatusCode::NOT_MODIFIED.into_response(); + let headers = response.headers_mut(); + headers.insert("cache-control", cache_control.parse().unwrap()); + headers.insert("etag", etag.parse().unwrap()); + headers.insert( + "cross-origin-opener-policy", + BROWSER_CAPSULE_COOP.parse().unwrap(), + ); + headers.insert( + "cross-origin-embedder-policy", + BROWSER_CAPSULE_COEP.parse().unwrap(), + ); + headers.insert( + "cross-origin-resource-policy", + resource_policy.parse().unwrap(), + ); + headers.insert( + axum::http::header::ACCESS_CONTROL_ALLOW_ORIGIN, + BROWSER_CAPSULE_OPAQUE_ORIGIN.parse().unwrap(), + ); + headers.insert("referrer-policy", "no-referrer".parse().unwrap()); + headers.insert("x-content-type-options", "nosniff".parse().unwrap()); + return response; + } + } let mut response = ( StatusCode::OK, [ ("content-type", content_type(relative_path)), - ("cache-control", BROWSER_CAPSULE_CACHE_CONTROL), + ("cache-control", cache_control), ("cross-origin-opener-policy", BROWSER_CAPSULE_COOP), ("cross-origin-embedder-policy", BROWSER_CAPSULE_COEP), ("cross-origin-resource-policy", resource_policy), @@ -212,6 +308,9 @@ async fn serve_browser_capsule_path( ) .into_response(); let headers = response.headers_mut(); + if let Some(etag) = etag.as_deref() { + headers.insert("etag", etag.parse().unwrap()); + } headers.insert( axum::http::header::ACCESS_CONTROL_ALLOW_ORIGIN, BROWSER_CAPSULE_OPAQUE_ORIGIN.parse().unwrap(), @@ -739,6 +838,112 @@ mod tests { .unwrap(); } + #[test] + fn versioned_static_assets_are_cacheable_and_documents_stay_unstored() { + assert!(browser_capsule_versioned_static_asset( + "browser.js", + Some("v=browser-20260907c") + )); + assert_eq!( + browser_capsule_cache_control("browser.js", Some("v=browser-20260907c")), + BROWSER_VERSIONED_ASSET_CACHE_CONTROL + ); + assert!( + !BROWSER_VERSIONED_ASSET_CACHE_CONTROL.contains("immutable"), + "versioned assets must revalidate; they must not stay cached for a year" + ); + assert!( + !BROWSER_VERSIONED_ASSET_CACHE_CONTROL.contains("31536000"), + "versioned assets must revalidate; they must not stay cached for a year" + ); + assert!(BROWSER_VERSIONED_ASSET_CACHE_CONTROL.contains("must-revalidate")); + assert_eq!( + browser_capsule_cache_control("index.html", Some("v=browser-20260907c")), + BROWSER_CAPSULE_CACHE_CONTROL + ); + assert_eq!( + browser_capsule_cache_control("browser.js", None), + BROWSER_CAPSULE_CACHE_CONTROL + ); + } + + #[test] + fn versioned_asset_etag_changes_when_bytes_change() { + let first = browser_capsule_etag(b"browser.js v1"); + let second = browser_capsule_etag(b"browser.js v2"); + assert!(first.starts_with('"') && first.ends_with('"')); + assert_ne!(first, second); + let mut headers = axum::http::HeaderMap::new(); + headers.insert(axum::http::header::IF_NONE_MATCH, first.parse().unwrap()); + assert!(if_none_match_contains(&headers, &first)); + assert!(!if_none_match_contains(&headers, &second)); + } + + #[tokio::test] + async fn versioned_static_assets_revalidate_by_etag() { + let data_dir = tempfile::tempdir().unwrap(); + write_test_browser_capsule(data_dir.path(), "test-browser", "Browser test", "app"); + fs::write( + data_dir.path().join("capsules/test-browser/browser.js"), + "export const version = 1;\n", + ) + .unwrap(); + let first = serve_browser_capsule_asset( + data_dir.path(), + &test_request_headers(), + "test-browser", + Some("browser.js"), + Some("v=browser-20260907c"), + ) + .await; + assert_eq!(first.status(), StatusCode::OK); + assert_eq!( + first + .headers() + .get("cache-control") + .and_then(|value| value.to_str().ok()), + Some(BROWSER_VERSIONED_ASSET_CACHE_CONTROL) + ); + let etag = first + .headers() + .get("etag") + .and_then(|value| value.to_str().ok()) + .unwrap() + .to_string(); + let mut headers = test_request_headers(); + headers.insert(axum::http::header::IF_NONE_MATCH, etag.parse().unwrap()); + let again = serve_browser_capsule_asset( + data_dir.path(), + &headers, + "test-browser", + Some("browser.js"), + Some("v=browser-20260907c"), + ) + .await; + assert_eq!(again.status(), StatusCode::NOT_MODIFIED); + fs::write( + data_dir.path().join("capsules/test-browser/browser.js"), + "export const version = 2;\n", + ) + .unwrap(); + let changed = serve_browser_capsule_asset( + data_dir.path(), + &headers, + "test-browser", + Some("browser.js"), + Some("v=browser-20260907c"), + ) + .await; + assert_eq!(changed.status(), StatusCode::OK); + assert_ne!( + changed + .headers() + .get("etag") + .and_then(|value| value.to_str().ok()), + Some(etag.as_str()) + ); + } + #[test] fn resolves_installed_data_browser_capsule() { let data_dir = tempfile::tempdir().unwrap(); diff --git a/elastos/crates/elastos-server/src/api/gateway_browser.rs b/elastos/crates/elastos-server/src/api/gateway_browser.rs index 59d0daef4..6af62b34b 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser.rs @@ -18,7 +18,7 @@ pub(super) use elastos_common::browser_protocol::{ BrowserWebrtcSignalRequest, BROWSER_DISPLAY_ATTACH_REQUEST_SCHEMA, }; use std::sync::{Mutex as StdMutex, Weak}; -use tokio::sync::{watch, Notify}; +use tokio::sync::{oneshot, watch, Notify}; #[path = "gateway_browser_engine.rs"] mod gateway_browser_engine; #[path = "gateway_browser_operator.rs"] @@ -66,6 +66,18 @@ pub(super) struct BrowserSummaryResponse { pub(super) struct BrowserSummaryQuery { #[serde(default)] pub(super) browser_instance: Option, + /// Session restore reads ownership only. Engine selection still requests live offers. + #[serde(default)] + pub(super) remote_services: Option, +} + +impl BrowserSummaryQuery { + fn omit_remote_services(&self) -> bool { + matches!( + self.remote_services.as_deref(), + Some("0" | "false" | "omit") + ) + } } pub(super) struct BrowserProviderResourceCall { @@ -219,13 +231,18 @@ pub(super) async fn browser_app_summary( Err(err) => return system_error_response(err), }; let context = authority.home_launch_context(); + let omit_remote_services = query.omit_remote_services(); let browser_instance = match browser_instance_id(query.browser_instance) { Ok(value) => value, Err(message) => return (StatusCode::BAD_REQUEST, message).into_response(), }; let mut engine_adapter = browser_engine_summary(state.provider_registry.as_ref(), &context.principal_id).await; - engine_adapter["remote_services"] = browser_remote_engine_summary(&state, &context).await; + engine_adapter["remote_services"] = if omit_remote_services { + serde_json::json!({"state":"available","offers":[]}) + } else { + browser_remote_engine_summary(&state, &context).await + }; let remote_adapters = engine_adapter["remote_services"]["offers"] .as_array() .into_iter() @@ -1452,14 +1469,41 @@ fn validate_browser_vz_did_not_act_settlement( Ok(()) } +fn browser_vz_launch_settlement_keys_are_exact( + settlement: &serde_json::Value, + keys: &[&str], +) -> bool { + let Some(object) = settlement.as_object() else { + return false; + }; + keys.iter().all(|key| object.contains_key(*key)) + && object + .keys() + .all(|key| keys.contains(&key.as_str()) || key == "profile_durability") +} + +fn browser_vz_profile_durability_is_safe(settlement: &serde_json::Value) -> bool { + match settlement.get("profile_durability") { + None => true, + Some(value) => matches!(value.as_str(), Some("failed" | "unknown" | "proved")), + } +} + +fn browser_profile_durability_from_receipt(receipt: &serde_json::Value) -> Option<&str> { + receipt + .get("profile_durability") + .and_then(serde_json::Value::as_str) + .filter(|value| matches!(*value, "failed" | "unknown" | "proved")) +} + fn validate_browser_vz_launch_settlement_binding( settlement: &serde_json::Value, authority: &serde_json::Value, ) -> Result<(bool, bool), String> { validate_browser_vz_transport_authority(authority)?; - let object = settlement - .as_object() - .ok_or_else(|| "Browser VZ launch settlement must be an object".to_string())?; + if settlement.as_object().is_none() { + return Err("Browser VZ launch settlement must be an object".to_string()); + } let keys = [ "schema", "state", @@ -1508,8 +1552,8 @@ fn validate_browser_vz_launch_settlement_binding( .filter(|value| value.len() <= 8_192 && !value.contains('\0')) .ok_or_else(|| "Browser VZ launch settlement message is invalid".to_string())?; let _ = message; - if object.len() != keys.len() - || keys.iter().any(|key| !object.contains_key(*key)) + if !browser_vz_launch_settlement_keys_are_exact(settlement, &keys) + || !browser_vz_profile_durability_is_safe(settlement) || settlement.get("schema").and_then(serde_json::Value::as_str) != Some("elastos.browser.vz-launch-settlement/v1") || settlement.get("binding_hash") != authority.get("binding_hash") @@ -2216,14 +2260,14 @@ fn validate_browser_dispatched_transport_terminal_receipt( if receipt.get("schema").and_then(serde_json::Value::as_str) == Some("elastos.browser.vz-launch-settlement/v1") { - let (acted, settlement_vm_acquired) = + let (acted, _settlement_vm_acquired) = validate_browser_vz_launch_settlement_binding(receipt, authority)?; if selected_engine_adapter.is_none() || receipt.get("state").and_then(serde_json::Value::as_str) != Some("terminal_post_effect_cleanup") || !acted || page_acquired - || settlement_vm_acquired != vm_acquired + || vm_acquired || receipt .get("generation") .and_then(serde_json::Value::as_str) @@ -2458,6 +2502,73 @@ async fn retry_pending_browser_lifecycle_obligations(state: &GatewayState) -> bo preparation_settled || launch_settled || engine_settled || stream_settled } +async fn finish_browser_launch_reconciliation_without_engine_effects( + state: &GatewayState, + reconciliation: &BrowserLaunchReconciliation, +) -> bool { + if let Some(authority) = reconciliation.transport_authority() { + if let Err(err) = gateway_browser_remote::close_transport_listener(state, authority).await { + tracing::warn!( + error = %err, + generation = %reconciliation.generation, + "Browser launch reconciliation could not close its VZ media listener" + ); + release_browser_launch_reconciliation_claim(&state.data_dir, reconciliation).await; + return false; + } + } + let stream_result = tokio::time::timeout( + BROWSER_LAUNCH_RECONCILIATION_CALL_TIMEOUT, + close_browser_stream_cleanup(state, reconciliation.stream_cleanup.clone()), + ) + .await + .map_err(|_| "Browser stream cleanup timed out".to_string()) + .and_then(|result| result); + if let Err(err) = stream_result { + tracing::warn!( + error = %err, + generation = %reconciliation.generation, + "Browser launch reconciliation terminal proof could not close its stream" + ); + release_browser_launch_reconciliation_claim(&state.data_dir, reconciliation).await; + return false; + } + if let Err(error) = gateway_browser_remote::mark_consumer_terminal_retirement( + &state.data_dir, + &reconciliation.principal_id, + &reconciliation.generation, + ) { + tracing::warn!(%error, "Remote Engine terminal retirement could not be retained"); + release_browser_launch_reconciliation_claim(&state.data_dir, reconciliation).await; + return false; + } + if let Err(err) = + forget_browser_launch_reconciliation_obligation(&state.data_dir, reconciliation).await + { + tracing::warn!( + error = %err, + generation = %reconciliation.generation, + "Browser launch reconciliation terminal release could not be committed" + ); + release_browser_launch_reconciliation_claim(&state.data_dir, reconciliation).await; + return false; + } + if let Err(error) = gateway_browser_remote::retire_consumer_generation( + &state.data_dir, + &reconciliation.principal_id, + &reconciliation.generation, + ) { + tracing::warn!(%error, "Remote Engine terminal route retirement remains pending"); + } + release_browser_open_job_instance_for_owner( + &state.data_dir, + &reconciliation.principal_id, + &reconciliation.owner_launch_id, + ) + .await; + true +} + async fn retry_pending_browser_launch_reconciliations(state: &GatewayState) -> bool { let mut settled = false; for reconciliation in claim_pending_browser_launch_reconciliations( @@ -2499,6 +2610,25 @@ async fn retry_pending_browser_launch_reconciliations(state: &GatewayState) -> b } continue; } + let grant_expired = match gateway_browser_remote::consumer_grant_expired( + &state.data_dir, + &reconciliation.principal_id, + &reconciliation.generation, + ) { + Ok(value) => value, + Err(_) => { + release_browser_launch_reconciliation_claim(&state.data_dir, &reconciliation).await; + continue; + } + }; + if grant_expired { + if finish_browser_launch_reconciliation_without_engine_effects(state, &reconciliation) + .await + { + settled = true; + } + continue; + } let result = if reconciliation.was_dispatched() { attempt_browser_launch_reconciliation_bounded( state, @@ -2519,77 +2649,12 @@ async fn retry_pending_browser_launch_reconciliations(state: &GatewayState) -> b match browser_launch_reconciliation_decision(result) { BrowserLaunchReconciliationDecision::DidNotAct | BrowserLaunchReconciliationDecision::TerminalPostEffectCleanup { .. } => { - if let Some(authority) = reconciliation.transport_authority() { - if let Err(err) = - gateway_browser_remote::close_transport_listener(state, authority).await - { - tracing::warn!( - error = %err, - generation = %reconciliation.generation, - "Browser launch reconciliation could not close its VZ media listener" - ); - release_browser_launch_reconciliation_claim( - &state.data_dir, - &reconciliation, - ) - .await; - continue; - } - } - let stream_result = tokio::time::timeout( - BROWSER_LAUNCH_RECONCILIATION_CALL_TIMEOUT, - close_browser_stream_cleanup(state, reconciliation.stream_cleanup.clone()), - ) - .await - .map_err(|_| "Browser stream cleanup timed out".to_string()) - .and_then(|result| result); - if let Err(err) = stream_result { - tracing::warn!( - error = %err, - generation = %reconciliation.generation, - "Browser launch reconciliation terminal proof could not close its stream" - ); - release_browser_launch_reconciliation_claim(&state.data_dir, &reconciliation) - .await; - continue; - } - if let Err(error) = gateway_browser_remote::mark_consumer_terminal_retirement( - &state.data_dir, - &reconciliation.principal_id, - &reconciliation.generation, - ) { - tracing::warn!(%error, "Remote Engine terminal retirement could not be retained"); - release_browser_launch_reconciliation_claim(&state.data_dir, &reconciliation) - .await; - continue; - } - if let Err(err) = forget_browser_launch_reconciliation_obligation( - &state.data_dir, + if finish_browser_launch_reconciliation_without_engine_effects( + state, &reconciliation, ) .await { - tracing::warn!( - error = %err, - generation = %reconciliation.generation, - "Browser launch reconciliation terminal release could not be committed" - ); - release_browser_launch_reconciliation_claim(&state.data_dir, &reconciliation) - .await; - } else { - if let Err(error) = gateway_browser_remote::retire_consumer_generation( - &state.data_dir, - &reconciliation.principal_id, - &reconciliation.generation, - ) { - tracing::warn!(%error, "Remote Engine terminal route retirement remains pending"); - } - release_browser_open_job_instance_for_owner( - &state.data_dir, - &reconciliation.principal_id, - &reconciliation.owner_launch_id, - ) - .await; settled = true; } } @@ -2642,7 +2707,17 @@ async fn retry_pending_browser_launch_reconciliations(state: &GatewayState) -> b .map_err(|_| "Browser stream cleanup timed out".to_string()) .and_then(|result| result); if engine_result.is_ok() && stream_result.is_ok() { - if let Err(err) = commit_browser_terminal_cleanup(state, &cleanup, None).await { + if let Err(err) = commit_browser_terminal_cleanup( + state, + &cleanup, + None, + engine_result + .as_ref() + .ok() + .and_then(browser_profile_durability_from_receipt), + ) + .await + { release_browser_engine_cleanup_claim(&state.data_dir, &cleanup).await; tracing::warn!( error = %err, @@ -2703,7 +2778,7 @@ async fn retry_pending_browser_engine_cleanups(state: &GatewayState) -> bool { } }; if terminal_retirement { - if commit_browser_terminal_cleanup(state, &cleanup, None) + if commit_browser_terminal_cleanup(state, &cleanup, None, None) .await .is_ok() { @@ -2735,34 +2810,61 @@ async fn retry_pending_browser_engine_cleanups(state: &GatewayState) -> bool { .await .map_err(|_| "Browser stream cleanup timed out".to_string()) .and_then(|result| result); - match (engine_result, stream_result) { - (Ok(_), Ok(())) => { - if let Err(err) = commit_browser_terminal_cleanup(state, &cleanup, None).await { - release_browser_engine_cleanup_claim(&state.data_dir, &cleanup).await; + let already_absent = matches!( + (&engine_result, &stream_result), + (Err(message), Ok(())) if browser_engine_close_is_already_absent(message) + ); + let ingress_released = if already_absent { + match release_absent_engine_viewer_ingress(state, &cleanup).await { + Ok(()) => true, + Err(err) => { tracing::warn!( page_id = %cleanup.page_id, error = %err, - "Browser terminal cleanup could not commit its receipt and owner release" + "Browser already-absent close could not retire its viewer ingress" ); - } else { - release_browser_open_job_instance_for_owner( - &state.data_dir, - &cleanup.principal_id, - &cleanup.owner_launch_id, - ) - .await; - settled = true; + false } } - (engine_result, stream_result) => { + } else { + true + }; + if (engine_result.is_ok() && stream_result.is_ok()) || (already_absent && ingress_released) + { + if let Err(err) = commit_browser_terminal_cleanup( + state, + &cleanup, + None, + engine_result + .as_ref() + .ok() + .and_then(browser_profile_durability_from_receipt), + ) + .await + { release_browser_engine_cleanup_claim(&state.data_dir, &cleanup).await; tracing::warn!( page_id = %cleanup.page_id, - engine_error = ?engine_result.err(), - stream_error = ?stream_result.err(), - "Browser pending engine cleanup retry failed" + error = %err, + "Browser terminal cleanup could not commit its receipt and owner release" ); + } else { + release_browser_open_job_instance_for_owner( + &state.data_dir, + &cleanup.principal_id, + &cleanup.owner_launch_id, + ) + .await; + settled = true; } + } else { + release_browser_engine_cleanup_claim(&state.data_dir, &cleanup).await; + tracing::warn!( + page_id = %cleanup.page_id, + engine_error = ?engine_result.err(), + stream_error = ?stream_result.err(), + "Browser pending engine cleanup retry failed" + ); } } settled @@ -2801,9 +2903,15 @@ async fn commit_browser_terminal_cleanup( state: &GatewayState, cleanup: &BrowserEngineCleanup, terminal_owner_launch_id: Option<&str>, + profile_durability: Option<&str>, ) -> Result<(), String> { - record_browser_reaped_page_tombstone(&state.data_dir, cleanup, terminal_owner_launch_id) - .await?; + record_browser_reaped_page_tombstone( + &state.data_dir, + cleanup, + terminal_owner_launch_id, + profile_durability, + ) + .await?; gateway_browser_remote::mark_consumer_terminal_retirement( &state.data_dir, &cleanup.principal_id, @@ -2821,6 +2929,11 @@ async fn commit_browser_terminal_cleanup( } async fn close_browser_page_record(state: &GatewayState, page: BrowserPageCleanup) { + forget_browser_page_status(&browser_page_status_cache_key( + &state.data_dir, + &page.engine_cleanup.principal_id, + &page.engine_cleanup.page_id, + )); let engine_cleanup = page.engine_cleanup; forget_browser_open_job_for_owner( &state.data_dir, @@ -2848,29 +2961,70 @@ async fn close_browser_page_record(state: &GatewayState, page: BrowserPageCleanu release_browser_stream_cleanup_claim(&state.data_dir, cleanup).await; } } - match (&engine_result, &stream_result) { - (Ok(_), Ok(())) => { - if let Err(err) = commit_browser_terminal_cleanup(state, &engine_cleanup, None).await { - release_browser_engine_cleanup_claim(&state.data_dir, &engine_cleanup).await; + let already_absent = matches!( + (&engine_result, &stream_result), + (Err(message), Ok(())) if browser_engine_close_is_already_absent(message) + ); + let ingress_released = if already_absent { + match release_absent_engine_viewer_ingress(state, &engine_cleanup).await { + Ok(()) => true, + Err(err) => { tracing::warn!( page_id = %engine_cleanup.page_id, error = %err, - "Browser stale page terminal receipt and release could not be committed" + "Browser already-absent stale close could not retire its viewer ingress" ); + false } } - _ => { + } else { + true + }; + if (engine_result.is_ok() && stream_result.is_ok()) || (already_absent && ingress_released) { + if let Err(err) = commit_browser_terminal_cleanup( + state, + &engine_cleanup, + None, + engine_result + .as_ref() + .ok() + .and_then(browser_profile_durability_from_receipt), + ) + .await + { release_browser_engine_cleanup_claim(&state.data_dir, &engine_cleanup).await; tracing::warn!( page_id = %engine_cleanup.page_id, - engine_error = ?engine_result.as_ref().err(), - stream_error = ?stream_result.as_ref().err(), - "Browser stale page engine cleanup failed" + error = %err, + "Browser stale page terminal receipt and release could not be committed" ); } + } else { + release_browser_engine_cleanup_claim(&state.data_dir, &engine_cleanup).await; + tracing::warn!( + page_id = %engine_cleanup.page_id, + engine_error = ?engine_result.as_ref().err(), + stream_error = ?stream_result.as_ref().err(), + "Browser stale page engine cleanup failed" + ); } } +fn browser_engine_close_is_already_absent(message: &str) -> bool { + message.contains("indeterminate after service restart: exact owned launcher unavailable") + || message.contains("Engine retained owner unavailable") +} + +async fn release_absent_engine_viewer_ingress( + state: &GatewayState, + cleanup: &BrowserEngineCleanup, +) -> Result<(), String> { + let Some(authority) = cleanup.transport_authority.as_ref() else { + return Ok(()); + }; + gateway_browser_remote::close_transport_listener(state, authority).await +} + async fn attempt_browser_engine_cleanup( state: &GatewayState, cleanup: &BrowserEngineCleanup, @@ -3102,7 +3256,17 @@ async fn reap_browser_open_effect_after_failure( ) .await; if engine_result.is_ok() && stream_result.is_ok() { - if let Err(err) = commit_browser_terminal_cleanup(state, &engine_cleanup, None).await { + if let Err(err) = commit_browser_terminal_cleanup( + state, + &engine_cleanup, + None, + engine_result + .as_ref() + .ok() + .and_then(browser_profile_durability_from_receipt), + ) + .await + { release_browser_engine_cleanup_claim(&state.data_dir, &engine_cleanup).await; tracing::warn!( page_id, @@ -3203,6 +3367,124 @@ async fn close_browser_stream_cleanup( } } +const BROWSER_PAGE_STATUS_INACTIVE: &str = "browser session is not active"; + +type BrowserPageStatusOutcome = Result; + +fn browser_page_status_flights() -> &'static StdMutex< + std::collections::HashMap>>, +> { + static FLIGHTS: std::sync::OnceLock< + StdMutex>>>, + > = std::sync::OnceLock::new(); + FLIGHTS.get_or_init(|| StdMutex::new(std::collections::HashMap::new())) +} + +fn browser_page_status_cache_key( + data_dir: &std::path::Path, + principal_id: &str, + page_id: &str, +) -> String { + format!("{}\n{principal_id}\n{page_id}", data_dir.display()) +} + +fn join_browser_page_status_flight( + key: String, +) -> (bool, Option>) { + let mut flights = match browser_page_status_flights().lock() { + Ok(guard) => guard, + Err(_) => return (true, None), + }; + if let Some(waiters) = flights.get_mut(&key) { + let (tx, rx) = oneshot::channel(); + waiters.push(tx); + return (false, Some(rx)); + } + flights.insert(key, Vec::new()); + (true, None) +} + +fn complete_browser_page_status_flight(key: &str, outcome: BrowserPageStatusOutcome) { + let waiters = match browser_page_status_flights().lock() { + Ok(mut flights) => flights.remove(key).unwrap_or_default(), + Err(_) => return, + }; + for waiter in waiters { + let _ = waiter.send(outcome.clone()); + } +} + +fn forget_browser_page_status(key: &str) { + complete_browser_page_status_flight(key, Err(BROWSER_PAGE_STATUS_INACTIVE.to_string())); +} + +fn browser_page_status_response(outcome: BrowserPageStatusOutcome) -> Response { + match outcome { + Ok(data) => Json(data).into_response(), + Err(message) if message == BROWSER_PAGE_STATUS_INACTIVE => { + (StatusCode::NOT_FOUND, BROWSER_PAGE_STATUS_INACTIVE).into_response() + } + Err(message) => gateway_provider_error_response("browser-engine", anyhow::anyhow!(message)), + } +} + +#[cfg(test)] +mod browser_page_status_flight_tests { + use super::{ + complete_browser_page_status_flight, forget_browser_page_status, + join_browser_page_status_flight, BROWSER_PAGE_STATUS_INACTIVE, + }; + + fn unique_key() -> String { + format!( + "test-page-status-{}", + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + ) + } + + #[test] + fn overlapping_reads_share_one_probe_and_the_next_read_starts_fresh() { + let key = unique_key(); + let (leader, waiter) = join_browser_page_status_flight(key.clone()); + assert!(leader); + assert!(waiter.is_none()); + let (leader, waiter) = join_browser_page_status_flight(key.clone()); + assert!(!leader); + let waiter = waiter.expect("follower waits for the in-flight probe"); + let first = serde_json::json!({"page_id":"page:one","actual_url":"https://example.test/"}); + complete_browser_page_status_flight(&key, Ok(first.clone())); + assert_eq!(waiter.blocking_recv().unwrap().unwrap(), first); + + let (leader, _) = join_browser_page_status_flight(key.clone()); + assert!(leader); + let (leader, waiter) = join_browser_page_status_flight(key.clone()); + assert!(!leader); + let waiter = waiter.expect("follower waits for the next probe"); + let second = + serde_json::json!({"page_id":"page:two","actual_url":"https://example.test/next"}); + complete_browser_page_status_flight(&key, Ok(second.clone())); + assert_eq!(waiter.blocking_recv().unwrap().unwrap(), second); + } + + #[test] + fn forget_completes_waiters_and_leaves_no_stored_observation() { + let key = unique_key(); + let _ = join_browser_page_status_flight(key.clone()); + let (_, waiter) = join_browser_page_status_flight(key.clone()); + let waiter = waiter.expect("follower waits for the in-flight probe"); + forget_browser_page_status(&key); + assert_eq!( + waiter.blocking_recv().unwrap(), + Err(BROWSER_PAGE_STATUS_INACTIVE.to_string()) + ); + let (leader, _) = join_browser_page_status_flight(key); + assert!(leader); + } +} + pub(super) async fn browser_app_page_status( State(state): State, headers: HeaderMap, @@ -3220,40 +3502,58 @@ pub(super) async fn browser_app_page_status( } let principal_id = context.principal_id; if !touch_browser_page(&state.data_dir, &page_id, &principal_id, &owner_launch_id).await { - return (StatusCode::NOT_FOUND, "browser session is not active").into_response(); + forget_browser_page_status(&browser_page_status_cache_key( + &state.data_dir, + &principal_id, + &page_id, + )); + return (StatusCode::NOT_FOUND, BROWSER_PAGE_STATUS_INACTIVE).into_response(); } - let call = match browser_provider_resource_call( - "browser-engine", - "page_status", - "elastos://browser-engine/page/status".to_string(), - serde_json::json!({ - "page_id": page_id, - "principal_id": principal_id.clone(), - }), - ) { - Ok(call) => call, - Err((status, message)) => return (status, message).into_response(), - }; - let response = match browser_provider_resource_response(&state, call).await { - Ok(value) => value, - Err((_status, message)) => { - return gateway_provider_error_response("browser-engine", anyhow::anyhow!(message)); - } - }; - if let Some(message) = provider_response_error_message(&response) { - return gateway_provider_error_response("browser-engine", anyhow::anyhow!(message)); + let status_key = browser_page_status_cache_key(&state.data_dir, &principal_id, &page_id); + let (leader, waiter) = join_browser_page_status_flight(status_key.clone()); + if !leader { + return match waiter { + Some(rx) => match rx.await { + Ok(outcome) => browser_page_status_response(outcome), + Err(_) => gateway_provider_error_response( + "browser-engine", + anyhow::anyhow!("browser-engine page-status probe closed"), + ), + }, + None => gateway_provider_error_response( + "browser-engine", + anyhow::anyhow!("browser-engine page-status probe is unavailable"), + ), + }; } - match provider_response_data(&response) { - Some(data) => { - let _ = touch_browser_page(&state.data_dir, &page_id, &principal_id, &owner_launch_id) - .await; - Json(data).into_response() - } - None => gateway_provider_error_response( + let outcome = async { + let call = browser_provider_resource_call( "browser-engine", - anyhow::anyhow!("browser-engine provider returned an invalid page-status response"), - ), + "page_status", + "elastos://browser-engine/page/status".to_string(), + serde_json::json!({ + "page_id": page_id, + "principal_id": principal_id.clone(), + }), + ) + .map_err(|(_status, message)| message)?; + let response = browser_provider_resource_response(&state, call) + .await + .map_err(|(_status, message)| message)?; + if let Some(message) = provider_response_error_message(&response) { + return Err(message); + } + provider_response_data(&response).ok_or_else(|| { + "browser-engine provider returned an invalid page-status response".to_string() + }) } + .await; + complete_browser_page_status_flight(&status_key, outcome.clone()); + if outcome.is_ok() { + let _ = + touch_browser_page(&state.data_dir, &page_id, &principal_id, &owner_launch_id).await; + } + browser_page_status_response(outcome) } pub(super) async fn browser_app_page_diagnostics( @@ -3682,6 +3982,7 @@ pub(super) async fn browser_app_page_close( &page_id, &cleanup_id, receipt.browser_instance.as_deref(), + receipt.profile_durability.as_deref(), )) .into_response(); } @@ -3735,8 +4036,38 @@ pub(super) async fn browser_app_page_close( } else { close_browser_stream_cleanup(&state, page_cleanup.stream_cleanup).await }; - let terminal = if engine_result.is_ok() && stream_result.is_ok() { - commit_browser_terminal_cleanup(&state, &engine_cleanup, Some(&owner_launch_id)).await + let already_absent = matches!( + (&engine_result, &stream_result), + (Err(message), Ok(())) if browser_engine_close_is_already_absent(message) + ); + let ingress_released = if already_absent { + match release_absent_engine_viewer_ingress(&state, &engine_cleanup).await { + Ok(()) => true, + Err(err) => { + tracing::warn!( + page_id = %engine_cleanup.page_id, + error = %err, + "Browser already-absent close could not retire its viewer ingress" + ); + false + } + } + } else { + true + }; + let terminal = if (engine_result.is_ok() && stream_result.is_ok()) + || (already_absent && ingress_released) + { + commit_browser_terminal_cleanup( + &state, + &engine_cleanup, + Some(&owner_launch_id), + engine_result + .as_ref() + .ok() + .and_then(browser_profile_durability_from_receipt), + ) + .await } else { release_browser_engine_cleanup_claim(&state.data_dir, &engine_cleanup).await; Ok(()) @@ -3754,6 +4085,15 @@ pub(super) async fn browser_app_page_close( &receipt, )) .into_response(), + Err(_) if already_absent && ingress_released => { + Json(browser_public_already_absent_close_receipt( + &page_id, + &cleanup_id, + cleanup_browser_instance, + None, + )) + .into_response() + } Err(message) => gateway_provider_error_response("browser-engine", anyhow::anyhow!(message)), } } @@ -3762,6 +4102,7 @@ fn browser_public_already_absent_close_receipt( page_id: &str, cleanup_id: &str, browser_instance: Option<&str>, + profile_durability: Option<&str>, ) -> serde_json::Value { let mut receipt = serde_json::json!({ "schema": "elastos.browser.close-result/v1", @@ -3781,6 +4122,11 @@ fn browser_public_already_absent_close_receipt( if let Some(browser_instance) = browser_instance { receipt["browser_instance"] = serde_json::json!(browser_instance); } + if let Some(durability) = + profile_durability.filter(|value| matches!(*value, "failed" | "unknown" | "proved")) + { + receipt["profile_durability"] = serde_json::json!(durability); + } receipt } @@ -3809,6 +4155,9 @@ fn browser_public_terminal_close_receipt( if let Some(browser_instance) = cleanup.browser_instance.as_deref() { public_receipt["browser_instance"] = serde_json::json!(browser_instance); } + if let Some(durability) = browser_profile_durability_from_receipt(receipt) { + public_receipt["profile_durability"] = serde_json::json!(durability); + } if let (Some(authority), Some(transport_receipt)) = ( cleanup.provider_cleanup.get("transport_authority"), cleanup.provider_cleanup.get("transport_receipt"), @@ -3890,6 +4239,174 @@ pub(super) fn browser_terminal_close_receipt( Ok(data) } +type BrowserDisplayAttachOutcome = Result; + +fn display_attach_flights() -> &'static StdMutex< + std::collections::HashMap>>, +> { + static FLIGHTS: std::sync::OnceLock< + StdMutex< + std::collections::HashMap>>, + >, + > = std::sync::OnceLock::new(); + FLIGHTS.get_or_init(|| StdMutex::new(std::collections::HashMap::new())) +} + +fn display_attach_flight_key( + data_dir: &std::path::Path, + principal_id: &str, + page_id: &str, + request_id: &str, +) -> String { + format!( + "{}\n{principal_id}\n{page_id}\n{request_id}", + data_dir.display() + ) +} + +fn join_display_attach_flight( + key: String, +) -> Result<(bool, oneshot::Receiver), ()> { + let mut flights = display_attach_flights().lock().map_err(|_| ())?; + let (tx, rx) = oneshot::channel(); + if let Some(waiters) = flights.get_mut(&key) { + waiters.push(tx); + return Ok((false, rx)); + } + flights.insert(key, vec![tx]); + Ok((true, rx)) +} + +fn complete_display_attach_flight(key: &str, outcome: BrowserDisplayAttachOutcome) { + let waiters = match display_attach_flights().lock() { + Ok(mut flights) => flights.remove(key).unwrap_or_default(), + Err(_) => return, + }; + for waiter in waiters { + let _ = waiter.send(outcome.clone()); + } +} + +async fn execute_browser_display_attach( + state: GatewayState, + dispatch: BrowserWebrtcDispatch, + page_id: String, + principal_id: String, + signal: serde_json::Value, + transport_authority: Option, +) -> BrowserDisplayAttachOutcome { + let request = serde_json::json!({ + "page_id": page_id, + "signal": signal, + "principal_id": principal_id, + }); + let outcome = async { + let call = browser_provider_resource_call( + "browser-engine", + "webrtc_signal", + "elastos://browser-engine/page/webrtc_signal".to_string(), + request, + ) + .map_err(|_| BrowserDisplayError::Uncertain)?; + let response = match browser_provider_resource_response(&state, call).await { + Ok(response) => response, + Err((_status, _message)) => return Err(BrowserDisplayError::Uncertain), + }; + let mut envelope = &response; + for _ in 0..4 { + match envelope.get("status").and_then(serde_json::Value::as_str) { + Some("error") => { + return Err(envelope + .get("code") + .and_then(serde_json::Value::as_str) + .and_then(BrowserDisplayError::from_code) + .unwrap_or(BrowserDisplayError::Uncertain)); + } + Some("ok") => match envelope.get("data") { + Some(data) => envelope = data, + None => break, + }, + _ => break, + } + } + let data = provider_response_data(&response).ok_or(BrowserDisplayError::Uncertain)?; + validate_browser_display_attach_result( + &data, + &page_id, + signal["request_id"].as_str().unwrap_or(""), + signal["display_generation"].as_str().unwrap_or(""), + )?; + Ok(data) + } + .await; + let data = finish_browser_page_webrtc(&dispatch, outcome).await?; + if let (Some(authority), Some(endpoint)) = ( + transport_authority.as_ref(), + state.carrier_endpoint.as_ref(), + ) { + if let Ok(Some(binding)) = gateway_browser_remote::consumer_binding( + &state.data_dir, + &serde_json::json!({ + "principal_id": principal_id, + "page_id": page_id, + }), + ) { + crate::carrier::browser_engine_media::rearm_media_stream( + &state.data_dir, + endpoint, + &binding.grant, + authority, + ) + .await; + } + } + Ok(data) +} + +fn browser_display_attach_http_response(outcome: BrowserDisplayAttachOutcome) -> Response { + match outcome { + Ok(data) => Json(data).into_response(), + Err(error) => browser_display_error_response(error), + } +} + +#[cfg(test)] +mod display_attach_flight_tests { + use super::{complete_display_attach_flight, join_display_attach_flight}; + use elastos_common::browser_protocol::BrowserDisplayError; + + fn unique_key() -> String { + format!( + "test-display-attach-{}", + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + ) + } + + #[test] + fn overlapping_same_id_shares_one_flight_and_the_next_starts_fresh() { + let key = unique_key(); + let (leader, first) = join_display_attach_flight(key.clone()).unwrap(); + assert!(leader); + let (leader, second) = join_display_attach_flight(key.clone()).unwrap(); + assert!(!leader); + let receipt = serde_json::json!({"request_id":"a","page_id":"page:one"}); + complete_display_attach_flight(&key, Ok(receipt.clone())); + assert_eq!(first.blocking_recv().unwrap().unwrap(), receipt); + assert_eq!(second.blocking_recv().unwrap().unwrap(), receipt); + + let (leader, retry) = join_display_attach_flight(key.clone()).unwrap(); + assert!(leader); + complete_display_attach_flight(&key, Err(BrowserDisplayError::Uncertain)); + assert_eq!( + retry.blocking_recv().unwrap(), + Err(BrowserDisplayError::Uncertain) + ); + } +} + pub(super) async fn browser_app_page_webrtc( State(state): State, headers: HeaderMap, @@ -3937,6 +4454,16 @@ pub(super) async fn browser_app_page_webrtc( return (StatusCode::NOT_FOUND, "browser session is not active").into_response(); } }; + if signal_type == "display_attach" { + if let Some(authority) = transport_authority.as_ref() { + let data_dir = state.data_dir.clone(); + let authority = authority.clone(); + tokio::spawn(async move { + crate::carrier::browser_engine_media::preadmit_media_stream(&data_dir, &authority) + .await; + }); + } + } if let Some(authority) = transport_authority.as_ref() { if let Err(message) = normalize_browser_vz_webrtc_signal(authority, &mut signal) { return gateway_provider_error_response("browser", anyhow::anyhow!(message)); @@ -3960,6 +4487,72 @@ pub(super) async fn browser_app_page_webrtc( Err(error) => browser_display_error_response(error), }; } + if signal_type == "display_attach" { + let request_id = signal + .get("request_id") + .and_then(serde_json::Value::as_str) + .unwrap_or("") + .to_string(); + let flight_key = + display_attach_flight_key(&state.data_dir, &principal_id, &page_id, &request_id); + return match join_display_attach_flight(flight_key.clone()) { + Ok((owns, waiter)) => { + if owns { + match begin_browser_page_webrtc( + &state.data_dir, + &page_id, + &principal_id, + &owner_launch_id, + &signal, + ) + .await + { + Ok(again) if again.cached.is_some() => { + let cached = again + .cached + .clone() + .expect("display attach cache was present"); + let outcome = finish_browser_page_webrtc(&again, Ok(cached)).await; + complete_display_attach_flight(&flight_key, outcome); + } + Ok(_) => { + let work_state = state.clone(); + tokio::spawn(async move { + let outcome = execute_browser_display_attach( + work_state, + dispatch, + page_id, + principal_id, + signal, + transport_authority, + ) + .await; + complete_display_attach_flight(&flight_key, outcome); + }); + } + Err(error) => { + complete_display_attach_flight(&flight_key, Err(error)); + } + } + } + match waiter.await { + Ok(outcome) => browser_display_attach_http_response(outcome), + Err(_) => browser_display_error_response(BrowserDisplayError::Uncertain), + } + } + Err(()) => browser_display_attach_http_response( + execute_browser_display_attach( + state, + dispatch, + page_id, + principal_id, + signal, + transport_authority, + ) + .await, + ), + }; + } let mut request = serde_json::json!({ "page_id": page_id, "signal": signal, "principal_id": principal_id, }); @@ -4041,7 +4634,28 @@ pub(super) async fn browser_app_page_webrtc( } .await; match finish_browser_page_webrtc(&dispatch, outcome).await { - Ok(data) => Json(data).into_response(), + Ok(data) => { + if signal_type == "display_attach" { + if let (Some(authority), Some(endpoint)) = ( + transport_authority.as_ref(), + state.carrier_endpoint.as_ref(), + ) { + if let Ok(Some(binding)) = gateway_browser_remote::consumer_binding( + &state.data_dir, + &serde_json::json!({"principal_id": principal_id, "page_id": page_id}), + ) { + crate::carrier::browser_engine_media::rearm_media_stream( + &state.data_dir, + endpoint, + &binding.grant, + authority, + ) + .await; + } + } + } + Json(data).into_response() + } Err(BrowserDisplayError::Uncertain) if legacy_error.is_some() => { gateway_provider_error_response( "browser-engine", diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs b/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs index b20d050ec..84130d228 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_engine.rs @@ -4,6 +4,7 @@ use super::*; use elastos_common::browser_protocol::{ BrowserEngineReadiness, BrowserEngineReadinessReason, BROWSER_ENGINE_READINESS_SCHEMA, }; +use sha2::Sha256; pub(in crate::api::gateway) async fn resolve_browser_engine_adapter( registry: &ProviderRegistry, @@ -881,6 +882,64 @@ mod tests { } } +const REMOTE_ENGINE_SUMMARY_TTL: std::time::Duration = std::time::Duration::from_secs(8); + +struct RemoteEngineSummaryCacheEntry { + fingerprint: String, + expires_at: std::time::Instant, + value: serde_json::Value, +} + +fn remote_engine_summary_cache( +) -> &'static std::sync::Mutex> { + static CACHE: std::sync::OnceLock< + std::sync::Mutex>, + > = std::sync::OnceLock::new(); + CACHE.get_or_init(|| std::sync::Mutex::new(std::collections::HashMap::new())) +} + +fn remote_engine_cache_key(data_dir: &FsPath, principal_id: &str) -> String { + hex::encode(Sha256::digest( + format!("{}\n{principal_id}", data_dir.display()).as_bytes(), + )) +} + +fn remote_engine_grant_fingerprint(grants: &[serde_json::Value]) -> String { + let mut rows = grants + .iter() + .map(|grant| { + format!( + "{}\n{}\n{}", + grant["id"].as_str().unwrap_or(""), + grant["expires_at"].as_u64().unwrap_or(0), + grant["operations"] + ) + }) + .collect::>(); + rows.sort(); + hex::encode(Sha256::digest(rows.join("\n").as_bytes())) +} + +fn cached_remote_engine_summary(key: &str, fingerprint: &str) -> Option { + let cache = remote_engine_summary_cache().lock().ok()?; + let entry = cache.get(key)?; + (entry.fingerprint == fingerprint && std::time::Instant::now() < entry.expires_at) + .then(|| entry.value.clone()) +} + +fn store_remote_engine_summary(key: String, fingerprint: String, value: serde_json::Value) { + if let Ok(mut cache) = remote_engine_summary_cache().lock() { + cache.insert( + key, + RemoteEngineSummaryCacheEntry { + fingerprint, + expires_at: std::time::Instant::now() + REMOTE_ENGINE_SUMMARY_TTL, + value, + }, + ); + } +} + /// Approved remote services are observed through the same Runtime provider plane. /// Execution selections retain an opaque Runtime route to their exact grant. pub(in crate::api::gateway) async fn browser_remote_engine_summary( @@ -900,6 +959,11 @@ pub(in crate::api::gateway) async fn browser_remote_engine_summary( let Some(registry) = state.provider_registry.as_ref() else { return serde_json::json!({"state":"unavailable","offers":[]}); }; + let fingerprint = remote_engine_grant_fingerprint(&grants); + let cache_key = remote_engine_cache_key(&state.data_dir, &context.principal_id); + if let Some(cached) = cached_remote_engine_summary(&cache_key, &fingerprint) { + return cached; + } let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5); let mut offers = Vec::new(); for grant in grants { @@ -966,5 +1030,43 @@ pub(in crate::api::gateway) async fn browser_remote_engine_summary( } offers.push(visible); } - serde_json::json!({"state":"available","offers":offers}) + let value = serde_json::json!({"state":"available","offers":offers}); + store_remote_engine_summary(cache_key, fingerprint, value.clone()); + value +} + +#[cfg(test)] +mod remote_engine_summary_cache_tests { + use super::*; + + #[test] + fn grant_fingerprint_changes_when_offer_identity_changes() { + let first = + serde_json::json!([{"id":"remote-engine-aa","expires_at":10,"operations":["status"]}]); + let second = + serde_json::json!([{"id":"remote-engine-bb","expires_at":10,"operations":["status"]}]); + assert_ne!( + remote_engine_grant_fingerprint(first.as_array().unwrap()), + remote_engine_grant_fingerprint(second.as_array().unwrap()) + ); + } + + #[test] + fn cache_returns_the_stored_observation_until_the_grant_set_changes() { + let key = format!( + "test-{}", + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let fingerprint = "fp-one".to_string(); + let value = serde_json::json!({"state":"available","offers":[{"id":"cached"}]}); + store_remote_engine_summary(key.clone(), fingerprint.clone(), value.clone()); + assert_eq!( + cached_remote_engine_summary(&key, &fingerprint), + Some(value) + ); + assert_eq!(cached_remote_engine_summary(&key, "fp-two"), None); + } } diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs b/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs index cb6b31dc9..a6dd9f432 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_remote.rs @@ -249,6 +249,55 @@ fn prepare_intent(request: &Value) -> Result { Ok(Value::Object(request)) } +async fn status_from_settlement_authority( + registry: Arc, + source: iroh::PublicKey, + request: &Value, +) -> Result { + let authority = request + .get("transport_authority") + .filter(|value| value.is_object()) + .context("Engine retained owner unavailable")?; + let generation = request["lifecycle_generation"] + .as_str() + .context("Engine retained owner unavailable")?; + let page_id = request["page_id"] + .as_str() + .context("Engine retained owner unavailable")?; + let stream_id = request["stream_id"] + .as_str() + .context("Engine retained owner unavailable")?; + let principal = request["principal_id"] + .as_str() + .context("Engine retained owner unavailable")?; + let expected_page = format!( + "page:vz-{}", + hex::encode(Sha256::digest(format!("{generation}\npage"))) + ); + let storage = RemoteEngineOwner::storage_principal_for(&source.to_string(), principal); + ensure!( + page_id == expected_page + && authority.get("generation").and_then(Value::as_str) == Some(generation) + && authority.get("page_id").and_then(Value::as_str) == Some(page_id) + && authority + .pointer("/egress/stream_id") + .and_then(Value::as_str) + == Some(stream_id) + && authority.get("principal_id").and_then(Value::as_str) == Some(storage.as_str()), + "Engine requester or generation changed" + ); + let mut local = request.clone(); + let object = local.as_object_mut().context("Engine request required")?; + for key in ["grant_id", "page_id", "_runtime_invocation"] { + object.remove(key); + } + local["op"] = json!("status"); + local["principal_id"] = json!(storage); + local["lifecycle_generation"] = json!(generation); + local["stream_id"] = json!(stream_id); + Ok(registry.send_raw("browser-engine", &local).await?) +} + pub(crate) async fn invoke( data_dir: &FsPath, registry: Arc, @@ -376,13 +425,12 @@ pub(crate) async fn invoke( }); page } + } else if let Some(page) = pages.lock().await.get(&key).cloned() { + page + } else if operation == "status" { + return status_from_settlement_authority(registry, source, request).await; } else { - pages - .lock() - .await - .get(&key) - .cloned() - .context("Engine retained owner unavailable")? + return Err(anyhow::anyhow!("Engine retained owner unavailable")); }; ensure!( page.owner.matches_request(&source, request), @@ -565,6 +613,126 @@ impl ServedPage { } } +const REMOTE_WALLET_CONSUMER_MEDIATION_SCHEMA: &str = + "elastos.browser.wallet-consumer-mediation/v1"; +const REMOTE_WALLET_CONSUMER_REQUEST_SCHEMA: &str = "elastos.browser.wallet-consumer-request/v1"; + +fn remote_wallet_consumer_mediation( + requester_endpoint: &str, + page_id: &str, + generation: &str, +) -> Value { + json!({ + "schema": REMOTE_WALLET_CONSUMER_MEDIATION_SCHEMA, + "resolution": "consumer_runtime", + "requester_endpoint": requester_endpoint, + "page_id": page_id, + "lifecycle_generation": generation, + "bus": "private", + "authority": "consumer_runtime", + }) +} + +fn remote_wallet_consumer_request( + mediation: &Value, + operation: &str, + page_url: &str, + origin: &str, + payload: Value, +) -> Result { + ensure!( + mediation["schema"] == REMOTE_WALLET_CONSUMER_MEDIATION_SCHEMA + && mediation["resolution"] == "consumer_runtime" + && mediation["authority"] == "consumer_runtime" + && mediation["bus"] == "private", + "Remote Wallet request requires consumer mediation" + ); + ensure!( + mediation.get("home_token").is_none() && mediation.get("account_access_url").is_none(), + "Remote Wallet request keeps tokens on the consumer" + ); + ensure!( + matches!( + operation, + "request_accounts" + | "request_signature" + | "request_transaction" + | "approval_status" + | "read" + | "broadcast_transaction" + ), + "Remote Wallet operation is outside the consumer mediation set" + ); + let requester_endpoint = mediation["requester_endpoint"] + .as_str() + .context("Remote Wallet consumer peer required")?; + let page_id = mediation["page_id"] + .as_str() + .context("Remote Wallet page required")?; + let generation = mediation["lifecycle_generation"] + .as_str() + .context("Remote Wallet generation required")?; + Ok(json!({ + "schema": REMOTE_WALLET_CONSUMER_REQUEST_SCHEMA, + "resolution": "consumer_runtime", + "requester_endpoint": requester_endpoint, + "page_id": page_id, + "lifecycle_generation": generation, + "operation": operation, + "page_url": page_url, + "origin": origin, + "payload": payload, + "bus": "private", + "authority": "consumer_runtime", + })) +} + +fn authenticated_engine_peer(binding: &ConsumerBinding) -> Result { + binding.grant["peer_did"] + .as_str() + .context("Remote Wallet Engine peer required")? + .parse() + .context("Remote Wallet Engine peer invalid") +} + +#[allow(dead_code)] +pub(in crate::api::gateway) fn admit_remote_wallet_consumer_request( + binding: &ConsumerBinding, + request: &Value, + source: &iroh::PublicKey, +) -> Result<()> { + ensure!( + request["schema"] == REMOTE_WALLET_CONSUMER_REQUEST_SCHEMA + && request["resolution"] == "consumer_runtime" + && request["page_id"] == binding.page_id + && request["lifecycle_generation"] == binding.generation + && request.get("home_token").is_none() + && request.get("account_access_url").is_none(), + "Remote Wallet request is not bound to this consumer page" + ); + ensure!( + *source == authenticated_engine_peer(binding)?, + "Remote Wallet request is not bound to the authenticated Engine peer" + ); + Ok(()) +} + +#[allow(dead_code)] +pub(in crate::api::gateway) fn forward_remote_wallet_consumer_request( + binding: &ConsumerBinding, + launch_wallet: &Value, + source: &iroh::PublicKey, + operation: &str, + page_url: &str, + origin: &str, + payload: Value, +) -> Result { + let request = + remote_wallet_consumer_request(launch_wallet, operation, page_url, origin, payload)?; + admit_remote_wallet_consumer_request(binding, &request, source)?; + Ok(request) +} + async fn prepare(page: &ServedPage) -> Result { let gateway = &page.gateway; let registry = gateway @@ -638,9 +806,27 @@ async fn prepare(page: &ServedPage) -> Result { profile["profile_key"] = json!(owner.profile_key); let viewer = browser_vz_viewer_turn_capability(&transport.authority, &transport.secret).map_err(anyhow::Error::msg)?; crate::carrier::browser_engine_media::bind_target(&gateway.data_dir, owner.requester_endpoint.parse()?, &transport.authority, page.closing.clone()).await?; + let wallet = remote_wallet_consumer_mediation(&owner.requester_endpoint, &owner.page_id, &owner.generation); + let origin = url::Url::parse(&url) + .ok() + .map(|parsed| parsed.origin().ascii_serialization()) + .filter(|origin| origin.starts_with("http")) + .context("Engine Wallet page origin required")?; + ensure!( + remote_wallet_consumer_request( + &wallet, + "request_accounts", + &url, + &origin, + json!({"method": "eth_requestAccounts"}), + )?["page_id"] + == owner.page_id, + "Engine Wallet launch request escaped its page" + ); let request = json!({"op":"launch","url":url,"stream_session":browser_engine_stream_session(&stream), "lifecycle_generation":owner.generation,"principal_id":principal,"profile":profile, - "wallet":{},"viewport":intent["viewport"],"display_mode":mode,"guarantee_level":guarantee, + "wallet":wallet, + "viewport":intent["viewport"],"display_mode":mode,"guarantee_level":guarantee, "adapter_id":owner.adapter_id,"page_id":owner.page_id,"vm_id":owner.vm_id, "transport_authority":transport.authority,"transport_secret":transport.secret}); Ok::<_, anyhow::Error>(Prepared {reservation:reservation.clone(),owner_launch_id, @@ -893,7 +1079,7 @@ async fn close(page: &ServedPage) -> Result { &page.owner.generation, ) .await?; - commit_browser_terminal_cleanup(&page.gateway, &cleanup.engine_cleanup, None) + commit_browser_terminal_cleanup(&page.gateway, &cleanup.engine_cleanup, None, None) .await .map_err(anyhow::Error::msg)?; release_browser_page_for_principal( @@ -1193,6 +1379,86 @@ pub(super) fn consumer_terminal_retirement( .is_some_and(|binding| binding.terminal_retirement)) } +pub(super) fn consumer_grant_expired( + data_dir: &FsPath, + principal: &str, + generation: &str, +) -> Result { + Ok(consumer_binding( + data_dir, + &json!({"principal_id":principal,"lifecycle_generation":generation}), + )? + .is_some_and(|binding| { + binding.grant["expires_at"] + .as_u64() + .is_none_or(|expiry| expiry <= now_ts()) + })) +} + +#[cfg(test)] +pub(in crate::api::gateway) fn consumer_binding_for_wallet_forward_test( + principal_id: &str, + page_id: &str, + generation: &str, + engine_peer: &iroh::PublicKey, +) -> ConsumerBinding { + ConsumerBinding { + grant: json!({"peer_did": engine_peer.to_string()}), + selection_id: "remote-engine-selection".into(), + adapter_id: "mock-browser-engine".into(), + principal_id: principal_id.into(), + page_id: page_id.into(), + generation: generation.into(), + stream_id: "stream:wallet-forward".into(), + prepared: None, + dispatch_started: true, + preparation_closed: false, + owner_launch_id: Some("launch:wallet-forward".into()), + browser_instance: Some("window:wallet-forward".into()), + reservation: None, + stream_cleanup: None, + cleanup_pending: false, + retry_after: u64::MAX, + terminal_retirement: false, + } +} + +#[cfg(test)] +pub(in crate::api::gateway) fn write_consumer_binding_for_test( + data_dir: &FsPath, + reservation: &BrowserLaunchReservation, + principal_id: &str, + stream_id: &str, + expires_at: u64, +) -> Result<()> { + insert_consumer( + data_dir, + &ConsumerBinding { + grant: json!({ + "id": "grant:reconciliation-test", + "grant_id": "grant:reconciliation-test", + "expires_at": expires_at, + }), + selection_id: reservation.engine_route_provider().to_string(), + adapter_id: "mock-browser-engine".into(), + principal_id: principal_id.into(), + page_id: reservation.page_id().to_string(), + generation: reservation.generation().to_string(), + stream_id: stream_id.into(), + prepared: None, + dispatch_started: true, + preparation_closed: false, + owner_launch_id: None, + browser_instance: None, + reservation: None, + stream_cleanup: None, + cleanup_pending: true, + retry_after: 0, + terminal_retirement: false, + }, + ) +} + pub(super) fn mark_consumer_reservation_terminal( data_dir: &FsPath, reservation: &BrowserLaunchReservation, @@ -1462,6 +1728,10 @@ pub(in crate::api::gateway) async fn prepare_consumer( terminal_retirement: false, }; insert_consumer(&gateway.data_dir, &binding)?; + ensure!( + authenticated_engine_peer(&binding)? == peer, + "Remote Wallet Engine peer differs from the selected grant" + ); browser_engine_binding::bind_egress( &gateway.data_dir, peer, @@ -1478,6 +1748,12 @@ pub(in crate::api::gateway) async fn prepare_consumer( let request = json!({"lifecycle_generation":reservation.generation(),"page_id":reservation.page_id(),"vm_id":reservation.vm_id(), "profile_key":profile["profile_key"],"adapter_id":choice.1,"stream_id":stream_id,"target":stream["target"], "url":input["url"],"viewport":input["viewport"],"display_mode":input["display_mode"],"guarantee_level":input["guarantee_level"]}); + let warm = gateway.carrier_endpoint.clone().map(|endpoint| { + let grant = choice.0.clone(); + tokio::spawn(async move { + crate::carrier::browser_engine_media::preconnect_engine(&endpoint, &grant).await + }) + }); let response = crate::carrier::call_browser_engine(registry, &choice.0, "prepare_launch", request).await?; let data = provider_response_data(&response).context("Remote Engine preparation missing")?; @@ -1517,6 +1793,7 @@ pub(in crate::api::gateway) async fn prepare_consumer( endpoint, &choice.0, &data["transport_authority"], + warm, ) .await?; let mut viewer = data["viewer_turn_capability"].clone(); @@ -1824,6 +2101,69 @@ mod tests { } } + fn test_engine_peer() -> iroh::PublicKey { + iroh::SecretKey::from_bytes(&[9; 32]).public() + } + + #[test] + fn remote_wallet_consumer_mediation_keeps_wallet_bus_on_the_consumer() { + let wallet = remote_wallet_consumer_mediation( + "peer-alice", + "page:vz-example", + "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ); + assert_eq!(wallet["schema"], REMOTE_WALLET_CONSUMER_MEDIATION_SCHEMA); + assert_eq!(wallet["resolution"], "consumer_runtime"); + assert_eq!(wallet["bus"], "private"); + assert_eq!(wallet["requester_endpoint"], "peer-alice"); + assert!(wallet.get("principal_id").is_none()); + assert!(wallet.get("home_token").is_none()); + assert!(wallet.get("bridge_url").is_none()); + } + + #[test] + fn remote_wallet_consumer_request_binds_authenticated_peer_page_and_generation() { + let engine = test_engine_peer(); + let binding = consumer_binding_for_wallet_forward_test( + "person:consumer", + "page:7", + "generation:7", + &engine, + ); + let mediation = remote_wallet_consumer_mediation( + "peer-consumer", + &binding.page_id, + &binding.generation, + ); + let request = remote_wallet_consumer_request( + &mediation, + "request_accounts", + "https://ela.city/", + "https://ela.city", + json!({"method": "eth_requestAccounts"}), + ) + .expect("consumer Wallet request"); + assert_eq!(request["schema"], REMOTE_WALLET_CONSUMER_REQUEST_SCHEMA); + assert!(request.get("home_token").is_none()); + assert!(request.get("principal_id").is_none()); + admit_remote_wallet_consumer_request(&binding, &request, &engine).expect("matching page"); + let mut foreign_page = binding.clone(); + foreign_page.page_id = "page:other".into(); + assert!(admit_remote_wallet_consumer_request(&foreign_page, &request, &engine).is_err()); + let foreign_peer = iroh::SecretKey::from_bytes(&[23; 32]).public(); + assert!(admit_remote_wallet_consumer_request(&binding, &request, &foreign_peer).is_err()); + let mut tokenized = mediation.clone(); + tokenized["home_token"] = json!("launch-token"); + assert!(remote_wallet_consumer_request( + &tokenized, + "request_accounts", + "https://ela.city/", + "https://ela.city", + json!({}) + ) + .is_err()); + } + #[test] fn remote_engine_consumer_dispatch_and_retirement_reject_stale_writes() { let root = tempfile::tempdir().unwrap(); diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs b/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs index 58c39b89f..e1b2dc1cd 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_route_tests.rs @@ -3139,7 +3139,7 @@ async fn test_exact_terminal_vz_settlement_releases_restart_reconciliation_oblig let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); assert_eq!(payload["outcome"]["state"], "terminal_post_effect_cleanup"); assert_eq!(payload["outcome"]["effects"]["page_acquired"], false); - assert_eq!(payload["outcome"]["effects"]["vm_acquired"], true); + assert_eq!(payload["outcome"]["effects"]["vm_acquired"], false); assert_eq!(reconciliation_calls.count(), 1); assert!(close_calls.snapshot().await.is_empty()); assert_eq!(browser_page_session_count(dir.path()).await, 0); @@ -4377,48 +4377,86 @@ async fn test_browser_eth_send_transaction_allows_external_connector_approval() ); } +fn completed_browser_personal_sign_approval( + context: &elastos_wallet_contract::VerifiedWalletInvocationContext, + request_id: &str, + page_url: &str, + origin: &str, +) -> serde_json::Value { + json!({ + "request_id": request_id, + "status": "completed", + "intent": "browser_personal_sign", + "requested_by_actor": BROWSER_CAPSULE_ID, + "capsule_id": BROWSER_CAPSULE_ID, + "resource": "elastos://wallet/eip155:1/sign/browser_personal_sign", + "reason": "Browser page requests personal_sign", + "account_id": "wallet:eip155:1:0x1111111111111111111111111111111111111111", + "chain_namespace": "eip155:1", + "address": "0x1111111111111111111111111111111111111111", + "proof_type": "managed_evm", + "payload_hash": "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "principal_id": context.principal_id(), + "session_id": context.session_id(), + "launch_id": context.launch_id(), + "created_at": 10, + "expires_at": 20, + "completed_at": 12, + "payload": { + "schema": "elastos.browser.wallet-signature-request/v1", + "method": "personal_sign", + "page_url": page_url, + "origin": origin, + "principal_id": context.principal_id(), + "session_id": context.session_id(), + "requires_wallet_approval": true + }, + "signed_result": { + "schema": "elastos.browser.personal-sign-result/v1", + "request_id": request_id, + "method": "personal_sign", + "signature": "0xsigned", + "signer": "0x1111111111111111111111111111111111111111" + } + }) +} + #[tokio::test] async fn test_browser_wallet_approval_status_returns_completed_signature() { let dir = tempfile::tempdir().unwrap(); let authority = passkey_authority(dir.path()); let browser_token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let runtime_authority = + runtime_wallet_authority_for_app_token(dir.path(), BROWSER_CAPSULE_ID, &browser_token); + let context = runtime_authority.verified_context(); + let page_url = "https://dapp.example/sign"; + let origin = "https://dapp.example"; let provider = MockWalletProvider { challenges: TokioMutex::default(), bitcoin_challenges: TokioMutex::default(), accounts: TokioMutex::default(), - approvals: TokioMutex::new(vec![json!({ - "request_id": "wallet-approval:browser", - "status": "completed", - "intent": "browser_personal_sign", - "capsule_id": BROWSER_CAPSULE_ID, - "resource": "elastos://wallet/eip155:1/sign/browser_personal_sign", - "reason": "Browser page requests personal_sign", - "account_id": "wallet:eip155:1:0x1111111111111111111111111111111111111111", - "chain_namespace": "eip155:1", - "address": "0x1111111111111111111111111111111111111111", - "proof_type": "managed_evm", - "payload_hash": "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", - "principal_id": authority.principal_id, - "created_at": 10, - "expires_at": 20, - "completed_at": 12, - "signed_result": { - "schema": "elastos.browser.personal-sign-result/v1", - "request_id": "wallet-approval:browser", - "method": "personal_sign", - "signature": "0xsigned", - "signer": "0x1111111111111111111111111111111111111111" - } - })]), + approvals: TokioMutex::new(vec![completed_browser_personal_sign_approval( + context, + "wallet-approval:browser", + page_url, + origin, + )]), defaults: TokioMutex::default(), }; let app = gateway_router(wallet_test_state_with_provider(dir.path(), provider).await); + let query = url::form_urlencoded::Serializer::new(String::new()) + .append_pair("page_url", page_url) + .append_pair("origin", origin) + .finish(); let response = app + .clone() .oneshot( test_browser_request("localhost:61180", "null") - .uri("/api/apps/browser/wallet/approvals/wallet-approval%3Abrowser") - .header("x-elastos-home-token", browser_token) + .uri(format!( + "/api/apps/browser/wallet/approvals/wallet-approval%3Abrowser?{query}" + )) + .header("x-elastos-home-token", browser_token.clone()) .body(Body::empty()) .unwrap(), ) @@ -4435,6 +4473,37 @@ async fn test_browser_wallet_approval_status_returns_completed_signature() { ); assert_eq!(payload["status"], "completed"); assert_eq!(payload["signature"], "0xsigned"); + + let foreign_query = url::form_urlencoded::Serializer::new(String::new()) + .append_pair("page_url", "https://attacker.example/sign") + .append_pair("origin", "https://attacker.example") + .finish(); + let foreign = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .uri(format!( + "/api/apps/browser/wallet/approvals/wallet-approval%3Abrowser?{foreign_query}" + )) + .header("x-elastos-home-token", browser_token.clone()) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(foreign.status(), StatusCode::NOT_FOUND); + + let missing_context = app + .oneshot( + test_browser_request("localhost:61180", "null") + .uri("/api/apps/browser/wallet/approvals/wallet-approval%3Abrowser") + .header("x-elastos-home-token", browser_token) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(missing_context.status(), StatusCode::BAD_REQUEST); } fn completed_browser_transaction_approval( @@ -5538,6 +5607,85 @@ async fn test_browser_page_runtime_routes_are_runtime_scoped() { assert_eq!(input_after_close.status(), StatusCode::NOT_FOUND); } +#[tokio::test] +async fn test_browser_ordinary_close_and_replay_keep_failed_profile_durability() { + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let app = gateway_router(browser_engine_attached_test_state(dir.path()).await); + let opened = + open_mock_browser_page_result(app.clone(), &token, "inject failed profile durability") + .await; + let page_id = opened["engine_page"]["page_id"] + .as_str() + .unwrap() + .to_string(); + let cleanup_id = browser_cleanup_id(&opened).to_string(); + let encoded_page_id = page_id.replace(':', "%3A"); + assert_eq!(page_id, "page:mock-failed-profile-durability"); + + let close = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri(format!("/api/apps/browser/pages/{encoded_page_id}/close")) + .header("x-elastos-home-token", token.clone()) + .header(CONTENT_TYPE, "application/json") + .body(browser_close_body(&cleanup_id)) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(close.status(), StatusCode::OK); + let body = axum::body::to_bytes(close.into_body(), usize::MAX) + .await + .unwrap(); + let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); + assert_eq!(payload["schema"], "elastos.browser.close-result/v1"); + assert_eq!(payload["page_id"], page_id); + assert_eq!(payload["closed"], true); + assert_eq!(payload["terminal_effects"]["child_absent"], true); + assert_eq!(payload["profile_durability"], "failed"); + + let replay = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri(format!("/api/apps/browser/pages/{encoded_page_id}/close")) + .header("x-elastos-home-token", token.clone()) + .header(CONTENT_TYPE, "application/json") + .body(browser_close_body(&cleanup_id)) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(replay.status(), StatusCode::OK); + let replay_body = axum::body::to_bytes(replay.into_body(), usize::MAX) + .await + .unwrap(); + let replay: serde_json::Value = serde_json::from_slice(&replay_body).unwrap(); + assert_eq!(replay["schema"], "elastos.browser.close-result/v1"); + assert_eq!(replay["already_closed"], true); + assert_eq!(replay["closed"], false); + assert_eq!(replay["profile_durability"], "failed"); + + let input_after_close = app + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri(format!("/api/apps/browser/pages/{encoded_page_id}/input")) + .header("x-elastos-home-token", token) + .header(CONTENT_TYPE, "application/json") + .body(Body::from(r#"{"event":{"type":"click","x":12,"y":34}}"#)) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(input_after_close.status(), StatusCode::NOT_FOUND); +} + #[tokio::test] async fn test_browser_page_routes_require_page_owner() { let dir = tempfile::tempdir().unwrap(); @@ -6222,6 +6370,97 @@ async fn test_browser_close_typed_already_absent_is_terminal_without_retry_oblig assert_eq!(close_calls.lock().await.len(), 1); } +#[tokio::test] +async fn test_browser_close_after_engine_restart_fail_closed_is_runtime_already_absent() { + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let close_calls = Arc::new(TokioMutex::new(Vec::new())); + let app = gateway_router( + browser_engine_retrying_close_test_state( + dir.path(), + close_calls.clone(), + MockBrowserEngineCloseFailure::RestartFailClosed, + 4, + None, + None, + ) + .await, + ); + let opened = + open_mock_browser_page_result(app.clone(), &token, "restart fail-closed cleanup").await; + let page_id = opened["engine_page"]["page_id"].as_str().unwrap(); + let cleanup_id = browser_cleanup_id(&opened); + let close = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri(format!("/api/apps/browser/pages/{page_id}/close")) + .header("x-elastos-home-token", token) + .header(CONTENT_TYPE, "application/json") + .body(browser_close_body(cleanup_id)) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(close.status(), StatusCode::OK); + let close_body = axum::body::to_bytes(close.into_body(), usize::MAX) + .await + .unwrap(); + let close: serde_json::Value = serde_json::from_slice(&close_body).unwrap(); + assert_eq!(close["already_closed"], true); + assert_eq!(close["cleanup"]["action"], "already_absent"); + assert_eq!(browser_engine_cleanup_obligation_count(dir.path()).await, 0); + assert_eq!(close_calls.lock().await.len(), 1); +} + +#[tokio::test] +async fn test_browser_close_after_engine_retained_owner_unavailable_is_runtime_already_absent() { + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let close_calls = Arc::new(TokioMutex::new(Vec::new())); + let app = gateway_router( + browser_engine_retrying_close_test_state( + dir.path(), + close_calls.clone(), + MockBrowserEngineCloseFailure::RetainedOwnerUnavailable, + 4, + None, + None, + ) + .await, + ); + let opened = + open_mock_browser_page_result(app.clone(), &token, "retained owner unavailable cleanup") + .await; + let page_id = opened["engine_page"]["page_id"].as_str().unwrap(); + let cleanup_id = browser_cleanup_id(&opened); + let close = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri(format!("/api/apps/browser/pages/{page_id}/close")) + .header("x-elastos-home-token", token) + .header(CONTENT_TYPE, "application/json") + .body(browser_close_body(cleanup_id)) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(close.status(), StatusCode::OK); + let close_body = axum::body::to_bytes(close.into_body(), usize::MAX) + .await + .unwrap(); + let close: serde_json::Value = serde_json::from_slice(&close_body).unwrap(); + assert_eq!(close["already_closed"], true); + assert_eq!(close["cleanup"]["action"], "already_absent"); + assert_eq!(browser_engine_cleanup_obligation_count(dir.path()).await, 0); + assert_eq!(close_calls.lock().await.len(), 1); +} + #[tokio::test] async fn test_browser_close_lost_response_retries_exact_terminal_receipt_after_restart() { let dir = tempfile::tempdir().unwrap(); @@ -7601,6 +7840,158 @@ async fn test_browser_account_access_creates_review_only_inbox_handoff_without_d assert!(String::from_utf8_lossy(&body).contains("Review Browser account access in Wallet")); } +#[tokio::test] +async fn test_browser_account_access_creates_review_for_connector_account() { + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let browser_token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let address = "0x3333333333333333333333333333333333333333"; + let account_id = format!("wallet:eip155:20:{address}"); + let provider = MockWalletProvider { + challenges: TokioMutex::default(), + bitcoin_challenges: TokioMutex::default(), + accounts: TokioMutex::new(vec![json!({ + "account_id": account_id, + "principal_id": authority.principal_id, + "proof_binding_id": "proof:wallet:siwe:eip155:20:0x3333333333333333333333333333333333333333", + "chain_namespace": "eip155:20", + "address": address, + "proof_type": "siwe", + "connector_id": "wallet-metamask", + "label": "Family", + "linked_at": crate::auth::now_ts() + })]), + approvals: TokioMutex::default(), + defaults: TokioMutex::default(), + }; + let app = gateway_router(wallet_test_state_with_provider(dir.path(), provider).await); + let response = app + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri("/api/apps/browser/wallet/request-accounts") + .header("x-elastos-home-token", browser_token) + .header(CONTENT_TYPE, "application/json") + .body(Body::from( + json!({ + "chain_namespace": "eip155:20", + "page_url": "https://dapp.example/connect", + "origin": "https://dapp.example", + }) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let created: serde_json::Value = serde_json::from_slice(&body).unwrap(); + assert_eq!( + created["schema"], + "elastos.browser.account-access-request-result/v1" + ); + assert_eq!(created["requires_approval"], true); + assert_eq!(created["approval_request"]["status"], "pending"); + assert!( + created.get("address").is_none(), + "unapproved account-access response disclosed an address" + ); +} + +#[tokio::test] +async fn test_browser_account_access_uses_selected_connector_when_managed_account_is_also_present() +{ + let dir = tempfile::tempdir().unwrap(); + let authority = passkey_authority(dir.path()); + let browser_token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &authority); + let wallet_token = app_token_for_authority(dir.path(), WALLET_CAPSULE_ID, &authority); + let managed_address = "0x1111111111111111111111111111111111111111"; + let connector_address = "0x3333333333333333333333333333333333333333"; + let managed_account_id = format!("wallet:eip155:20:{managed_address}"); + let connector_account_id = format!("wallet:eip155:20:{connector_address}"); + let provider = MockWalletProvider { + challenges: TokioMutex::default(), + bitcoin_challenges: TokioMutex::default(), + accounts: TokioMutex::new(vec![ + json!({ + "account_id": managed_account_id, + "principal_id": authority.principal_id, + "proof_binding_id": "proof:wallet:managed:browser-consent", + "chain_namespace": "eip155:20", + "address": managed_address, + "proof_type": "managed_evm", + "signing_available": true, + "signing_status": "managed_key_available", + "linked_at": crate::auth::now_ts(), + }), + json!({ + "account_id": connector_account_id, + "principal_id": authority.principal_id, + "proof_binding_id": "proof:wallet:siwe:eip155:20:0x3333333333333333333333333333333333333333", + "chain_namespace": "eip155:20", + "address": connector_address, + "proof_type": "siwe", + "connector_id": "wallet-metamask", + "label": "Family", + "linked_at": crate::auth::now_ts() + }), + ]), + approvals: TokioMutex::default(), + defaults: TokioMutex::new(vec![json!({ + "principal_id": authority.principal_id, + "chain_namespace": "eip155:20", + "intent": "browser_connect", + "account_id": connector_account_id, + "set_at": crate::auth::now_ts() + })]), + }; + let app = gateway_router(wallet_test_state_with_provider(dir.path(), provider).await); + let response = app + .clone() + .oneshot( + test_browser_request("localhost:61180", "null") + .method("POST") + .uri("/api/apps/browser/wallet/request-accounts") + .header("x-elastos-home-token", browser_token) + .header(CONTENT_TYPE, "application/json") + .body(Body::from( + json!({ + "chain_namespace": "eip155:20", + "page_url": "https://dapp.example/connect", + "origin": "https://dapp.example", + }) + .to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + let summary = app + .oneshot( + test_browser_request("localhost:61180", "null") + .uri("/api/apps/wallet/wallet/summary") + .header("x-elastos-home-token", wallet_token) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(summary.status(), StatusCode::OK); + let body = axum::body::to_bytes(summary.into_body(), usize::MAX) + .await + .unwrap(); + let summary: serde_json::Value = serde_json::from_slice(&body).unwrap(); + let review = &summary["wallet_approvals"]["approval_requests"][0]["review"]; + assert_eq!(review["kind"], "account_access"); + assert_eq!(review["address"], connector_address); + assert_ne!(review["address"], managed_address); +} + #[tokio::test] async fn test_browser_account_access_status_returns_only_exact_origin_bound_grant() { let dir = tempfile::tempdir().unwrap(); @@ -8171,6 +8562,100 @@ async fn display_attach_route_concurrent_request_is_bounded_and_close_wins_late_ assert_eq!(browser_engine_cleanup_obligation_count(dir.path()).await, 0); } +#[tokio::test] +async fn display_attach_route_same_id_joins_one_provider_dispatch() { + let dir = tempfile::tempdir().unwrap(); + let owner = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &owner); + let (release, gate) = tokio::sync::oneshot::channel(); + let (app, provider) = display_attach_test_app(dir.path(), false, Some(gate)).await; + let opened = open_mock_browser_page_result(app.clone(), &token, "same id attach join").await; + let page_id = opened["engine_page"]["page_id"] + .as_str() + .unwrap() + .to_string(); + let first = tokio::spawn({ + let app = app.clone(); + let token = token.clone(); + let page_id = page_id.clone(); + async move { display_attach_post(app, &token, &page_id, display_attach_test_body()).await } + }); + tokio::time::timeout(Duration::from_secs(2), provider.entered.notified()) + .await + .unwrap(); + let second = tokio::spawn({ + let app = app.clone(); + let token = token.clone(); + let page_id = page_id.clone(); + async move { display_attach_post(app, &token, &page_id, display_attach_test_body()).await } + }); + tokio::time::sleep(Duration::from_millis(50)).await; + assert_eq!(provider.signals.lock().await.len(), 1); + release.send(()).unwrap(); + let first = first.await.unwrap(); + let second = second.await.unwrap(); + assert_eq!(first.0, StatusCode::OK); + assert_eq!(second, first); + assert_eq!(provider.signals.lock().await.len(), 1); +} + +#[tokio::test] +async fn display_attach_route_survives_leader_disconnect() { + let dir = tempfile::tempdir().unwrap(); + let owner = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &owner); + let (release, gate) = tokio::sync::oneshot::channel(); + let (app, provider) = display_attach_test_app(dir.path(), false, Some(gate)).await; + let opened = + open_mock_browser_page_result(app.clone(), &token, "leader disconnect attach").await; + let page_id = opened["engine_page"]["page_id"] + .as_str() + .unwrap() + .to_string(); + let first = tokio::spawn({ + let app = app.clone(); + let token = token.clone(); + let page_id = page_id.clone(); + async move { display_attach_post(app, &token, &page_id, display_attach_test_body()).await } + }); + tokio::time::timeout(Duration::from_secs(2), provider.entered.notified()) + .await + .unwrap(); + first.abort(); + tokio::time::sleep(Duration::from_millis(20)).await; + let second = tokio::spawn({ + let app = app.clone(); + let token = token.clone(); + let page_id = page_id.clone(); + async move { display_attach_post(app, &token, &page_id, display_attach_test_body()).await } + }); + release.send(()).unwrap(); + let second = second.await.unwrap(); + assert_eq!(second.0, StatusCode::OK); + assert_eq!(second.1["display_generation"], DISPLAY_ATTACH_TEST_NEW); + assert_eq!(provider.signals.lock().await.len(), 1); +} + +#[tokio::test] +async fn display_attach_route_late_same_id_uses_cached_outcome() { + let dir = tempfile::tempdir().unwrap(); + let owner = passkey_authority(dir.path()); + let token = app_token_for_authority(dir.path(), BROWSER_CAPSULE_ID, &owner); + let (app, provider) = display_attach_test_app(dir.path(), false, None).await; + let opened = open_mock_browser_page_result(app.clone(), &token, "late same id cache").await; + let page_id = opened["engine_page"]["page_id"] + .as_str() + .unwrap() + .to_string(); + let first = + display_attach_post(app.clone(), &token, &page_id, display_attach_test_body()).await; + assert_eq!(first.0, StatusCode::OK); + let second = + display_attach_post(app.clone(), &token, &page_id, display_attach_test_body()).await; + assert_eq!(second, first); + assert_eq!(provider.signals.lock().await.len(), 1); +} + #[derive(Default)] struct OperatorTestProvider { inspection: InspectionTestProvider, diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_sessions.rs b/elastos/crates/elastos-server/src/api/gateway_browser_sessions.rs index 88544f246..8738e099c 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_sessions.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_sessions.rs @@ -416,11 +416,14 @@ struct BrowserReapedPageTombstone { browser_instance: Option, terminal_kind: String, reaped_at_unix_ms: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + profile_durability: Option, } #[derive(Debug, Clone, PartialEq, Eq)] pub(in crate::api::gateway) struct BrowserReapedPageTerminalReceipt { pub(in crate::api::gateway) browser_instance: Option, + pub(in crate::api::gateway) profile_durability: Option, } #[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] @@ -2398,6 +2401,7 @@ pub(in crate::api::gateway) async fn record_browser_reaped_page_tombstone( data_dir: &Path, cleanup: &BrowserEngineCleanup, terminal_owner_launch_id: Option<&str>, + profile_durability: Option<&str>, ) -> Result<(), String> { let now_unix_ms = browser_now_unix_ms()?; let mut owner_launch_ids = vec![cleanup.owner_launch_id.clone()]; @@ -2419,6 +2423,9 @@ pub(in crate::api::gateway) async fn record_browser_reaped_page_tombstone( browser_instance: cleanup.browser_instance.clone(), terminal_kind: "already_absent".to_string(), reaped_at_unix_ms: now_unix_ms, + profile_durability: profile_durability + .filter(|value| matches!(*value, "failed" | "unknown" | "proved")) + .map(str::to_string), }, now_unix_ms, ) @@ -2882,6 +2889,10 @@ fn browser_reaped_page_tombstone_is_safe( .is_none_or(|instance| browser_instance_id(Some(instance.to_string())).is_ok()) && tombstone.reaped_at_unix_ms > 0 && tombstone.reaped_at_unix_ms <= now_unix_ms.saturating_add(MAX_CLOCK_SKEW_MS) + && tombstone + .profile_durability + .as_deref() + .is_none_or(|value| matches!(value, "failed" | "unknown" | "proved")) } fn browser_reaped_page_tombstone_is_expired( @@ -2955,11 +2966,16 @@ fn persist_browser_reaped_page_tombstone( "Browser terminal close receipt owner aliases exceeded their bound".to_string(), ); } - if owner_launch_ids == previous.owner_launch_ids { + if owner_launch_ids == previous.owner_launch_ids + && previous.profile_durability == tombstone.profile_durability + { return Ok(()); } let mut expanded = previous.clone(); expanded.owner_launch_ids = owner_launch_ids; + if expanded.profile_durability.is_none() { + expanded.profile_durability = tombstone.profile_durability.clone(); + } return write_browser_json_atomic( data_dir, &browser_reaped_page_tombstone_path(data_dir, &expanded.cleanup_id), @@ -3016,6 +3032,7 @@ fn browser_reaped_page_tombstone_matches_at( { return Ok(Some(BrowserReapedPageTerminalReceipt { browser_instance: tombstone.browser_instance, + profile_durability: tombstone.profile_durability, })); } if browser_instance.is_none() @@ -3031,6 +3048,7 @@ fn browser_reaped_page_tombstone_matches_at( persist_browser_reaped_page_tombstone(data_dir, expanded.clone(), now_unix_ms)?; Ok(Some(BrowserReapedPageTerminalReceipt { browser_instance: expanded.browser_instance, + profile_durability: expanded.profile_durability, })) } @@ -5094,6 +5112,7 @@ mod tests { browser_instance: Some("browser:0123456789abcdef0123456789abcdef".to_string()), terminal_kind: "already_absent".to_string(), reaped_at_unix_ms: now_unix_ms, + profile_durability: None, }, now_unix_ms, ) diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_wallet.rs b/elastos/crates/elastos-server/src/api/gateway_browser_wallet.rs index f7bb22889..471b56d0a 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_wallet.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_wallet.rs @@ -6,13 +6,13 @@ mod gateway_browser_wallet_bridge; #[path = "gateway_browser_wallet_reads.rs"] mod gateway_browser_wallet_reads; -use gateway_browser_wallet_bridge::browser_wallet_account_is_signable_evm; +use gateway_browser_wallet_bridge::{ + browser_account_access_account, browser_account_access_request_uses_supported_signer, + browser_wallet_account_is_signable_evm, BROWSER_SUPPORTED_EVM_CHAIN_NAMESPACES, +}; pub(in crate::api::gateway) use gateway_browser_wallet_bridge::{ browser_chain_namespace_network, browser_wallet_bridge_payload, is_browser_wallet_intent, }; -use gateway_browser_wallet_bridge::{ - browser_projected_evm_accounts, BROWSER_SUPPORTED_EVM_CHAIN_NAMESPACES, -}; use gateway_browser_wallet_reads::browser_wallet_read; fn browser_wallet_cors_origin(headers: &HeaderMap) -> HeaderValue { @@ -387,21 +387,15 @@ async fn create_browser_wallet_account_access_request( )); } let summary = system_wallet_accounts_summary(state, authority).await; - let account = browser_projected_evm_accounts(&summary) - .into_iter() - .find(|account| { - account.chain_namespace == input.chain_namespace - && browser_wallet_account_is_signable_evm(account) - && is_managed_wallet_proof_type(&account.proof_type) - && account.connector_id.is_none() - }) - .ok_or_else(|| { + let account = browser_account_access_account(&summary, &input.chain_namespace).ok_or_else( + || { ( StatusCode::FORBIDDEN, - "Browser account access requires a Runtime-managed EVM account for the selected chain" + "Browser account access requires a principal-owned EVM account for the selected chain" .to_string(), ) - })?; + }, + )?; let chain_namespaces = BROWSER_SUPPORTED_EVM_CHAIN_NAMESPACES .iter() .map(|namespace| (*namespace).to_string()) @@ -957,6 +951,62 @@ fn completed_browser_wallet_result_string<'a>( .ok_or_else(|| completed_browser_wallet_result_error(&format!("is missing {field}"))) } +fn bind_completed_browser_signature_caller( + request: &serde_json::Value, + authority: &RuntimeWalletAuthority, + page_url: Option<&str>, + origin: Option<&str>, +) -> Result<(), (StatusCode, String)> { + let context = authority.verified_context(); + let page_url = page_url.ok_or_else(|| { + ( + StatusCode::BAD_REQUEST, + "Browser signature status requires the exact page URL".to_string(), + ) + })?; + let origin = origin.ok_or_else(|| { + ( + StatusCode::BAD_REQUEST, + "Browser signature status requires the exact page origin".to_string(), + ) + })?; + let canonical_origin = validate_browser_wallet_page_origin(page_url, origin)?; + for (field, expected) in [ + ("principal_id", context.principal_id()), + ("session_id", context.session_id()), + ("launch_id", context.launch_id()), + ("requested_by_actor", BROWSER_CAPSULE_ID), + ] { + if request.get(field).and_then(|value| value.as_str()) != Some(expected) { + return Err(( + StatusCode::NOT_FOUND, + "browser wallet approval request not found".to_string(), + )); + } + } + let payload = request + .get("payload") + .filter(|value| value.is_object()) + .ok_or_else(|| { + ( + StatusCode::NOT_FOUND, + "browser wallet approval request not found".to_string(), + ) + })?; + for (field, expected) in [ + ("page_url", page_url), + ("origin", canonical_origin.as_str()), + ] { + if completed_browser_wallet_result_string(payload, field)? != expected { + return Err(( + StatusCode::NOT_FOUND, + "browser wallet approval request not found".to_string(), + )); + } + } + Ok(()) +} + fn browser_account_access_request_payload<'a>( request: &'a serde_json::Value, authority: &RuntimeWalletAuthority, @@ -977,17 +1027,9 @@ fn browser_account_access_request_payload<'a>( ) })?; let canonical_origin = validate_browser_wallet_page_origin(page_url, origin)?; - if !is_managed_wallet_proof_type( - request - .get("proof_type") - .and_then(|value| value.as_str()) - .unwrap_or_default(), - ) || request - .get("connector_id") - .is_some_and(|value| !value.is_null()) - { + if !browser_account_access_request_uses_supported_signer(request) { return Err(completed_browser_wallet_result_error( - "is not backed by a Runtime-managed signer", + "is not backed by a principal-owned signer", )); } for (field, expected) in [ @@ -1285,6 +1327,7 @@ async fn browser_wallet_approval_status( intent, Some("browser_personal_sign") | Some("browser_typed_data_sign") ) { + bind_completed_browser_signature_caller(request, authority, page_url, origin)?; let signature = result .get("signature") .and_then(|value| value.as_str()) diff --git a/elastos/crates/elastos-server/src/api/gateway_browser_wallet_bridge.rs b/elastos/crates/elastos-server/src/api/gateway_browser_wallet_bridge.rs index 24ea4dec1..b3b95eee7 100644 --- a/elastos/crates/elastos-server/src/api/gateway_browser_wallet_bridge.rs +++ b/elastos/crates/elastos-server/src/api/gateway_browser_wallet_bridge.rs @@ -25,13 +25,23 @@ pub(in crate::api::gateway) fn browser_chain_namespace_network( } } +fn browser_wallet_account_is_selectable_evm(account: &SystemWalletAccountSummary) -> bool { + if !account.chain_namespace.starts_with("eip155:") { + return false; + } + if is_managed_wallet_proof_type(&account.proof_type) { + return account.signing_available && !browser_wallet_account_has_connector(account); + } + browser_wallet_account_has_connector(account) +} + fn browser_default_evm_account( summary: &SystemWalletAccountsSummary, ) -> Option<&SystemWalletDefaultSummary> { let has_linked_evm_account = |default: &SystemWalletDefaultSummary| { summary.accounts.iter().any(|account| { account.account_id == default.account_id - && browser_wallet_account_is_signable_evm(account) + && browser_wallet_account_is_selectable_evm(account) }) }; ["browser_connect", "transaction_intent"] @@ -52,7 +62,7 @@ fn browser_default_account_id(summary: &SystemWalletAccountsSummary) -> Option Opt .accounts .iter() .find(|account| { - browser_wallet_account_is_signable_evm(account) + browser_wallet_account_is_selectable_evm(account) && browser_chain_namespace_network(&account.chain_namespace).is_some() }) .map(|account| account.chain_namespace.clone()) @@ -82,7 +92,7 @@ pub(super) fn browser_projected_evm_accounts( let mut accounts = summary .accounts .iter() - .filter(|account| browser_wallet_account_is_signable_evm(account)) + .filter(|account| browser_wallet_account_is_selectable_evm(account)) .cloned() .collect::>(); accounts.sort_by_key(|account| { @@ -112,6 +122,58 @@ pub(super) fn browser_wallet_account_is_signable_evm(account: &SystemWalletAccou account.chain_namespace.starts_with("eip155:") && account.signing_available } +fn browser_wallet_account_has_connector(account: &SystemWalletAccountSummary) -> bool { + account + .connector_id + .as_deref() + .is_some_and(|connector_id| !connector_id.is_empty()) +} + +pub(super) fn browser_account_access_account( + summary: &SystemWalletAccountsSummary, + chain_namespace: &str, +) -> Option { + if browser_chain_namespace_network(chain_namespace).is_none() { + return None; + } + let default_account_id = browser_default_account_id(summary); + let projected = browser_projected_evm_accounts(summary); + if let Some(default_id) = default_account_id.as_deref() { + if let Some(account) = projected.iter().find(|account| { + account.chain_namespace == chain_namespace && account.account_id == default_id + }) { + return Some(account.clone()); + } + if let Some(account) = summary.accounts.iter().find(|account| { + account.account_id == default_id && browser_wallet_account_is_selectable_evm(account) + }) { + let mut projected_account = account.clone(); + projected_account.chain_namespace = chain_namespace.to_string(); + return Some(projected_account); + } + } + projected + .into_iter() + .find(|account| account.chain_namespace == chain_namespace) +} + +pub(super) fn browser_account_access_request_uses_supported_signer( + request: &serde_json::Value, +) -> bool { + let proof_type = request + .get("proof_type") + .and_then(|value| value.as_str()) + .unwrap_or_default(); + let connector_id = request + .get("connector_id") + .and_then(|value| value.as_str()) + .filter(|value| !value.is_empty()); + if is_managed_wallet_proof_type(proof_type) { + return connector_id.is_none(); + } + connector_id.is_some() +} + pub(in crate::api::gateway) async fn browser_wallet_bridge_payload( state: &GatewayState, context: &HomeLaunchTokenContext, @@ -224,6 +286,20 @@ mod browser_wallet_bridge_tests { } } + fn connector_account(chain_namespace: &str, account_id: &str) -> SystemWalletAccountSummary { + SystemWalletAccountSummary { + account_id: account_id.to_string(), + chain_namespace: chain_namespace.to_string(), + address: "0x3333333333333333333333333333333333333333".to_string(), + proof_type: "siwe".to_string(), + signing_available: false, + signing_status: Some("external_connector_available".to_string()), + label: Some("Family".to_string()), + connector_id: Some("wallet-metamask".to_string()), + linked_at: 1, + } + } + fn default_account( chain_namespace: &str, account_id: &str, @@ -391,4 +467,81 @@ mod browser_wallet_bridge_tests { Some("eip155:20") ); } + + #[test] + fn browser_account_access_honors_selected_connector_when_managed_account_is_also_present() { + let managed = account("eip155:20", "wallet:eip155:20:0x111"); + let connector = connector_account("eip155:20", "wallet:eip155:20:0x333"); + let summary = SystemWalletAccountsSummary { + available: true, + linked_count: 2, + accounts: vec![managed.clone(), connector.clone()], + default_accounts: vec![default_account( + "eip155:20", + "wallet:eip155:20:0x333", + "browser_connect", + )], + note: None, + }; + + let selected = browser_account_access_account(&summary, "eip155:20") + .expect("selected connector account"); + assert_eq!(selected.account_id, connector.account_id); + assert_eq!(selected.connector_id.as_deref(), Some("wallet-metamask")); + assert!(!selected.signing_available); + assert_eq!( + browser_default_account_id(&summary).as_deref(), + Some("wallet:eip155:20:0x333") + ); + } + + #[test] + fn browser_account_access_keeps_managed_default_when_connector_is_not_selected() { + let managed = account("eip155:20", "wallet:eip155:20:0x111"); + let connector = connector_account("eip155:20", "wallet:eip155:20:0x333"); + let summary = SystemWalletAccountsSummary { + available: true, + linked_count: 2, + accounts: vec![managed.clone(), connector], + default_accounts: vec![default_account( + "eip155:20", + "wallet:eip155:20:0x111", + "browser_connect", + )], + note: None, + }; + + let selected = browser_account_access_account(&summary, "eip155:20") + .expect("selected managed account"); + assert_eq!(selected.account_id, managed.account_id); + assert!(selected.signing_available); + assert!(selected.connector_id.is_none()); + } + + #[test] + fn browser_account_access_skips_unsignable_managed_default_for_selected_connector() { + let summary = SystemWalletAccountsSummary { + available: true, + linked_count: 2, + accounts: vec![ + unavailable_account("eip155:20", "wallet:eip155:20:0x111"), + connector_account("eip155:20", "wallet:eip155:20:0x333"), + ], + default_accounts: vec![default_account( + "eip155:20", + "wallet:eip155:20:0x111", + "browser_connect", + )], + note: None, + }; + + let selected = browser_account_access_account(&summary, "eip155:20") + .expect("connector remains selectable"); + assert_eq!(selected.account_id, "wallet:eip155:20:0x333"); + assert_eq!(selected.connector_id.as_deref(), Some("wallet-metamask")); + assert_eq!( + browser_default_account_id(&summary).as_deref(), + Some("wallet:eip155:20:0x333") + ); + } } diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/browser_reconciliation.rs b/elastos/crates/elastos-server/src/api/gateway_tests/browser_reconciliation.rs index 73589412b..edd9a6137 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/browser_reconciliation.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/browser_reconciliation.rs @@ -839,3 +839,92 @@ async fn exact_cleanup_ownership_remains_until_typed_terminal_receipt() { .expect("replacement only after terminal receipt"); release_browser_launch(&replacement).await; } + +#[tokio::test(start_paused = true)] +async fn expired_remote_engine_grant_forgets_launch_reconciliation_without_engine_call() { + let dir = tempfile::tempdir().unwrap(); + let (state, _close_calls, reconciliation_calls) = browser_engine_reconciliation_test_state( + dir.path(), + MockDispatchedBrowserLaunchFailure::TransientThenLateSuccess, + ) + .await; + let principal_id = "person:local:expired-grant-reconciliation"; + let reservation = record_pending_launch( + &state, + principal_id, + "launch:expired-grant-reconciliation", + "stream:expired-grant-reconciliation", + ) + .await; + gateway_browser_remote::write_consumer_binding_for_test( + &state.data_dir, + &reservation, + principal_id, + "stream:expired-grant-reconciliation", + 1, + ) + .expect("expired remote Engine grant record"); + let reconciler = + start_controlled_browser_lifecycle_reconciler(state.clone()).expect("Runtime reconciler"); + reconciler.wait_for_completed_sweeps(1).await; + assert_eq!( + browser_launch_reconciliation_obligation_count(&state.data_dir).await, + 0 + ); + assert!( + reconciliation_calls.snapshot().await.is_empty(), + "an expired remote Engine grant retires the leftover without an Engine status call" + ); + let replacement = reserve_browser_launch( + &state.data_dir, + principal_id, + browser_lifecycle("launch:replacement-after-expired-grant"), + ) + .await + .expect("replacement after expired remote Engine grant retirement"); + release_browser_launch(&replacement).await; + reconciler.cancel(); + reconciler.join().await.expect("Runtime shutdown"); +} + +#[tokio::test(start_paused = true)] +async fn current_remote_engine_grant_still_asks_engine_for_launch_reconciliation() { + let dir = tempfile::tempdir().unwrap(); + let (state, close_calls, reconciliation_calls) = browser_engine_reconciliation_test_state( + dir.path(), + MockDispatchedBrowserLaunchFailure::TransientThenLateSuccess, + ) + .await; + let principal_id = "person:local:current-grant-reconciliation"; + let reservation = record_pending_launch( + &state, + principal_id, + "launch:current-grant-reconciliation", + "stream:current-grant-reconciliation", + ) + .await; + gateway_browser_remote::write_consumer_binding_for_test( + &state.data_dir, + &reservation, + principal_id, + "stream:current-grant-reconciliation", + crate::auth::now_ts().saturating_add(3600), + ) + .expect("current remote Engine grant record"); + let reconciler = start_browser_lifecycle_reconciler(state.clone()).expect("Runtime reconciler"); + reconciliation_calls.wait_for_count(1).await; + assert_eq!( + browser_launch_reconciliation_obligation_count(&state.data_dir).await, + 1 + ); + finish_current_reconciliation_sweep().await; + advance_until_reconciliation_call_count(&reconciliation_calls, 2, Duration::from_millis(100)) + .await; + close_calls.wait_for_count(1).await; + assert_eq!( + browser_launch_reconciliation_obligation_count(&state.data_dir).await, + 0 + ); + reconciler.cancel(); + reconciler.join().await.expect("Runtime shutdown"); +} diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/remote_engine.rs b/elastos/crates/elastos-server/src/api/gateway_tests/remote_engine.rs index 148597dce..ee1f95603 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/remote_engine.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/remote_engine.rs @@ -282,7 +282,54 @@ async fn remote_engine_preparation_replay_close_and_late_input_keep_exact_native .unwrap() .starts_with(root.path().to_str().unwrap())); assert_eq!(native["profile"]["profile_key"], prepare["profile_key"]); - assert_eq!(native["wallet"], json!({})); + assert_eq!( + native["wallet"]["schema"], + "elastos.browser.wallet-consumer-mediation/v1" + ); + assert_eq!(native["wallet"]["resolution"], "consumer_runtime"); + assert_eq!(native["wallet"]["page_id"], page_id); + assert_eq!(native["wallet"]["lifecycle_generation"], generation); + assert_eq!( + native["wallet"]["requester_endpoint"], + owner.requester_endpoint + ); + assert!(native["wallet"].get("principal_id").is_none()); + assert_ne!(owner.storage_principal(), grant.requester_principal_id); + assert!(native["wallet"].get("home_token").is_none()); + assert!(native["wallet"].get("account_access_url").is_none()); + let engine_peer = endpoint.id(); + let consumer = gateway_browser_remote::consumer_binding_for_wallet_forward_test( + &grant.requester_principal_id, + page_id.as_str(), + generation.as_str(), + &engine_peer, + ); + let forwarded = gateway_browser_remote::forward_remote_wallet_consumer_request( + &consumer, + &native["wallet"], + &engine_peer, + "request_accounts", + "https://ela.city/", + "https://ela.city", + json!({"method": "eth_requestAccounts"}), + ) + .expect("Wallet request built from the real launch payload"); + assert_eq!(forwarded["page_id"], page_id); + assert_eq!(forwarded["lifecycle_generation"], generation); + assert!(forwarded.get("principal_id").is_none()); + let foreign_peer = iroh::SecretKey::from_bytes(&[23; 32]).public(); + assert!( + gateway_browser_remote::forward_remote_wallet_consumer_request( + &consumer, + &native["wallet"], + &foreign_peer, + "request_accounts", + "https://ela.city/", + "https://ela.city", + json!({"method": "eth_requestAccounts"}), + ) + .is_err() + ); assert_eq!(native["transport_authority"]["generation"], generation); assert_eq!(requests.iter().filter(|r| r["op"] == "launch").count(), 1); let mut stale = base.clone(); @@ -1020,3 +1067,122 @@ pub(super) async fn exercise_revoked_remote_preparation_cleanup( )) .await; } + +#[tokio::test] +async fn remote_engine_status_after_restart_uses_settlement_authority() { + let root = tempfile::tempdir().unwrap(); + let provider = Arc::new(RemoteEngineFixture::default()); + let registry = Arc::new(ProviderRegistry::new()); + registry + .register_sub_provider("browser-engine", provider.clone()) + .await + .unwrap(); + let endpoint = iroh::Endpoint::builder(iroh::endpoint::presets::Minimal) + .bind() + .await + .unwrap(); + let (trusted, _) = generate_keypair(); + let network = configured_discovery_network_profile_for_test(&trusted, "remote-engine-restart"); + let source = iroh::SecretKey::from_bytes(&[46; 32]).public(); + let principal = "person:local:5021af2e018b5ebcbf577e0cc1554a8c"; + let generation = format!("sha256:{}", "f".repeat(64)); + let page_id = format!( + "page:vz-{}", + hex::encode(Sha256::digest(format!("{generation}\npage"))) + ); + let stream_id = "stream:source-home-browser-exit:465196cc18314cb9"; + let storage = RemoteEngineOwner::storage_principal_for(&source.to_string(), principal); + let authority = json!({ + "principal_id": storage, + "generation": generation, + "page_id": page_id, + "egress": {"stream_id": stream_id}, + }); + let request = json!({ + "op": "status", + "principal_id": principal, + "grant_id": "services-remote-engine-grant-c64af1d3ac031112", + "page_id": page_id, + "lifecycle_generation": generation, + "stream_id": stream_id, + "adapter_id": "browser-vm-product", + "transport_authority": authority, + }); + let call = |source: iroh::PublicKey, request: Value| { + let root = root.path().to_owned(); + let registry = registry.clone(); + let endpoint = endpoint.clone(); + let network = network.clone(); + async move { + gateway_browser_remote::invoke( + &root, registry, endpoint, source, network, None, &request, + ) + .await + } + }; + let settled = call(source, request.clone()).await.unwrap(); + assert_eq!(settled["status"], "ok", "{settled}"); + assert_eq!( + settled["data"]["schema"], + "elastos.browser.engine.launch-reconciliation/v1" + ); + assert_eq!(settled["data"]["lifecycle_generation"], generation); + assert_eq!(settled["data"]["stream_id"], stream_id); + let forwarded = provider.calls.lock().await; + assert_eq!(forwarded.len(), 1); + assert_eq!(forwarded[0]["op"], "status"); + assert_eq!(forwarded[0]["principal_id"], storage); + assert!(forwarded[0].get("page_id").is_none()); + drop(forwarded); + + let foreign = iroh::SecretKey::from_bytes(&[47; 32]).public(); + assert!( + call(foreign, request.clone()) + .await + .unwrap_err() + .to_string() + .contains("Engine requester or generation changed"), + "wrong peer must stay rejected" + ); + + let mut wrong_principal = request.clone(); + wrong_principal["principal_id"] = json!("person:local:other"); + assert!( + call(source, wrong_principal) + .await + .unwrap_err() + .to_string() + .contains("Engine requester or generation changed"), + "wrong principal must stay rejected" + ); + + let mut wrong_generation = request.clone(); + let other = format!("sha256:{}", "a".repeat(64)); + wrong_generation["lifecycle_generation"] = json!(other); + wrong_generation["page_id"] = json!(format!( + "page:vz-{}", + hex::encode(Sha256::digest(format!("{other}\npage"))) + )); + assert!( + call(source, wrong_generation) + .await + .unwrap_err() + .to_string() + .contains("Engine requester or generation changed"), + "wrong generation must stay rejected" + ); + + let mut missing = request.clone(); + missing + .as_object_mut() + .unwrap() + .remove("transport_authority"); + assert!( + call(source, missing) + .await + .unwrap_err() + .to_string() + .contains("Engine retained owner unavailable"), + "status without settlement authority stays unavailable" + ); +} diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs b/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs index 6ede679bb..e941218c5 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/support_providers.rs @@ -3228,6 +3228,8 @@ enum MockBrowserEngineCloseFailure { Transport, Adapter, AlreadyClosed, + RestartFailClosed, + RetainedOwnerUnavailable, } struct MockRetryingBrowserEngineProvider { @@ -3285,6 +3287,9 @@ fn mock_browser_launch_page_id(request: &serde_json::Value) -> String { if reason.contains("simulate close failure") { return "page:mock-browser-close-fails".to_string(); } + if reason.contains("inject failed profile durability") { + return "page:mock-failed-profile-durability".to_string(); + } let digest = sha2::Sha256::digest(format!("{url}:{reason}").as_bytes()); format!("page:mock-browser-engine-{}", hex::encode(&digest[..4])) } @@ -3302,22 +3307,30 @@ fn mock_browser_terminal_cleanup_response(request: &serde_json::Value) -> serde_ .get("runtime_cleanup") .cloned() .unwrap_or(serde_json::Value::Null); + let page_id = binding + .get("page_id") + .and_then(serde_json::Value::as_str) + .unwrap_or(""); + let mut data = json!({ + "schema": "elastos.browser.engine-cleanup-result/v2", + "page_id": binding.get("page_id").cloned().unwrap_or(serde_json::Value::Null), + "generation": binding.get("generation").cloned().unwrap_or(serde_json::Value::Null), + "binding": binding, + "terminal": true, + "effects": { + "page_absent": true, + "child_absent": true, + "vm_absent": true, + "route_absent": true, + "socket_absent": true + } + }); + if page_id.contains("failed-profile-durability") { + data["profile_durability"] = json!("failed"); + } json!({ "status": "ok", - "data": { - "schema": "elastos.browser.engine-cleanup-result/v2", - "page_id": binding.get("page_id").cloned().unwrap_or(serde_json::Value::Null), - "generation": binding.get("generation").cloned().unwrap_or(serde_json::Value::Null), - "binding": binding, - "terminal": true, - "effects": { - "page_absent": true, - "child_absent": true, - "vm_absent": true, - "route_absent": true, - "socket_absent": true - } - } + "data": data }) } @@ -4073,7 +4086,7 @@ impl Provider for MockReconciliatingBrowserEngineProvider { "transport_authority": authority, "effects": { "page_acquired": false, - "vm_acquired": true, + "vm_acquired": false, }, "terminal_cleanup_receipt": settlement, } @@ -4307,6 +4320,16 @@ impl Provider for MockRetryingBrowserEngineProvider { MockBrowserEngineCloseFailure::AlreadyClosed => { Ok(mock_browser_terminal_cleanup_response(request)) } + MockBrowserEngineCloseFailure::RestartFailClosed => Ok(json!({ + "status": "error", + "code": "engine_close_indeterminate", + "message": "Browser VM cleanup remains indeterminate after service restart: exact owned launcher unavailable" + })), + MockBrowserEngineCloseFailure::RetainedOwnerUnavailable => Ok(json!({ + "status": "error", + "code": "engine_close_indeterminate", + "message": "Engine retained owner unavailable" + })), }; } let response = ::send_raw( diff --git a/elastos/crates/elastos-server/src/carrier_engine_binding.rs b/elastos/crates/elastos-server/src/carrier_engine_binding.rs index 2ce5d1245..2c23a934e 100644 --- a/elastos/crates/elastos-server/src/carrier_engine_binding.rs +++ b/elastos/crates/elastos-server/src/carrier_engine_binding.rs @@ -129,12 +129,13 @@ impl RemoteEngineOwner { /// Provider storage is scoped to both identities. A peer cannot name a host /// directory or borrow another peer's profile by supplying its principal. pub(crate) fn storage_principal(&self) -> String { + Self::storage_principal_for(&self.requester_endpoint, &self.requester_principal_id) + } + + pub(crate) fn storage_principal_for(endpoint: &str, principal: &str) -> String { format!( "remote-engine-{}", - hex::encode(Sha256::digest(format!( - "{}\n{}", - self.requester_endpoint, self.requester_principal_id - ))) + hex::encode(Sha256::digest(format!("{endpoint}\n{principal}"))) ) } } diff --git a/elastos/crates/elastos-server/src/carrier_engine_media.rs b/elastos/crates/elastos-server/src/carrier_engine_media.rs index 88c4f531f..eacd355c0 100644 --- a/elastos/crates/elastos-server/src/carrier_engine_media.rs +++ b/elastos/crates/elastos-server/src/carrier_engine_media.rs @@ -14,7 +14,7 @@ use std::{ }; use tokio::{ io::AsyncWriteExt, - sync::{watch, Mutex, Semaphore}, + sync::{oneshot, watch, Mutex, Semaphore}, }; struct MediaTarget { @@ -146,6 +146,7 @@ pub(crate) async fn serve( tokio::net::TcpStream::connect(address), ) .await??; + disable_media_nagle(&target)?; ensure!( !retiring.load(Ordering::Acquire), "Engine media retired during connect" @@ -165,11 +166,87 @@ pub(crate) async fn serve( Ok(()) } -async fn connect( +enum WarmPayload { + Ready(super::BrowserCarrierStream), + Pending { + client: super::CarrierClient, + send: iroh::endpoint::SendStream, + recv: iroh::endpoint::RecvStream, + }, +} + +const STREAM_WARM_DEADLINE: Duration = Duration::from_secs(5); + +struct MediaWarm { + generation: String, + first: Option>>, + worker: Option, + attach_armed: bool, +} + +struct TakenWarm { + rx: oneshot::Receiver>, + worker: Option, +} + +fn abort_warm_worker(worker: Option) { + if let Some(worker) = worker { + worker.abort(); + } +} + +struct AbortOnDrop(Option); + +impl Drop for AbortOnDrop { + fn drop(&mut self) { + abort_warm_worker(self.0.take()); + } +} + +async fn warm_payload_with_deadline( + prepared_abort: Option, + deadline: Duration, + work: F, +) -> Result +where + F: std::future::Future>, +{ + match tokio::time::timeout(deadline, work).await { + Ok(result) => result, + Err(_) => { + abort_warm_worker(prepared_abort); + tracing::info!("browser_media_warm warm_deadline"); + Err("Engine media warm deadline".into()) + } + } +} + +static MEDIA_WARMS: OnceLock>> = OnceLock::new(); + +fn clone_client_handle(client: &super::CarrierClient) -> super::CarrierClient { + super::CarrierClient { + conn: client.conn.clone(), + _endpoint: client._endpoint.clone(), + owns_endpoint: false, + } +} + +fn media_admit_request(authority: &Value) -> Value { + json!({"op":"browser_engine_media","page_id":authority["page_id"], + "generation":authority["generation"],"binding_hash":authority["binding_hash"]}) +} + +pub(crate) async fn preconnect_engine( endpoint: &iroh::Endpoint, grant: &Value, - authority: &Value, -) -> Result { +) -> Result { + warm_media_client(endpoint, grant).await +} + +async fn warm_media_client( + endpoint: &iroh::Endpoint, + grant: &Value, +) -> Result { let peer = grant["peer_did"] .as_str() .context("Engine peer missing")? @@ -179,47 +256,399 @@ async fn connect( .as_str() .context("Engine route missing")?, ); - tokio::time::timeout(Duration::from_secs(5), async { - for address in addresses.into_iter().filter(|a| a.id == peer) { - let Ok(client) = - super::CarrierClient::connect_known_endpoint(endpoint, address, 5).await - else { - continue; + for address in addresses.into_iter().filter(|a| a.id == peer) { + if let Ok(client) = super::CarrierClient::connect_known_endpoint(endpoint, address, 5).await + { + return Ok(client); + } + } + anyhow::bail!("Engine media route unavailable") +} + +async fn admit_opened_stream( + mut send: iroh::endpoint::SendStream, + mut recv: iroh::endpoint::RecvStream, + client: super::CarrierClient, + authority: &Value, +) -> Result { + super::write_json_line(&mut send, &media_admit_request(authority)).await?; + let mut response = Vec::new(); + loop { + let mut b = [0; 1]; + recv.read_exact(&mut b).await?; + ensure!( + response.len() < 1024, + "Engine media acknowledgement too large" + ); + if b[0] == b'\n' { + break; + } + response.push(b[0]); + } + ensure!( + serde_json::from_slice::(&response)?["ok"] == true, + "Engine media was not admitted" + ); + Ok(super::BrowserCarrierStream { + send, + recv, + _client: client, + }) +} + +async fn open_media_stream( + client: super::CarrierClient, + authority: &Value, +) -> Result { + let (send, recv) = client.conn.open_bi().await?; + admit_opened_stream(send, recv, client, authority).await +} + +async fn open_pending_stream(client: super::CarrierClient) -> Result { + let (send, recv) = client.conn.open_bi().await?; + Ok(WarmPayload::Pending { client, send, recv }) +} + +async fn take_warm_stream(root: &Path, authority: &Value) -> Option { + let page = authority["page_id"].as_str()?; + let generation = authority["generation"].as_str()?; + let mut warms = MEDIA_WARMS.get_or_init(Default::default).lock().await; + let warm = warms.get_mut(&(root.to_owned(), page.into()))?; + if warm.generation != generation || !warm.attach_armed { + return None; + } + warm.attach_armed = false; + Some(TakenWarm { + rx: warm.first.take()?, + worker: warm.worker.take(), + }) +} + +fn spawn_stream_warm( + tx: oneshot::Sender>, + prepared: Option>>, + endpoint: iroh::Endpoint, + grant: Value, + authority: Value, + admit: bool, +) -> tokio::task::AbortHandle { + let prepared_abort = prepared.as_ref().map(|task| task.abort_handle()); + tokio::spawn(async move { + let _cancel_prepared = AbortOnDrop(prepared_abort.clone()); + let result = warm_payload_with_deadline(prepared_abort, STREAM_WARM_DEADLINE, async { + let client = match prepared { + Some(task) => match task.await { + Ok(Ok(client)) => Ok(client), + Ok(Err(err)) => Err(err.to_string()), + Err(err) => Err(err.to_string()), + }, + None => warm_media_client(&endpoint, &grant) + .await + .map_err(|err| err.to_string()), }; - let (mut send, mut recv) = client.conn.open_bi().await?; - super::write_json_line( - &mut send, - &json!({"op":"browser_engine_media","page_id":authority["page_id"], - "generation":authority["generation"],"binding_hash":authority["binding_hash"]}), - ) - .await?; - let mut response = Vec::new(); - loop { - let mut b = [0; 1]; - recv.read_exact(&mut b).await?; - ensure!( - response.len() < 1024, - "Engine media acknowledgement too large" - ); - if b[0] == b'\n' { - break; - } - response.push(b[0]); + match client { + Ok(client) if admit => open_media_stream(client, &authority) + .await + .map(WarmPayload::Ready) + .map_err(|err| err.to_string()), + Ok(client) => open_pending_stream(client) + .await + .map_err(|err| err.to_string()), + Err(err) => Err(err), } - ensure!( - serde_json::from_slice::(&response)?["ok"] == true, - "Engine media was not admitted" - ); - return Ok(super::BrowserCarrierStream { - send, - recv, - _client: client, - }); + }) + .await; + let _ = tx.send(result); + }) + .abort_handle() +} + +async fn replace_media_warm(key: (PathBuf, String), warm: MediaWarm) { + let previous = MEDIA_WARMS + .get_or_init(Default::default) + .lock() + .await + .insert(key, warm); + if let Some(previous) = previous { + abort_warm_worker(previous.worker); + } +} + +#[cfg(test)] +struct PreadmitCommitBarrier { + prepared: Arc, + release: watch::Sender, +} + +#[cfg(test)] +static PREADMIT_COMMIT_BARRIER: OnceLock>>> = + OnceLock::new(); + +#[cfg(test)] +async fn await_preadmit_commit_barrier() { + let barrier = PREADMIT_COMMIT_BARRIER + .get_or_init(Default::default) + .lock() + .ok() + .and_then(|slot| slot.clone()); + let Some(barrier) = barrier else { + return; + }; + let mut released = barrier.release.subscribe(); + barrier.prepared.wait().await; + while !*released.borrow() { + if released.changed().await.is_err() { + break; } - anyhow::bail!("Engine media route unavailable") + } +} + +async fn commit_prepared_media( + key: (PathBuf, String), + generation: String, + armed: bool, + prepared: T, + into_payload: impl FnOnce(T) -> Result, +) -> bool { + #[cfg(test)] + await_preadmit_commit_barrier().await; + + let ingresses = INGRESS.get_or_init(Default::default).lock().await; + if ingresses.get(&key).is_none() { + return false; + } + let (tx, rx) = oneshot::channel(); + let previous = { + let mut warms = MEDIA_WARMS.get_or_init(Default::default).lock().await; + warms.insert( + key, + MediaWarm { + generation, + first: Some(rx), + worker: None, + attach_armed: armed, + }, + ) + }; + drop(ingresses); + if let Some(previous) = previous { + abort_warm_worker(previous.worker); + } + let _ = tx.send(into_payload(prepared)); + true +} + +async fn install_stream_warm( + root: &Path, + page: &str, + generation: &str, + prepared: Option>>, + endpoint: iroh::Endpoint, + grant: Value, + authority: Value, + attach_armed: bool, + admit: bool, +) { + let (tx, rx) = oneshot::channel(); + let worker = spawn_stream_warm(tx, prepared, endpoint, grant, authority, admit); + replace_media_warm( + (root.to_owned(), page.into()), + MediaWarm { + generation: generation.into(), + first: Some(rx), + worker: Some(worker), + attach_armed, + }, + ) + .await; +} + +async fn install_pending_spare( + root: &Path, + page: &str, + generation: &str, + client: &super::CarrierClient, +) { + { + let warms = MEDIA_WARMS.get_or_init(Default::default).lock().await; + if let Some(warm) = warms.get(&(root.to_owned(), page.into())) { + if warm.generation == generation && warm.first.is_some() { + tracing::info!(page, "browser_media_warm spare_keep"); + return; + } + } + } + match open_pending_stream(clone_client_handle(client)).await { + Ok(payload) => { + let (tx, rx) = oneshot::channel(); + replace_media_warm( + (root.to_owned(), page.into()), + MediaWarm { + generation: generation.into(), + first: Some(rx), + worker: None, + attach_armed: false, + }, + ) + .await; + let _ = tx.send(Ok(payload)); + tracing::info!(page, "browser_media_warm spare_pending"); + } + Err(error) => tracing::info!(page, error = %error, "browser_media_warm spare_failed"), + } +} + +pub(crate) async fn preadmit_media_stream(root: &Path, authority: &Value) { + let Some(page) = authority["page_id"].as_str() else { + return; + }; + let Some(generation) = authority["generation"].as_str() else { + return; + }; + let key = (root.to_owned(), page.into()); + let mut warms = MEDIA_WARMS.get_or_init(Default::default).lock().await; + let Some(warm) = warms.get_mut(&key) else { + return; + }; + if warm.generation != generation { + return; + } + let Some(rx) = warm.first.take() else { + return; + }; + let armed = warm.attach_armed; + let worker = warm.worker.take(); + drop(warms); + let payload = match tokio::time::timeout(Duration::from_secs(2), rx).await { + Ok(Ok(Ok(payload))) => payload, + _ => { + abort_warm_worker(worker); + tracing::info!(page, "browser_media_warm preadmit_miss"); + return; + } + }; + let ready = match payload { + WarmPayload::Ready(stream) => { + tracing::info!(page, "browser_media_warm preadmit_already_ready"); + stream + } + WarmPayload::Pending { client, send, recv } => match tokio::time::timeout( + STREAM_WARM_DEADLINE, + admit_opened_stream(send, recv, client, authority), + ) + .await + { + Ok(Ok(stream)) => { + tracing::info!(page, "browser_media_warm preadmit_ready"); + stream + } + Ok(Err(error)) => { + tracing::info!(page, error = %error, "browser_media_warm preadmit_failed"); + return; + } + Err(_) => { + tracing::info!(page, "browser_media_warm preadmit_deadline"); + return; + } + }, + }; + if !commit_prepared_media(key, generation.into(), armed, ready, |stream| { + Ok(WarmPayload::Ready(stream)) }) .await - .context("Engine media connection deadline")? + { + tracing::info!(page, "browser_media_warm preadmit_retired"); + } +} + +pub(crate) async fn rearm_media_stream( + root: &Path, + endpoint: &iroh::Endpoint, + grant: &Value, + authority: &Value, +) { + let Some(page) = authority["page_id"].as_str() else { + return; + }; + let Some(generation) = authority["generation"].as_str() else { + return; + }; + let key = (root.to_owned(), page.into()); + let ingresses = INGRESS.get_or_init(Default::default).lock().await; + let Some(ingress) = ingresses.get(&key) else { + return; + }; + if ingress.generation != generation || *ingress.cancel.borrow() || *ingress.closed.borrow() { + return; + } + drop(ingresses); + let mut warms = MEDIA_WARMS.get_or_init(Default::default).lock().await; + if let Some(warm) = warms.get_mut(&key) { + if warm.generation == generation && warm.first.is_some() { + warm.attach_armed = true; + tracing::info!(page, "browser_media_warm rearm_keep"); + return; + } + } + drop(warms); + tracing::info!(page, "browser_media_warm rearm_pending"); + install_stream_warm( + root, + page, + generation, + None, + endpoint.clone(), + grant.clone(), + authority.clone(), + true, + false, + ) + .await; +} + +async fn connect( + root: &Path, + endpoint: &iroh::Endpoint, + grant: &Value, + authority: &Value, +) -> Result { + let stream = if let Some(taken) = take_warm_stream(root, authority).await { + let TakenWarm { rx, worker } = taken; + if let Ok(Ok(Ok(payload))) = tokio::time::timeout(Duration::from_secs(5), rx).await { + match payload { + WarmPayload::Ready(stream) => { + tracing::info!("browser_media_warm take_ready"); + stream + } + WarmPayload::Pending { client, send, recv } => { + tracing::info!("browser_media_warm take_pending"); + admit_opened_stream(send, recv, client, authority).await? + } + } + } else { + abort_warm_worker(worker); + tracing::info!("browser_media_warm take_failed"); + tokio::time::timeout(Duration::from_secs(5), async { + open_media_stream(warm_media_client(endpoint, grant).await?, authority).await + }) + .await + .context("Engine media connection deadline")?? + } + } else { + tracing::info!("browser_media_warm take_miss"); + tokio::time::timeout(Duration::from_secs(5), async { + open_media_stream(warm_media_client(endpoint, grant).await?, authority).await + }) + .await + .context("Engine media connection deadline")?? + }; + if let (Some(page), Some(generation)) = ( + authority["page_id"].as_str(), + authority["generation"].as_str(), + ) { + install_pending_spare(root, page, generation, &stream._client).await; + } + Ok(stream) } #[derive(serde::Deserialize)] @@ -269,6 +698,14 @@ fn read_config(root: &Path) -> Result { ); Ok(config) } + +fn disable_media_nagle(stream: &tokio::net::TcpStream) -> Result<()> { + stream + .set_nodelay(true) + .context("Runtime media TCP requires TCP_NODELAY")?; + Ok(()) +} + fn valid_host(host: &str) -> bool { if let Ok(ip) = host.parse::() { return !ip.is_unspecified() && !ip.is_multicast(); @@ -292,6 +729,7 @@ pub(crate) async fn start_ingress( endpoint: &iroh::Endpoint, grant: &Value, authority: &Value, + prepared: Option>>, ) -> Result { let config = read_config(root)?; let page = authority["page_id"] @@ -318,7 +756,12 @@ pub(crate) async fn start_ingress( ), "Viewer ingress owner changed" ); - return Ok(ingress.descriptor.clone()); + let descriptor = ingress.descriptor.clone(); + drop(ingresses); + if let Some(task) = prepared { + task.abort(); + } + return Ok(descriptor); } let mut listener = None; for port in config.port_start..=config.port_end { @@ -338,8 +781,20 @@ pub(crate) async fn start_ingress( "turn_url":format!("turn:{host}:{}?transport=tcp",listener.local_addr()?.port())}); let (cancel, mut cancelled) = watch::channel(false); let (done, closed) = watch::channel(false); + install_stream_warm( + root, + page, + generation, + prepared, + endpoint.clone(), + grant.clone(), + authority.clone(), + true, + true, + ) + .await; ingresses.insert( - key, + key.clone(), Ingress { generation: generation.into(), descriptor: descriptor.clone(), @@ -348,6 +803,7 @@ pub(crate) async fn start_ingress( }, ); drop(ingresses); + let root = root.to_owned(); let endpoint = endpoint.clone(); let grant = grant.clone(); let authority = authority.clone(); @@ -360,10 +816,11 @@ pub(crate) async fn start_ingress( result=listener.accept()=>match result { Ok((socket,_))=>{ let Ok(slot)=slots.clone().try_acquire_owned() else {drop(socket);continue;}; - let endpoint=endpoint.clone();let grant=grant.clone();let authority=authority.clone(); + let root=root.clone();let endpoint=endpoint.clone();let grant=grant.clone();let authority=authority.clone(); children.spawn(async move { let _slot=slot; - let mut remote=connect(&endpoint,&grant,&authority).await?; + disable_media_nagle(&socket)?; + let mut remote=connect(&root,&endpoint,&grant,&authority).await?; let (mut read,mut write)=socket.into_split(); tokio::try_join!(async {tokio::io::copy(&mut read,&mut remote.send).await?;remote.send.finish()?;Ok::<_,anyhow::Error>(())}, async {tokio::io::copy(&mut remote.recv,&mut write).await?;write.shutdown().await?;Ok::<_,anyhow::Error>(())})?; @@ -404,6 +861,14 @@ pub(crate) async fn close_ingress(root: &Path, page: &str, generation: &str) -> .await??; ingresses = INGRESS.get_or_init(Default::default).lock().await; ingresses.remove(&key); + let previous = MEDIA_WARMS + .get_or_init(Default::default) + .lock() + .await + .remove(&key); + if let Some(previous) = previous { + abort_warm_worker(previous.worker); + } Ok(()) } @@ -512,9 +977,9 @@ mod tests { std::fs::write(consumer_root.path().join("config/browser-viewer-ingress.json"),serde_json::to_vec(&json!({ "schema":"elastos.browser.viewer-ingress-config/v1","listen_host":"127.0.0.1","advertised_host":"127.0.0.1", "port_start":port,"port_end":port})).unwrap()).unwrap(); - let ingress=start_ingress(consumer_root.path(),&consumer.endpoint,&grant,&authority).await.unwrap(); + let ingress=start_ingress(consumer_root.path(),&consumer.endpoint,&grant,&authority,None).await.unwrap(); validate_ingress(&authority,&ingress,ingress["turn_url"].as_str().unwrap()).unwrap(); - assert_eq!(start_ingress(consumer_root.path(),&consumer.endpoint,&grant,&authority).await.unwrap(),ingress); + assert_eq!(start_ingress(consumer_root.path(),&consumer.endpoint,&grant,&authority,None).await.unwrap(),ingress); assert!(!ingress.to_string().contains(&engine.endpoint.id().to_string())); let echo=tokio::spawn(async move { let (mut socket,_)=target.accept().await.unwrap();let mut bytes=[0;8]; @@ -525,16 +990,220 @@ mod tests { let mut viewer=tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST,port)).await.unwrap(); viewer.write_all(b"turndata").await.unwrap();let mut bytes=[0;8];viewer.read_exact(&mut bytes).await.unwrap();assert_eq!(&bytes,b"received"); let anonymous=iroh::Endpoint::builder(iroh::endpoint::presets::Minimal).bind().await.unwrap(); - assert!(connect(&anonymous,&grant,&authority).await.is_err(),"routing information cannot confer Runtime authority"); + assert!(connect(consumer_root.path(),&anonymous,&grant,&authority).await.is_err(),"routing information cannot confer Runtime authority"); let mut stale=authority.clone();stale["generation"]=json!(runtime_hash("stale")); - assert!(connect(&consumer.endpoint,&grant,&stale).await.is_err()); + assert!(connect(consumer_root.path(),&consumer.endpoint,&grant,&stale).await.is_err()); assert!(close_ingress(consumer_root.path(),"page:media-test","foreign").await.is_err()); remove_target(engine_root.path(),"page:media-test",authority["generation"].as_str().unwrap()).await.unwrap(); echo.await.unwrap(); close_ingress(consumer_root.path(),"page:media-test",authority["generation"].as_str().unwrap()).await.unwrap(); assert!(tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST,port)).await.is_err()); - assert!(connect(&consumer.endpoint,&grant,&authority).await.is_err()); + assert!(connect(consumer_root.path(),&consumer.endpoint,&grant,&authority).await.is_err()); anonymous.close().await;consumer.endpoint.close().await;engine.endpoint.close().await; }).await.unwrap(); } + + #[tokio::test] + async fn stream_warm_deadline_aborts_hung_prepare() { + let prepared = tokio::spawn(std::future::pending::<()>()); + let abort = prepared.abort_handle(); + let started = tokio::time::Instant::now(); + let result = warm_payload_with_deadline( + Some(abort.clone()), + Duration::from_millis(50), + std::future::pending(), + ) + .await; + match result { + Err(error) => assert_eq!(error, "Engine media warm deadline"), + Ok(_) => panic!("hung prepare must miss the warm deadline"), + } + assert!(started.elapsed() < Duration::from_millis(500)); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(abort.is_finished()); + drop(prepared); + } + + #[tokio::test] + async fn take_warm_timeout_aborts_owned_worker() { + let root = tempfile::tempdir().unwrap(); + let authority = json!({"page_id":"page:warm-timeout","generation":"gen"}); + let (_tx, rx) = oneshot::channel::>(); + let worker = tokio::spawn(std::future::pending::<()>()); + let abort = worker.abort_handle(); + MEDIA_WARMS + .get_or_init(Default::default) + .lock() + .await + .insert( + (root.path().to_owned(), "page:warm-timeout".into()), + MediaWarm { + generation: "gen".into(), + first: Some(rx), + worker: Some(abort.clone()), + attach_armed: true, + }, + ); + let TakenWarm { rx, worker } = take_warm_stream(root.path(), &authority).await.unwrap(); + assert!(tokio::time::timeout(Duration::from_millis(20), rx) + .await + .is_err()); + abort_warm_worker(worker); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(abort.is_finished()); + } + + #[tokio::test] + async fn close_ingress_aborts_prepared_warm_worker() { + let root = tempfile::tempdir().unwrap(); + let unused = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = unused.local_addr().unwrap().port(); + drop(unused); + std::fs::create_dir_all(root.path().join("config")).unwrap(); + std::fs::write( + root.path().join("config/browser-viewer-ingress.json"), + serde_json::to_vec(&json!({ + "schema":"elastos.browser.viewer-ingress-config/v1", + "listen_host":"127.0.0.1", + "advertised_host":"127.0.0.1", + "port_start":port, + "port_end":port + })) + .unwrap(), + ) + .unwrap(); + let endpoint = iroh::Endpoint::builder(iroh::endpoint::presets::Minimal) + .bind() + .await + .unwrap(); + let grant = json!({"peer_did":endpoint.id().to_string(),"connect_ticket":"ticket"}); + let authority = json!({ + "page_id":"page:warm-close", + "generation":"gen", + "binding_hash":runtime_hash("binding") + }); + let prepared = tokio::spawn(std::future::pending::>()); + let abort = prepared.abort_handle(); + start_ingress(root.path(), &endpoint, &grant, &authority, Some(prepared)) + .await + .unwrap(); + close_ingress(root.path(), "page:warm-close", "gen") + .await + .unwrap(); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(abort.is_finished()); + endpoint.close().await; + } + + struct ReleasedPreparedStream(Arc); + + impl Drop for ReleasedPreparedStream { + fn drop(&mut self) { + self.0.store(true, Ordering::SeqCst); + } + } + + struct PreadmitBarrierGuard(Arc); + + impl Drop for PreadmitBarrierGuard { + fn drop(&mut self) { + if let Ok(mut slot) = PREADMIT_COMMIT_BARRIER.get_or_init(Default::default).lock() { + *slot = None; + } + let _ = self.0.release.send(true); + } + } + + fn install_preadmit_commit_barrier() -> PreadmitBarrierGuard { + let (release, _) = watch::channel(false); + let barrier = Arc::new(PreadmitCommitBarrier { + prepared: Arc::new(tokio::sync::Barrier::new(2)), + release, + }); + *PREADMIT_COMMIT_BARRIER + .get_or_init(Default::default) + .lock() + .expect("preadmit barrier") = Some(barrier.clone()); + PreadmitBarrierGuard(barrier) + } + + async fn insert_fast_close_ingress(root: &Path, page: &str, generation: &str) { + let (cancel, _) = watch::channel(false); + let (_done, closed) = watch::channel(true); + INGRESS.get_or_init(Default::default).lock().await.insert( + (root.to_owned(), page.into()), + Ingress { + generation: generation.into(), + descriptor: json!({}), + cancel, + closed, + }, + ); + } + + #[tokio::test] + async fn late_preadmit_after_close_releases_prepared_stream() { + let root = tempfile::tempdir().unwrap(); + let page = "page:late-preadmit"; + let key = (root.path().to_owned(), page.into()); + insert_fast_close_ingress(root.path(), page, "gen").await; + let barrier = install_preadmit_commit_barrier(); + let released = Arc::new(AtomicBool::new(false)); + let pending = tokio::spawn({ + let key = key.clone(); + let released = released.clone(); + async move { + commit_prepared_media( + key, + "gen".into(), + true, + ReleasedPreparedStream(released), + |_| Err("test".into()), + ) + .await + } + }); + barrier.0.prepared.wait().await; + close_ingress(root.path(), page, "gen").await.unwrap(); + let _ = barrier.0.release.send(true); + assert_eq!(pending.await.unwrap(), false); + assert!(released.load(Ordering::SeqCst)); + assert!(MEDIA_WARMS + .get_or_init(Default::default) + .lock() + .await + .get(&key) + .is_none()); + assert!(INGRESS + .get_or_init(Default::default) + .lock() + .await + .get(&key) + .is_none()); + } + + #[tokio::test] + async fn preadmit_commit_retains_stream_while_ingress_owns_the_page() { + let root = tempfile::tempdir().unwrap(); + let page = "page:preadmit-retain"; + let key = (root.path().to_owned(), page.into()); + insert_fast_close_ingress(root.path(), page, "gen").await; + assert!( + commit_prepared_media(key.clone(), "gen".into(), true, (), |_| Err("kept".into())) + .await + ); + assert!(MEDIA_WARMS + .get_or_init(Default::default) + .lock() + .await + .get(&key) + .is_some()); + close_ingress(root.path(), page, "gen").await.unwrap(); + assert!(MEDIA_WARMS + .get_or_init(Default::default) + .lock() + .await + .get(&key) + .is_none()); + } } diff --git a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs index 3ea941d64..ea490da1e 100644 --- a/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs +++ b/elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs @@ -2,16 +2,15 @@ use std::fmt::Write as _; use std::fs::{self, File, OpenOptions}; use std::io::{BufRead, BufReader, ErrorKind, Read, Write}; use std::net::Shutdown; -use std::os::fd::{AsRawFd, OwnedFd, RawFd}; +use std::os::fd::{AsRawFd, FromRawFd, OwnedFd, RawFd}; use std::os::unix::fs::{FileTypeExt, MetadataExt, OpenOptionsExt, PermissionsExt}; use std::os::unix::net::{UnixListener, UnixStream}; +use std::os::unix::process::CommandExt as _; use std::path::{Path, PathBuf}; -use std::process; -#[cfg(target_os = "macos")] -use std::process::Command; +use std::process::{self, Command, Stdio}; use std::sync::{ - atomic::{AtomicBool, Ordering}, - Arc, + atomic::{AtomicBool, AtomicI32, Ordering}, + Arc, Once, }; use std::thread; use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; @@ -36,6 +35,8 @@ const VZ_TRANSPORT_SECRET_SCHEMA: &str = "elastos.browser.vz-transport-secret/v1 const VZ_LAUNCH_SETTLEMENT_SCHEMA: &str = "elastos.browser.vz-launch-settlement/v1"; const VZ_MEDIA_DIAGNOSTIC_SCHEMA: &str = "elastos.browser.media-diagnostic/v1"; const DEFAULT_CONTROL_PORT: u32 = 19092; +const TURN_PORT_ABSENCE_BUDGET: Duration = Duration::from_secs(3); +const TURN_PORT_ABSENCE_POLL: Duration = Duration::from_millis(50); const DEFAULT_PROFILE_DISK_MIB: u64 = 2048; const UNIX_SOCKET_PATH_BUDGET: usize = 100; const EGRESS_COPY_BUFFER_BYTES: usize = 256 * 1024; @@ -45,6 +46,7 @@ const MAX_CONTROL_HTTP_RESPONSE_BYTES: usize = 64 * 1024 * 1024; const MAX_GUEST_CONTROL_ERROR_CHARS: usize = 512; const MAX_SETTLEMENT_MESSAGE_CHARS: usize = 2 * 1024; const DEFAULT_CONTROL_PROXY_REQUEST_TIMEOUT_MS: u32 = 120_000; +const DEFAULT_PROFILE_FLUSH_TIMEOUT_MS: u32 = 30_000; const BROWSER_VM_TARGET_VERSION: &str = match option_env!("ELASTOS_RELEASE_VERSION") { Some(version) => version, None => concat!(env!("CARGO_PKG_VERSION"), "-dev"), @@ -252,11 +254,25 @@ impl VzLaunchOwner { self.settlement_with_absence(message, absence) } + async fn settle_failed_profile_flush(&mut self, message: impl AsRef) -> String { + let absence = self.cleanup().await; + self.settlement_with_absence_and_profile_durability(message, absence, Some("failed")) + } + fn settlement_with_absence(&self, message: impl AsRef, absence: Value) -> String { + self.settlement_with_absence_and_profile_durability(message, absence, None) + } + + fn settlement_with_absence_and_profile_durability( + &self, + message: impl AsRef, + absence: Value, + profile_durability: Option<&str>, + ) -> String { let terminal = absence .as_object() .is_some_and(|values| values.values().all(|value| value == &Value::Bool(true))); - serde_json::to_string(&json!({ + let mut settlement = json!({ "schema": VZ_LAUNCH_SETTLEMENT_SCHEMA, "state": if terminal { "terminal_post_effect_cleanup" @@ -272,8 +288,11 @@ impl VzLaunchOwner { "media_stream_id": self.identity.media_stream_id, "effects": self.effects(), "absence": absence, - })) - .unwrap_or_else(|_| { + }); + if let Some(durability) = profile_durability { + settlement["profile_durability"] = json!(durability); + } + serde_json::to_string(&settlement).unwrap_or_else(|_| { "Browser VZ launch cleanup settlement serialization failed; cleanup remains indeterminate" .to_string() }) @@ -360,10 +379,7 @@ impl VzLaunchOwner { .as_str() .is_some_and(|vm_id| remove_owned_vm_state(&self.paths, vm_id, vm_absent)); let (turn_listener_absent, turn_relay_ports_absent) = if probe_turn_ports { - ( - turn_listener_port_absent(&self.transport), - turn_relay_ports_absent(&self.transport), - ) + wait_for_owned_turn_ports_absent(&self.transport).await } else { (true, true) }; @@ -679,7 +695,41 @@ async fn run() -> Result<(), String> { trace_stage("open_guest_page_done", ""); println!("{}", post_effect_try!(serde_json::to_string(&result))); - wait_for_shutdown_or_transport_expiry(Some(&owner.transport)).await; + let shutdown_reason = wait_for_shutdown_or_transport_expiry( + Some(&owner.transport), + owner.provider.clone(), + owner.handle.clone(), + ) + .await; + eprintln!("{}", format_shutdown_wait_reason(&shutdown_reason)); + if let ShutdownWaitReason::Signal { sender_pid, .. } = &shutdown_reason { + eprintln!( + "browser-vz-engine-supervisor stage=shutdown_wait_sender sender_pid={sender_pid} comm={}", + shutdown_signal_sender_comm(*sender_pid) + ); + } + if let ShutdownWaitReason::GuestStopped { state } = &shutdown_reason { + return Err(owner + .settle_failure(format!("Browser VZ guest or framework stopped ({state})")) + .await); + } + let mut profile_durability = None; + if let (Some(provider), Some(handle)) = (owner.provider.clone(), owner.handle.clone()) { + let flush = + request_guest_profile_disk_flush(provider, &handle, owner.paths.control_port).await; + eprintln!( + "browser-vz-engine-supervisor stage=guest_profile_flush {}", + format_guest_profile_flush(&flush) + ); + if !guest_profile_disk_flush_accepted(&flush) { + return Err(owner + .settle_failed_profile_flush( + "Browser VZ guest shutdown did not flush the profile disk", + ) + .await); + } + profile_durability = Some("proved"); + } let absence = owner.cleanup().await; if !absence .as_object() @@ -690,6 +740,16 @@ async fn run() -> Result<(), String> { absence, )); } + if let Some(durability) = profile_durability { + eprintln!( + "{}", + owner.settlement_with_absence_and_profile_durability( + "Browser VZ guest shutdown flushed the profile disk", + absence, + Some(durability), + ) + ); + } Ok(()) } @@ -1520,12 +1580,22 @@ impl LaunchTurn { .and_then(|_| file.sync_all()) .map_err(|err| format!("Browser VZ TURN config write failed: {err}"))?; drop(file); - let mut child = std::process::Command::new(&program) + let mut command = std::process::Command::new(&program); + command .arg("-c") .arg(&config_path) .stdin(std::process::Stdio::null()) .stdout(std::process::Stdio::piped()) - .stderr(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()); + unsafe { + command.pre_exec(|| { + if libc::setpgid(0, 0) != 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + }); + } + let mut child = command .spawn() .map_err(|err| format!("Browser VZ TURN process failed to start: {err}"))?; let mut log_threads = Vec::with_capacity(2); @@ -1590,9 +1660,12 @@ impl LaunchTurn { } fn terminate_and_reap(&mut self) -> bool { + let child_pid = self.child.id() as i32; + signal_owned_turn_process_group(child_pid, libc::SIGTERM); let child_absent = match self.child.try_wait() { Ok(Some(_)) => true, Ok(None) => { + signal_owned_turn_process_group(child_pid, libc::SIGKILL); let _ = self.child.kill(); let deadline = Instant::now() + Duration::from_secs(10); loop { @@ -2005,6 +2078,29 @@ fn turn_listener_port_absent(transport: &VzTransportLaunch) -> bool { std::net::TcpListener::bind((bindings.listen_host.as_str(), bindings.listen_port)).is_ok() } +fn signal_owned_turn_process_group(child_pid: i32, signum: libc::c_int) { + if child_pid > 1 { + unsafe { + libc::killpg(child_pid, signum); + } + } +} + +async fn wait_for_owned_turn_ports_absent(transport: &VzTransportLaunch) -> (bool, bool) { + let deadline = Instant::now() + TURN_PORT_ABSENCE_BUDGET; + loop { + let listener = turn_listener_port_absent(transport); + let relays = turn_relay_ports_absent(transport); + if listener && relays { + return (true, true); + } + if Instant::now() >= deadline { + return (listener, relays); + } + tokio::time::sleep(TURN_PORT_ABSENCE_POLL).await; + } +} + fn turn_relay_ports_absent(transport: &VzTransportLaunch) -> bool { let Ok(bindings) = turn_port_bindings(transport) else { return false; @@ -2306,12 +2402,244 @@ fn prepare_launch_rootfs(paths: &LaunchPaths) -> Result Result { + if let Ok(explicit) = std::env::var("ELASTOS_DEBUGFS_BIN") { + let path = PathBuf::from(explicit); + if path.is_file() { + return Ok(path); + } + return Err(format!( + "ELASTOS_DEBUGFS_BIN is not a file: {}", + path.display() + )); + } + for candidate in [ + "/opt/homebrew/opt/e2fsprogs/sbin/debugfs", + "/usr/local/opt/e2fsprogs/sbin/debugfs", + "/usr/sbin/debugfs", + "/sbin/debugfs", + ] { + let path = PathBuf::from(candidate); + if path.is_file() { + return Ok(path); + } + } + Err("debugfs is required to overlay the guest Browser control service".to_string()) +} + +fn host_selkies_control_service_path() -> Result { + if let Ok(explicit) = std::env::var("ELASTOS_BROWSER_SELKIES_CONTROL_SERVICE") { + let path = PathBuf::from(explicit); + if path.is_file() { + return Ok(path); + } + return Err(format!( + "ELASTOS_BROWSER_SELKIES_CONTROL_SERVICE is not a file: {}", + path.display() + )); + } + let exe = std::env::current_exe().map_err(|err| err.to_string())?; + if let Some(bin_dir) = exe.parent() { + if let Some(data_dir) = bin_dir.parent() { + let candidate = data_dir.join("scripts/browser-selkies-control-service.mjs"); + if candidate.is_file() { + return Ok(candidate); + } + } + } + Err("Browser selkies control service source is missing".to_string()) +} + +fn overlay_guest_selkies_control_service(rootfs: &Path) -> Result<(), String> { + let source = host_selkies_control_service_path()?; + let debugfs = find_debugfs_bin()?; + let staging = + std::env::temp_dir().join(format!("evz-selkies-control-{}.mjs", std::process::id())); + fs::copy(&source, &staging).map_err(|err| { + format!( + "stage guest Browser control service {} failed: {err}", + source.display() + ) + })?; + let rm = Command::new(&debugfs) + .args([ + "-w", + "-R", + &format!("rm {GUEST_SELKIES_CONTROL_SERVICE}"), + rootfs + .as_os_str() + .to_str() + .ok_or_else(|| "per-launch Browser VM rootfs path is not UTF-8".to_string())?, + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .map_err(|err| format!("debugfs rm guest Browser control service failed: {err}"))?; + if !rm.success() { + let _ = fs::remove_file(&staging); + return Err("debugfs rm guest Browser control service failed".to_string()); + } + let write = Command::new(&debugfs) + .args([ + "-w", + "-R", + &format!( + "write {} {GUEST_SELKIES_CONTROL_SERVICE}", + staging.display() + ), + rootfs + .as_os_str() + .to_str() + .ok_or_else(|| "per-launch Browser VM rootfs path is not UTF-8".to_string())?, + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .map_err(|err| format!("debugfs write guest Browser control service failed: {err}"))?; + let _ = fs::remove_file(&staging); + if !write.success() { + return Err("debugfs write guest Browser control service failed".to_string()); + } + eprintln!( + "browser-vz-engine-supervisor stage=guest_control_overlay source={}", + source.display() + ); + Ok(()) +} + +fn host_data_script(name: &str) -> Option { + let exe = std::env::current_exe().ok()?; + let candidate = exe.parent()?.parent()?.join("scripts").join(name); + candidate.is_file().then_some(candidate) +} + +fn webrtc_send_overlay_requested() -> bool { + std::env::var("ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY") + .ok() + .as_deref() + == Some("1") +} + +fn host_webrtc_send_overlay_sources() -> Option<(PathBuf, PathBuf)> { + let gst = std::env::var("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC") + .ok() + .map(PathBuf::from) + .or_else(|| host_data_script("browser-webrtc-send-gstwebrtc.py")); + let start = std::env::var("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START") + .ok() + .map(PathBuf::from) + .or_else(|| host_data_script("browser-webrtc-send-selkies-start.sh")); + match (gst, start) { + (Some(gst), Some(start)) if gst.is_file() && start.is_file() => Some((gst, start)), + _ => None, + } +} + +fn sha256_file_label(path: &Path) -> Result { + let bytes = + fs::read(path).map_err(|err| format!("read overlay {} failed: {err}", path.display()))?; + Ok(sha256_label(&bytes)) +} + +fn require_overlay_input_hash(env_name: &str, actual: &str) -> Result<(), String> { + let expected = std::env::var(env_name).map_err(|_| { + format!("{env_name} is required when Browser webrtc send overlay is enabled") + })?; + let expected = expected + .strip_prefix("sha256:") + .unwrap_or(expected.as_str()); + let actual = actual.strip_prefix("sha256:").unwrap_or(actual); + if expected != actual { + return Err(format!("{env_name} does not match overlay input")); + } + Ok(()) +} + +fn overlay_guest_file(rootfs: &Path, source: &Path, guest_path: &str) -> Result<(), String> { + let debugfs = find_debugfs_bin()?; + let staging = std::env::temp_dir().join(format!( + "evz-overlay-{}-{}", + std::process::id(), + source + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or("guest") + )); + fs::copy(source, &staging) + .map_err(|err| format!("stage guest overlay {} failed: {err}", source.display()))?; + let rootfs_text = rootfs + .as_os_str() + .to_str() + .ok_or_else(|| "per-launch Browser VM rootfs path is not UTF-8".to_string())?; + let rm = Command::new(&debugfs) + .args(["-w", "-R", &format!("rm {guest_path}"), rootfs_text]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .map_err(|err| format!("debugfs rm {guest_path} failed: {err}"))?; + if !rm.success() { + let _ = fs::remove_file(&staging); + return Err(format!("debugfs rm {guest_path} failed")); + } + let write = Command::new(&debugfs) + .args([ + "-w", + "-R", + &format!("write {} {guest_path}", staging.display()), + rootfs_text, + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .map_err(|err| format!("debugfs write {guest_path} failed: {err}"))?; + let _ = fs::remove_file(&staging); + if !write.success() { + return Err(format!("debugfs write {guest_path} failed")); + } + Ok(()) +} + +fn overlay_guest_webrtc_send_diagnostic(rootfs: &Path) -> Result<(), String> { + if !webrtc_send_overlay_requested() { + return Ok(()); + } + let Some((gst, start)) = host_webrtc_send_overlay_sources() else { + return Err( + "Browser webrtc send overlay is enabled but host overlay files are absent".to_string(), + ); + }; + let gst_hash = sha256_file_label(&gst)?; + let start_hash = sha256_file_label(&start)?; + require_overlay_input_hash("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC_SHA256", &gst_hash)?; + require_overlay_input_hash( + "ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START_SHA256", + &start_hash, + )?; + overlay_guest_file(rootfs, &gst, GUEST_GSTWEBRTC_APP)?; + overlay_guest_file(rootfs, &start, GUEST_SELKIES_START)?; + eprintln!( + "browser-vz-engine-supervisor stage=guest_webrtc_send_overlay gst={} gst_sha256={} start={} start_sha256={}", + gst.display(), + gst_hash, + start.display(), + start_hash + ); + Ok(()) +} + fn clone_or_copy_file(source: &Path, destination: &Path) -> std::io::Result<()> { if let Some(parent) = destination.parent() { fs::create_dir_all(parent)?; @@ -2720,6 +3048,61 @@ fn is_guest_control_response_timeout(error: &str) -> bool { error.contains("Browser VM control HTTP response timed out") } +fn guest_profile_disk_flush_accepted(result: &Value) -> bool { + result.get("ok") == Some(&Value::Bool(true)) + && result.get("profile_disk_flushed") == Some(&Value::Bool(true)) + && result.get("profile_disk_unmounted") == Some(&Value::Bool(true)) + && result.get("leftover_writers_killed") != Some(&Value::Bool(true)) +} + +fn format_guest_profile_flush(result: &Value) -> String { + let flushed = result.get("profile_disk_flushed") == Some(&Value::Bool(true)); + let unmounted = result.get("profile_disk_unmounted") == Some(&Value::Bool(true)); + let leftover_killed = result.get("leftover_writers_killed") == Some(&Value::Bool(true)); + let fields = format!( + "profile_disk_flushed={flushed} profile_disk_unmounted={unmounted} leftover_writers_killed={leftover_killed}" + ); + if guest_profile_disk_flush_accepted(result) { + format!("ok=true {fields}") + } else if let Some(error) = result.get("error").and_then(Value::as_str) { + format!("ok=false {fields} error={error}") + } else { + format!("ok=false {fields}") + } +} + +async fn request_guest_profile_disk_flush( + provider: Arc, + handle: &CapsuleHandle, + control_port: u32, +) -> Value { + let timeout = match env_u32( + "ELASTOS_BROWSER_VM_PROFILE_FLUSH_TIMEOUT_MS", + DEFAULT_PROFILE_FLUSH_TIMEOUT_MS, + ) { + Ok(ms) => Duration::from_millis(ms as u64), + Err(error) => { + return json!({ + "ok": false, + "error": bounded_message(&error), + }); + } + }; + match connect_vsock_with_retry(provider, handle, control_port, timeout).await { + Ok(fd) => match http_json_over_fd(fd, "POST", "/shutdown", None, Some(timeout)) { + Ok(result) => result, + Err(error) => json!({ + "ok": false, + "error": bounded_message(&error), + }), + }, + Err(error) => json!({ + "ok": false, + "error": bounded_message(&error), + }), + } +} + async fn connect_vsock_with_retry( provider: Arc, handle: &CapsuleHandle, @@ -3628,38 +4011,225 @@ fn write_error_response(mut stream: UnixStream, message: &str) -> Result<(), Str stream.write_all(&body).map_err(|err| err.to_string()) } -async fn wait_for_shutdown_signal() { - #[cfg(unix)] - { - use tokio::signal::unix::{signal, SignalKind}; - if let Ok(mut term) = signal(SignalKind::terminate()) { - tokio::select! { - _ = tokio::signal::ctrl_c() => {} - _ = term.recv() => {} +#[derive(Debug, PartialEq, Eq)] +enum ShutdownWaitReason { + Signal { + signum: i32, + sender_pid: u32, + si_code: i32, + }, + TransportExpired { + expires_at_unix_ms: u64, + now_unix_ms: u64, + }, + GuestStopped { + state: String, + }, +} + +fn transport_remaining_ms(expires_at_unix_ms: u64, now_unix_ms: u64) -> u64 { + expires_at_unix_ms.checked_sub(now_unix_ms).unwrap_or(0) +} + +fn format_shutdown_wait_reason(reason: &ShutdownWaitReason) -> String { + match reason { + ShutdownWaitReason::Signal { + signum, + sender_pid, + si_code, + } => format!( + "browser-vz-engine-supervisor stage=shutdown_wait reason=signal signum={signum} sender_pid={sender_pid} si_code={si_code}" + ), + ShutdownWaitReason::TransportExpired { + expires_at_unix_ms, + now_unix_ms, + } => format!( + "browser-vz-engine-supervisor stage=shutdown_wait reason=transport_expired expires_at_unix_ms={expires_at_unix_ms} now_unix_ms={now_unix_ms}" + ), + ShutdownWaitReason::GuestStopped { state } => format!( + "browser-vz-engine-supervisor stage=shutdown_wait reason=guest_stopped state={state}" + ), + } +} + +static SHUTDOWN_SIGNAL_WRITE_FD: AtomicI32 = AtomicI32::new(-1); +static SHUTDOWN_SIGNAL_INSTALL: Once = Once::new(); + +extern "C" fn report_shutdown_signal( + signum: libc::c_int, + info: *mut libc::siginfo_t, + _: *mut libc::c_void, +) { + let fd = SHUTDOWN_SIGNAL_WRITE_FD.load(Ordering::Relaxed); + if fd < 0 { + return; + } + let sender_pid = if info.is_null() { + 0 + } else { + unsafe { (*info).si_pid }.max(0) + }; + let si_code = if info.is_null() { + 0 + } else { + unsafe { (*info).si_code } + }; + let mut buf = [0u8; 12]; + buf[0..4].copy_from_slice(&signum.to_ne_bytes()); + buf[4..8].copy_from_slice(&sender_pid.to_ne_bytes()); + buf[8..12].copy_from_slice(&si_code.to_ne_bytes()); + unsafe { + libc::write(fd, buf.as_ptr().cast(), buf.len()); + } +} + +fn install_shutdown_signal_reporter() -> Result { + let mut fds = [0 as RawFd; 2]; + if unsafe { libc::pipe(fds.as_mut_ptr()) } != 0 { + return Err("Browser VZ shutdown signal pipe failed".to_string()); + } + unsafe { + libc::fcntl(fds[0], libc::F_SETFD, libc::FD_CLOEXEC); + libc::fcntl(fds[1], libc::F_SETFD, libc::FD_CLOEXEC); + libc::fcntl(fds[0], libc::F_SETFL, libc::O_NONBLOCK); + } + SHUTDOWN_SIGNAL_WRITE_FD.store(fds[1], Ordering::SeqCst); + SHUTDOWN_SIGNAL_INSTALL.call_once(|| { + let mut action: libc::sigaction = unsafe { std::mem::zeroed() }; + action.sa_flags = libc::SA_SIGINFO | libc::SA_RESTART; + action.sa_sigaction = report_shutdown_signal as libc::sighandler_t; + unsafe { + libc::sigemptyset(&mut action.sa_mask); + let _ = libc::sigaction(libc::SIGTERM, &action, std::ptr::null_mut()); + let _ = libc::sigaction(libc::SIGINT, &action, std::ptr::null_mut()); + } + }); + Ok(unsafe { OwnedFd::from_raw_fd(fds[0]) }) +} + +fn parse_delivered_shutdown_signal(buf: [u8; 12]) -> ShutdownWaitReason { + ShutdownWaitReason::Signal { + signum: i32::from_ne_bytes(buf[0..4].try_into().expect("signum bytes")), + sender_pid: u32::from_ne_bytes(buf[4..8].try_into().expect("sender bytes")), + si_code: i32::from_ne_bytes(buf[8..12].try_into().expect("si_code bytes")), + } +} + +fn shutdown_signal_sender_comm(sender_pid: u32) -> String { + if sender_pid == 0 { + return "unknown".to_string(); + } + let output = process::Command::new("ps") + .args(["-p", &sender_pid.to_string(), "-o", "comm="]) + .output(); + match output { + Ok(output) if output.status.success() => { + let comm = String::from_utf8_lossy(&output.stdout).trim().to_string(); + if comm.is_empty() { + "unknown".to_string() + } else { + comm } - return; } + _ => "unknown".to_string(), } - let _ = tokio::signal::ctrl_c().await; } -async fn wait_for_shutdown_or_transport_expiry(transport: Option<&VzTransportLaunch>) { - let Some(transport) = transport else { - wait_for_shutdown_signal().await; - return; +async fn wait_for_shutdown_signal() -> ShutdownWaitReason { + let read_fd = match install_shutdown_signal_reporter() { + Ok(fd) => fd, + Err(_) => { + let _ = tokio::signal::ctrl_c().await; + return ShutdownWaitReason::Signal { + signum: libc::SIGINT, + sender_pid: 0, + si_code: 0, + }; + } }; + let afd = match tokio::io::unix::AsyncFd::new(read_fd) { + Ok(afd) => afd, + Err(_) => { + return ShutdownWaitReason::Signal { + signum: 0, + sender_pid: 0, + si_code: 0, + }; + } + }; + loop { + let mut ready = match afd.readable().await { + Ok(ready) => ready, + Err(_) => { + return ShutdownWaitReason::Signal { + signum: 0, + sender_pid: 0, + si_code: 0, + }; + } + }; + let mut buf = [0u8; 12]; + let read = unsafe { libc::read(afd.as_raw_fd(), buf.as_mut_ptr().cast(), buf.len()) }; + ready.clear_ready(); + if read == 12 { + return parse_delivered_shutdown_signal(buf); + } + if read < 0 { + let error = std::io::Error::last_os_error(); + if error.kind() == ErrorKind::WouldBlock { + continue; + } + } + return ShutdownWaitReason::Signal { + signum: 0, + sender_pid: 0, + si_code: 0, + }; + } +} + +async fn wait_until_guest_stopped(provider: Arc, handle: CapsuleHandle) -> String { + provider.wait_for_guest_exit(&handle).await +} + +async fn wait_for_shutdown_or_transport_expiry( + transport: Option<&VzTransportLaunch>, + provider: Option>, + handle: Option, +) -> ShutdownWaitReason { let expires_at = transport - .authority - .get("expires_at_unix_ms") + .and_then(|value| value.authority.get("expires_at_unix_ms")) .and_then(Value::as_u64) .unwrap_or(0); - let remaining = current_unix_millis() - .ok() - .and_then(|now| expires_at.checked_sub(now)) - .unwrap_or(0); - tokio::select! { - _ = wait_for_shutdown_signal() => {} - _ = tokio::time::sleep(Duration::from_millis(remaining)) => {} + let now = current_unix_millis().unwrap_or(0); + let remaining = if transport.is_some() { + transport_remaining_ms(expires_at, now) + } else { + u64::MAX + }; + if let (Some(provider), Some(handle)) = (provider, handle) { + tokio::select! { + reason = wait_for_shutdown_signal() => reason, + _ = tokio::time::sleep(Duration::from_millis(remaining)) => { + ShutdownWaitReason::TransportExpired { + expires_at_unix_ms: expires_at, + now_unix_ms: now, + } + } + state = wait_until_guest_stopped(provider, handle) => { + ShutdownWaitReason::GuestStopped { state } + } + } + } else { + tokio::select! { + reason = wait_for_shutdown_signal() => reason, + _ = tokio::time::sleep(Duration::from_millis(remaining)) => { + ShutdownWaitReason::TransportExpired { + expires_at_unix_ms: expires_at, + now_unix_ms: now, + } + } + } } } @@ -4141,6 +4711,82 @@ mod tests { assert!(serde_json::from_str::(&error).is_err()); } + #[tokio::test(flavor = "current_thread")] + async fn shutdown_wait_does_not_return_signal_without_a_delivered_signal() { + // A current-thread timeout around wait_for_shutdown_signal() never + // fires: the signal waiter parks on AsyncFd and the no-transport + // branch sleeps u64::MAX. Expire the transport instead. + let mut transport = transport_fixture('w'); + transport.authority["expires_at_unix_ms"] = json!(current_unix_millis().unwrap() + 50); + let outcome = wait_for_shutdown_or_transport_expiry(Some(&transport), None, None).await; + assert!( + matches!(outcome, ShutdownWaitReason::TransportExpired { .. }), + "shutdown wait invented a signal: {outcome:?}" + ); + } + + #[tokio::test(flavor = "current_thread")] + async fn owned_turn_port_absence_waits_for_released_relay_udp() { + let listen = std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let listen_port = listen.local_addr().unwrap().port(); + drop(listen); + let holder = std::net::UdpSocket::bind(("127.0.0.1", 0)).unwrap(); + let relay_port = holder.local_addr().unwrap().port(); + let tcp_holder = std::net::TcpListener::bind(("127.0.0.1", relay_port)).ok(); + let mut transport = transport_fixture('p'); + transport.authority["turn"]["listen_host"] = json!("127.0.0.1"); + transport.authority["turn"]["listen_port"] = json!(listen_port); + transport.authority["turn"]["relay_host"] = json!("127.0.0.1"); + transport.authority["turn"]["relay_port_min"] = json!(relay_port); + transport.authority["turn"]["relay_port_max"] = json!(relay_port); + let release = tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(80)).await; + drop(holder); + drop(tcp_holder); + }); + let (listener, relays) = wait_for_owned_turn_ports_absent(&transport).await; + release.await.unwrap(); + assert!(listener, "owned TURN listen port stayed occupied"); + assert!( + relays, + "owned TURN relay port stayed occupied after release" + ); + } + + #[test] + fn leftover_transport_remaining_ms_stays_live_after_launch_ready() { + assert_eq!( + transport_remaining_ms(1_789_458_477_000, 1_789_457_887_000), + 590_000 + ); + assert_eq!(transport_remaining_ms(1_000, 2_000), 0); + } + + #[test] + fn shutdown_wait_reason_names_signal_expiry_and_guest_stop() { + assert_eq!( + format_shutdown_wait_reason(&ShutdownWaitReason::Signal { + signum: 15, + sender_pid: 64094, + si_code: 0, + }), + "browser-vz-engine-supervisor stage=shutdown_wait reason=signal signum=15 sender_pid=64094 si_code=0" + ); + assert_eq!( + format_shutdown_wait_reason(&ShutdownWaitReason::TransportExpired { + expires_at_unix_ms: 1_789_458_477_000, + now_unix_ms: 1_789_457_887_000, + }), + "browser-vz-engine-supervisor stage=shutdown_wait reason=transport_expired expires_at_unix_ms=1789458477000 now_unix_ms=1789457887000" + ); + assert_eq!( + format_shutdown_wait_reason(&ShutdownWaitReason::GuestStopped { + state: "Error".to_string(), + }), + "browser-vz-engine-supervisor stage=shutdown_wait reason=guest_stopped state=Error" + ); + } + #[test] fn turn_log_diagnostics_classify_outcomes_without_retaining_log_content() { assert_eq!( @@ -4823,6 +5469,181 @@ mod tests { assert_eq!(parse_http_json_response(response.as_bytes()).unwrap(), logs); } + #[test] + fn guest_profile_disk_flush_requires_unmount_receipt() { + assert!(guest_profile_disk_flush_accepted(&json!({ + "schema": "elastos.browser.selkies-control.shutdown/v1", + "ok": true, + "profile_disk_flushed": true, + "profile_disk_unmounted": true, + }))); + assert!(!guest_profile_disk_flush_accepted(&json!({ + "schema": "elastos.browser.selkies-control.shutdown/v1", + "ok": true, + }))); + assert!(!guest_profile_disk_flush_accepted(&json!({ + "schema": "elastos.browser.selkies-control.shutdown/v1", + "ok": true, + "profile_disk_flushed": true, + "profile_disk_unmounted": true, + "leftover_writers_killed": true, + }))); + } + + #[test] + fn failed_profile_flush_keeps_truthful_child_absence_and_failed_durability() { + let absence = json!({ + "child_absent": true, + "supervisor_child_absent": true, + "control_socket_absent": true, + "route_absent": true, + "turn_listener_absent": true, + "turn_relay_ports_absent": true, + "ordinary_stream_bridge_absent": true, + "media_stream_bridge_absent": true, + "session_directory_absent": true, + "vm_absent": true, + }); + let terminal = absence + .as_object() + .is_some_and(|values| values.values().all(|value| value == &Value::Bool(true))); + assert!(terminal); + let mut settlement = json!({ + "schema": VZ_LAUNCH_SETTLEMENT_SCHEMA, + "state": "terminal_post_effect_cleanup", + "message": "Browser VZ guest shutdown did not flush the profile disk", + "absence": absence, + }); + settlement["profile_durability"] = json!("failed"); + assert_eq!(settlement["absence"]["child_absent"], true); + assert_eq!(settlement["profile_durability"], "failed"); + assert_eq!(settlement["state"], "terminal_post_effect_cleanup"); + } + + #[test] + fn accepted_profile_flush_keeps_truthful_child_absence_and_proved_durability() { + let absence = json!({ + "child_absent": true, + "supervisor_child_absent": true, + "control_socket_absent": true, + "route_absent": true, + "turn_listener_absent": true, + "turn_relay_ports_absent": true, + "ordinary_stream_bridge_absent": true, + "media_stream_bridge_absent": true, + "session_directory_absent": true, + "vm_absent": true, + }); + let terminal = absence + .as_object() + .is_some_and(|values| values.values().all(|value| value == &Value::Bool(true))); + assert!(terminal); + let mut settlement = json!({ + "schema": VZ_LAUNCH_SETTLEMENT_SCHEMA, + "state": "terminal_post_effect_cleanup", + "message": "Browser VZ guest shutdown flushed the profile disk", + "absence": absence, + }); + settlement["profile_durability"] = json!("proved"); + assert_eq!(settlement["absence"]["child_absent"], true); + assert_eq!(settlement["profile_durability"], "proved"); + assert_eq!(settlement["state"], "terminal_post_effect_cleanup"); + } + + #[test] + fn guest_profile_disk_flush_without_error_keeps_disk_fields() { + assert_eq!( + format_guest_profile_flush(&json!({ + "ok": false, + "profile_disk_flushed": false, + "profile_disk_unmounted": false, + "leftover_writers_killed": false, + })), + "ok=false profile_disk_flushed=false profile_disk_unmounted=false leftover_writers_killed=false" + ); + } + + #[test] + fn guest_profile_disk_flush_unmount_receipt_keeps_error_text() { + assert_eq!( + format_guest_profile_flush(&json!({ + "ok": false, + "error": "Browser VZ guest shutdown did not flush the profile disk", + })), + "ok=false profile_disk_flushed=false profile_disk_unmounted=false leftover_writers_killed=false error=Browser VZ guest shutdown did not flush the profile disk" + ); + } + + #[test] + fn overlay_guest_control_fails_closed_without_debugfs() { + let _guard = ENV_LOCK.lock().unwrap(); + let _debugfs = EnvVarRestore::capture("ELASTOS_DEBUGFS_BIN"); + let _source = EnvVarRestore::capture("ELASTOS_BROWSER_SELKIES_CONTROL_SERVICE"); + let tmp = tempfile::tempdir().unwrap(); + let source = tmp.path().join("browser-selkies-control-service.mjs"); + fs::write( + &source, + b"export function flushAndUnmountBrowserProfileDisk() {}\n", + ) + .unwrap(); + let rootfs = tmp.path().join("rootfs.ext4"); + fs::write(&rootfs, b"rootfs").unwrap(); + std::env::set_var("ELASTOS_DEBUGFS_BIN", tmp.path().join("absent-debugfs")); + std::env::set_var("ELASTOS_BROWSER_SELKIES_CONTROL_SERVICE", &source); + let error = overlay_guest_selkies_control_service(&rootfs).unwrap_err(); + assert!( + error.contains("ELASTOS_DEBUGFS_BIN is not a file"), + "{error}" + ); + } + + #[test] + fn overlay_webrtc_send_is_noop_without_opt_in() { + let _guard = ENV_LOCK.lock().unwrap(); + let _overlay = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY"); + let _gst = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC"); + let _start = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START"); + std::env::remove_var("ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY"); + let tmp = tempfile::tempdir().unwrap(); + let gst = tmp.path().join("gstwebrtc.py"); + let start = tmp.path().join("selkies-start.sh"); + fs::write(&gst, b"guest-gst").unwrap(); + fs::write(&start, b"guest-start").unwrap(); + std::env::set_var("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC", &gst); + std::env::set_var("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START", &start); + let rootfs = tmp.path().join("rootfs.ext4"); + fs::write(&rootfs, b"rootfs").unwrap(); + overlay_guest_webrtc_send_diagnostic(&rootfs).unwrap(); + } + + #[test] + fn overlay_webrtc_send_fails_closed_without_bound_hashes() { + let _guard = ENV_LOCK.lock().unwrap(); + let _overlay = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY"); + let _gst = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC"); + let _start = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START"); + let _gst_hash = EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC_SHA256"); + let _start_hash = + EnvVarRestore::capture("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START_SHA256"); + std::env::set_var("ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY", "1"); + let tmp = tempfile::tempdir().unwrap(); + let gst = tmp.path().join("gstwebrtc.py"); + let start = tmp.path().join("selkies-start.sh"); + fs::write(&gst, b"guest-gst").unwrap(); + fs::write(&start, b"guest-start").unwrap(); + std::env::set_var("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC", &gst); + std::env::set_var("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START", &start); + std::env::remove_var("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC_SHA256"); + std::env::remove_var("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START_SHA256"); + let rootfs = tmp.path().join("rootfs.ext4"); + fs::write(&rootfs, b"rootfs").unwrap(); + let error = overlay_guest_webrtc_send_diagnostic(&rootfs).unwrap_err(); + assert!( + error.contains("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC_SHA256"), + "{error}" + ); + } + #[test] fn guest_control_response_timeout_has_clear_error() { let (client, _server) = UnixStream::pair().unwrap(); diff --git a/elastos/crates/elastos-vz/src/ffi/delegate.rs b/elastos/crates/elastos-vz/src/ffi/delegate.rs index 08f331508..b5c216646 100644 --- a/elastos/crates/elastos-vz/src/ffi/delegate.rs +++ b/elastos/crates/elastos-vz/src/ffi/delegate.rs @@ -72,6 +72,30 @@ pub(crate) enum DelegateExit { ForcedAfterTimeout, } +fn truncate_delegate_exit_message(message: &mut String, max_bytes: usize) { + if message.len() <= max_bytes { + return; + } + let mut end = max_bytes; + while end > 0 && !message.is_char_boundary(end) { + end -= 1; + } + message.truncate(end); +} + +pub(crate) fn format_delegate_exit_reason(exit: &DelegateExit) -> String { + match exit { + DelegateExit::GuestCleanStop => "guest_clean_stop".to_string(), + DelegateExit::HostInitiatedStop => "host_initiated_stop".to_string(), + DelegateExit::ForcedAfterTimeout => "forced_after_timeout".to_string(), + DelegateExit::StoppedWithError(message) => { + let mut message = message.replace(['\n', '\r'], " "); + truncate_delegate_exit_message(&mut message, 512); + format!("stopped_with_error {message}") + } + } +} + /// Shared exit-signal handle — held by both /// [`ElastosVzDelegate`]'s ivars and the /// [`VzMachineHandle`][super::lifecycle::VzMachineHandle]'s @@ -151,6 +175,11 @@ impl ElastosVzDelegate { /// and become no-ops at the channel level. fn signal_exit(&self, exit: DelegateExit) { let ivars = self.ivars(); + let reason = format_delegate_exit_reason(&exit); + eprintln!( + "browser-vz-engine-supervisor stage=vz_delegate_exit vm_id={} reason={reason}", + ivars.vm_id + ); tracing::info!( target: "vz-delegate", vm_id = %ivars.vm_id, @@ -207,4 +236,28 @@ mod tests { "sender must be consumed after first signal" ); } + + #[test] + fn format_delegate_exit_reason_keeps_apple_error_on_one_line() { + assert_eq!( + format_delegate_exit_reason(&DelegateExit::GuestCleanStop), + "guest_clean_stop" + ); + assert_eq!( + format_delegate_exit_reason(&DelegateExit::StoppedWithError( + "VZError\nkernel panic\rcode=1".into() + )), + "stopped_with_error VZError kernel panic code=1" + ); + } + + #[test] + fn format_delegate_exit_reason_truncates_before_multibyte_char() { + let message = format!("{}é", "a".repeat(511)); + assert_eq!(message.len(), 513); + assert_eq!( + format_delegate_exit_reason(&DelegateExit::StoppedWithError(message)), + format!("stopped_with_error {}", "a".repeat(511)) + ); + } } diff --git a/elastos/crates/elastos-vz/src/ffi/lifecycle.rs b/elastos/crates/elastos-vz/src/ffi/lifecycle.rs index ecf31e3db..ffde144ad 100644 --- a/elastos/crates/elastos-vz/src/ffi/lifecycle.rs +++ b/elastos/crates/elastos-vz/src/ffi/lifecycle.rs @@ -478,14 +478,23 @@ impl VzMachineHandle { /// /// The receiver is consumed on first call; subsequent calls /// return a typed error rather than block indefinitely. + pub(crate) fn take_exit_receiver( + &self, + ) -> Result, String> { + self.exit_rx + .lock() + .expect("exit_rx mutex") + .take() + .ok_or_else(|| { + format!( + "vz wait_for_exit (vm_id='{}'): receiver already consumed", + self.vm_id + ) + }) + } + pub(crate) async fn wait_for_exit_classified(&self) -> Result { - let rx = self.exit_rx.lock().expect("exit_rx mutex").take(); - let Some(rx) = rx else { - return Err(format!( - "vz wait_for_exit (vm_id='{}'): receiver already consumed", - self.vm_id - )); - }; + let rx = self.take_exit_receiver()?; match rx.await { Ok(exit) => Ok(delegate_exit_to_reason(exit)), Err(_) => Err(format!( diff --git a/elastos/crates/elastos-vz/src/provider.rs b/elastos/crates/elastos-vz/src/provider.rs index 968b9bd88..6ee460417 100644 --- a/elastos/crates/elastos-vz/src/provider.rs +++ b/elastos/crates/elastos-vz/src/provider.rs @@ -66,6 +66,44 @@ impl VzProvider { Self::new(VzConfig::default()) } + pub async fn guest_is_running(&self, handle: &CapsuleHandle) -> bool { + let vms = self.vms.read().await; + vms.get(&handle.id).is_some_and(|vm| vm.is_running()) + } + + pub async fn guest_state_label(&self, handle: &CapsuleHandle) -> String { + let vms = self.vms.read().await; + match vms.get(&handle.id) { + None => "absent".to_string(), + Some(vm) => vm.guest_state_label(), + } + } + + pub async fn wait_for_guest_exit(&self, handle: &CapsuleHandle) -> String { + #[cfg(target_os = "macos")] + { + let receiver = { + let vms = self.vms.read().await; + match vms.get(&handle.id) { + None => return "absent".to_string(), + Some(vm) => vm.take_guest_exit_receiver(), + } + }; + if let Some(rx) = receiver { + return match rx.await { + Ok(exit) => crate::ffi::delegate::format_delegate_exit_reason(&exit), + Err(_) => "delegate_sender_dropped".to_string(), + }; + } + } + loop { + if !self.guest_is_running(handle).await { + return self.guest_state_label(handle).await; + } + tokio::time::sleep(std::time::Duration::from_millis(250)).await; + } + } + /// Initialise on-disk state directories. pub async fn init(&self) -> Result<()> { tokio::fs::create_dir_all(&self.config.state_dir) diff --git a/elastos/crates/elastos-vz/src/vm.rs b/elastos/crates/elastos-vz/src/vm.rs index cc89cfa6c..132b28255 100644 --- a/elastos/crates/elastos-vz/src/vm.rs +++ b/elastos/crates/elastos-vz/src/vm.rs @@ -145,6 +145,23 @@ impl RunningVm { self.last_vz_error.as_ref() } + pub(crate) fn guest_state_label(&self) -> String { + #[cfg(target_os = "macos")] + if let Some(machine) = self.handle.as_ref() { + return format!("{:?}", machine.current_state()); + } + format!("{:?}", self.status) + } + + #[cfg(target_os = "macos")] + pub(crate) fn take_guest_exit_receiver( + &self, + ) -> Option> { + self.handle + .as_ref() + .and_then(|machine| machine.take_exit_receiver().ok()) + } + /// Typed Vz exit reason from the most recent terminal /// observation (macOS only). Returns `None` before any stop /// or `wait_for_exit_code` call. diff --git a/scripts/browser-display-attachment.test.mjs b/scripts/browser-display-attachment.test.mjs index 4c09607a4..c4e0d31c3 100644 --- a/scripts/browser-display-attachment.test.mjs +++ b/scripts/browser-display-attachment.test.mjs @@ -8,6 +8,8 @@ import { SelkiesPage, MinimalWebSocketClient } from './browser-selkies-control-s const id = 'a'.repeat(32), otherId = 'b'.repeat(32); const videoSdp = 'v=0\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\n'; const audioSdp = 'v=0\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\n'; +const relayIce = { candidate: 'candidate:1 1 UDP 2130706431 203.0.113.5 3478 typ relay', sdpMLineIndex: 0 }; +const hostIce = { candidate: 'candidate:2 1 UDP 2130706430 192.0.2.8 9 typ host', sdpMLineIndex: 0 }; function deferred() { let resolve, reject; const promise = new Promise((a, b) => { resolve = a; reject = b; }); return { promise, resolve, reject }; } function fixture() { const calls = [], closed = []; @@ -106,19 +108,123 @@ for (const [video, audio] of [['raw_json', 'peer_routed'], ['peer_routed', 'raw_ }); } +test('reload reuse asks the live producer for a new offer and does not send SESSION', async () => { + const f = fixture(); + delete f.page.openLegacySelkiesSession; + delete f.page.openLegacySelkiesAudioSession; + f.page.serverPeerId = '1'; + f.page.audioServerPeerId = '2'; + f.page.ws.legacyHelloAccepted = true; + f.page.audioWs.legacyHelloAccepted = true; + f.page.ws.closed = false; + f.page.audioWs.closed = false; + Object.assign(f.page.config, { signalTimeoutMs: 1000 }); + const pending = f.page.signal(f.request); + await nextTurn(); + assert.deepEqual( + f.calls.filter((entry) => entry[0] === 'send').sort((a, b) => a[1].localeCompare(b[1])), + [ + ['send', 'audio', JSON.stringify({ elastos_display_renegotiate: true })], + ['send', 'video', JSON.stringify({ elastos_display_renegotiate: true })], + ], + ); + assert.equal(f.calls.some((entry) => String(entry[2] || '').includes('SESSION')), false); + f.page.handleMessage(JSON.stringify({ sdp: { type: 'offer', sdp: videoSdp }, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ sdp: { type: 'offer', sdp: audioSdp }, from: '2' })); + f.page.handleMessage(JSON.stringify({ ice: hostIce, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ ice: hostIce, from: '2' })); + f.page.handleMessage(JSON.stringify({ ice: relayIce, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ ice: relayIce, from: '2' })); + const result = await pending; + assert.equal(result.initial_offer.sdp, videoSdp); + assert.equal(result.audio_offer.sdp, audioSdp); + assert.deepEqual(result.initial_offer.candidates, [relayIce]); + assert.deepEqual(result.audio_offer.candidates, [relayIce]); + assert.equal(f.page.ws.closed, false); + assert.equal(f.page.audioWs.closed, false); + assert.deepEqual(f.closed, []); +}); + +test('reload reuse rebinds a later video offer after the first pair is already taken', async () => { + const f = fixture(); + const laterVideo = 'v=0\r\nm=video 9 UDP/TLS/RTP/SAVPF 97\r\n'; + const laterRelay = { candidate: 'candidate:9 1 UDP 2130706431 203.0.113.9 3478 typ relay', sdpMLineIndex: 0 }; + delete f.page.openLegacySelkiesSession; + delete f.page.openLegacySelkiesAudioSession; + f.page.serverPeerId = '1'; + f.page.audioServerPeerId = '2'; + f.page.ws.legacyHelloAccepted = true; + f.page.audioWs.legacyHelloAccepted = true; + f.page.ws.closed = false; + f.page.audioWs.closed = false; + Object.assign(f.page.config, { signalTimeoutMs: 1000 }); + const pending = f.page.signal(f.request); + await nextTurn(); + f.page.handleMessage(JSON.stringify({ sdp: { type: 'offer', sdp: videoSdp }, from: '1' })); + f.page.handleMessage(JSON.stringify({ ice: relayIce, from: '1' })); + await nextTurn(); + f.page.handleMessage(JSON.stringify({ sdp: { type: 'offer', sdp: laterVideo }, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ sdp: { type: 'offer', sdp: audioSdp }, from: '2' })); + f.page.handleAudioMessage(JSON.stringify({ ice: relayIce, from: '2' })); + f.page.handleMessage(JSON.stringify({ ice: laterRelay, from: '1' })); + const result = await pending; + assert.equal(result.initial_offer.sdp, laterVideo); + assert.deepEqual(result.initial_offer.candidates, [laterRelay]); + assert.equal(result.audio_offer.sdp, audioSdp); + assert.deepEqual(result.audio_offer.candidates, [relayIce]); + assert.deepEqual(f.page.ack("answer").candidates, [laterRelay]); +}); + +test('reload reuse keeps the later offer and only publishes relay ICE', async () => { + const f = fixture(); + const laterVideo = 'v=0\r\nm=video 9 UDP/TLS/RTP/SAVPF 97\r\n'; + const laterAudio = 'v=0\r\nm=audio 9 UDP/TLS/RTP/SAVPF 112\r\n'; + delete f.page.openLegacySelkiesSession; + delete f.page.openLegacySelkiesAudioSession; + f.page.serverPeerId = '1'; + f.page.audioServerPeerId = '2'; + f.page.ws.legacyHelloAccepted = true; + f.page.audioWs.legacyHelloAccepted = true; + f.page.ws.closed = false; + f.page.audioWs.closed = false; + Object.assign(f.page.config, { signalTimeoutMs: 1000 }); + const pending = f.page.signal(f.request); + await nextTurn(); + f.page.handleMessage(JSON.stringify({ sdp: { type: 'offer', sdp: videoSdp }, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ sdp: { type: 'offer', sdp: audioSdp }, from: '2' })); + f.page.handleMessage(JSON.stringify({ ice: hostIce, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ ice: hostIce, from: '2' })); + f.page.handleMessage(JSON.stringify({ sdp: { type: 'offer', sdp: laterVideo }, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ sdp: { type: 'offer', sdp: laterAudio }, from: '2' })); + f.page.handleMessage(JSON.stringify({ ice: relayIce, from: '1' })); + f.page.handleAudioMessage(JSON.stringify({ ice: relayIce, from: '2' })); + const result = await pending; + assert.equal(result.initial_offer.sdp, laterVideo); + assert.equal(result.audio_offer.sdp, laterAudio); + assert.deepEqual(result.initial_offer.candidates, [relayIce]); + assert.deepEqual(result.audio_offer.candidates, [relayIce]); + assert.deepEqual(f.page.ack("answer").candidates, [relayIce]); + assert.deepEqual(f.page.ackAudio("answer").candidates, [relayIce]); +}); + test('paired attachment preserves the page and returns only fresh display offers', async () => { const f = fixture(), browserPage = f.page.browserPage, oldVideo = f.page.ws, oldAudio = f.page.audioWs; + f.page.serverPeerId = '1'; + f.page.audioServerPeerId = '2'; + oldVideo.legacyHelloAccepted = true; + oldAudio.legacyHelloAccepted = true; f.page.remoteCandidateHistory.push({ candidate: 'old-video' }); f.page.audioRemoteCandidateHistory.push({ candidate: 'old-audio' }); const result = await f.page.signal(f.request); - assert.deepEqual(f.calls, [['close', 'video'], ['close', 'audio'], ['open', 'video'], ['open', 'audio']]); + assert.deepEqual(f.calls, [['open', 'video'], ['open', 'audio']]); + assert.equal(f.page.ws, oldVideo); + assert.equal(f.page.audioWs, oldAudio); assert.notEqual(result.display_generation, f.generation); assert.equal(result.previous_display_generation, f.generation); assert.equal(result.page_id, 'page:test'); assert.equal(result.request_id, id); assert.deepEqual(result.initial_offer.candidates, []); assert.deepEqual(result.audio_offer.candidates, []); assert.deepEqual(Object.keys(result).sort(), ['schema', 'page_id', 'request_id', 'previous_display_generation', 'display_generation', 'initial_offer', 'audio_offer'].sort()); assert.equal(f.page.browserPage, browserPage); assert.equal(f.page.closed, false); assert.deepEqual(f.closed, []); - oldVideo.close(); oldAudio.close(); assert.equal(f.page.displayAvailable, true); assert.equal(f.page.videoClosed, false); assert.equal(f.page.audioClosed, false); }); @@ -297,7 +403,25 @@ async function legacyBroker(t, { releaseMs = 120, rejectionReason = 'invalid pee for (let i = 0; i < payload.length; i++) payload[i] ^= mask[i % 4]; if (opcode === 8) { socket.end(frame(8, Buffer.alloc(0))); return; } if (opcode !== 1) continue; - const words = payload.toString().split(' '); assert.equal(words[0], 'HELLO'); + const text = payload.toString(); + if (text.startsWith('{')) { + const data = JSON.parse(text); + assert.equal(data.elastos_display_renegotiate, true); + if (!uid) continue; + const sdp = uid === '1' ? videoSdp : audioSdp; + socket.write(frame(1, JSON.stringify({ sdp: { type: 'offer', sdp } }))); + socket.write(frame(1, JSON.stringify({ ice: relayIce }))); + continue; + } + const words = text.split(' '); + if (words[0] === 'SESSION') { + if (!uid) continue; + const sdp = uid === '1' ? videoSdp : audioSdp; + socket.write(frame(1, JSON.stringify({ sdp: { type: 'offer', sdp } }))); + socket.write(frame(1, JSON.stringify({ ice: relayIce }))); + continue; + } + assert.equal(words[0], 'HELLO'); const requested = words[1]; if (peers.has(requested)) { rejected.push(requested); @@ -331,22 +455,28 @@ async function actualLegacyPage(t, broker) { test('actual legacy broker keeps retired viewer IDs until producer teardown; attachment retries only UID rejection', async t => { const broker = await legacyBroker(t, { releaseMs: { '1': 120, '3': 320 } }), f = await actualLegacyPage(t, broker); assert.deepEqual(broker.accepted, ['1', '3']); + const liveVideo = f.page.ws, liveAudio = f.page.audioWs; const result = await f.page.signal(f.request); - assert.ok(broker.rejected.includes('1')); - assert.ok(broker.rejected.includes('3')); - assert.equal(broker.accepted.length, 4); - assert.deepEqual(broker.accepted.slice(2).sort(), ['1', '3']); + assert.deepEqual(broker.rejected, []); + assert.deepEqual(broker.accepted, ['1', '3']); + assert.equal(f.page.ws, liveVideo); + assert.equal(f.page.audioWs, liveAudio); assert.equal(f.page.closed, false); assert.equal(f.page.displayAvailable, true); assert.equal(result.initial_offer.type, 'offer'); assert.equal(result.audio_offer.type, 'offer'); assert.notEqual(result.display_generation, f.generation); const next = await f.page.signal({ ...f.request, request_id: otherId, display_generation: result.display_generation }); assert.notEqual(next.display_generation, result.display_generation); - assert.equal(broker.accepted.length, 6); - assert.deepEqual(broker.accepted.slice(4).sort(), ['1', '3']); + assert.deepEqual(broker.accepted, ['1', '3']); + assert.equal(f.page.ws, liveVideo); + assert.equal(f.page.audioWs, liveAudio); }); test('UID error after HELLO acknowledgment is terminal even when its code and reason match', async t => { const broker = await legacyBroker(t, { acknowledgeRejected: true }), f = await actualLegacyPage(t, broker); + f.page.serverPeerId = null; + f.page.audioServerPeerId = null; + delete f.page.ws.legacyHelloAccepted; + delete f.page.audioWs.legacyHelloAccepted; await assert.rejects(f.page.signal(f.request), { code: 'display_attach_failed' }); assert.ok(broker.rejected.length > 0); assert.equal(new Set(broker.rejected).size, broker.rejected.length, 'neither channel may retry after HELLO'); @@ -354,6 +484,10 @@ test('UID error after HELLO acknowledgment is terminal even when its code and re test('actual broker protocol rejection other than the exact UID release response remains terminal', async t => { const broker = await legacyBroker(t, { rejectionReason: 'invalid protocol' }), f = await actualLegacyPage(t, broker); + f.page.serverPeerId = null; + f.page.audioServerPeerId = null; + delete f.page.ws.legacyHelloAccepted; + delete f.page.audioWs.legacyHelloAccepted; await assert.rejects(f.page.signal(f.request), { code: 'display_attach_failed' }); assert.ok(broker.rejected.length > 0); assert.equal(new Set(broker.rejected).size, broker.rejected.length, 'neither channel may retry another protocol error'); diff --git a/scripts/browser-display-metrics-freshness.test.mjs b/scripts/browser-display-metrics-freshness.test.mjs index 4db880d2e..3a86a8f7d 100644 --- a/scripts/browser-display-metrics-freshness.test.mjs +++ b/scripts/browser-display-metrics-freshness.test.mjs @@ -39,7 +39,7 @@ function fixture({ debugMetrics = true, audio = true } = {}) { const peers = [], timers = new Map(), updates = [], recoveries = []; const state = { active: true, owner: 1, fetch: null }; class Peer extends Events { - connectionState = 'new'; iceConnectionState = 'new'; signalingState = 'stable'; + connectionState = 'new'; iceConnectionState = 'new'; iceGatheringState = 'complete'; signalingState = 'stable'; reads = 0; bytes = 100; frames = 10; response = null; constructor() { super(); this.kind = peers.length % (audio ? 2 : 1) ? 'audio' : 'video'; peers.push(this); } addTransceiver() {} @@ -63,6 +63,7 @@ function fixture({ debugMetrics = true, audio = true } = {}) { iceCandidateType: () => 'relay', normalizeDisplayIceServers: value => value || [], normalizeEngineCandidate: value => value, normalizeIceCandidateForRuntime: value => value, sdpHasOnlyRelayCandidates: () => true, stripTrickleCandidatesFromSdp: value => value, + waitForLocalAnswerIce: async () => {}, validateRuntimeLaunchTurn: async () => [], }); vm.runInContext(statusSource.slice(statusSource.indexOf('export async function collectWebrtcStats'), statusSource.indexOf('export function browserMetricsText')).replace('export async function', 'async function'), context); @@ -207,19 +208,21 @@ for (const outcome of ['success', 'failure']) test(`old query ${outcome} cannot assert.deepEqual(f.recoveries, []); f.controller.close(); }); -test('audio peer appearing during a video-only query invalidates the captured pair', async () => { +test('refresh during in-flight answers keeps the live video and audio pair', async () => { const f = fixture(), answer = deferred(), started = deferred(), stats = deferred(); f.state.fetch = body => { if (!body.channel) { started.resolve(); return answer.promise; } return f.ack(body); }; const connecting = f.connect(); await started.promise; - assert.equal(f.peers.length, 1); f.peers[0].response = () => stats.promise; + assert.equal(f.peers.length, 2); f.peers[0].response = () => stats.promise; const refreshing = f.controller.refreshMetrics(); answer.resolve(f.ack({ type: 'answer' })); await connecting; assert.equal(f.peers.length, 2); - stats.resolve(statsReport('video', 100)); assert.equal(await refreshing, null); - assert.equal(f.controller.metricsState().latestWebrtcStats, null); + stats.resolve(statsReport('video', 100)); + const fresh = await refreshing; + assert.equal(fresh.latestWebrtcStats.video_bytes_received, 100); + assert.equal(fresh.latestWebrtcStats.audio_bytes_received, 100); f.peers[0].response = null; assert.equal((await f.controller.refreshMetrics()).latestWebrtcStats.audio_bytes_received, 100); f.controller.close(); diff --git a/scripts/browser-display-signaling-generation.test.mjs b/scripts/browser-display-signaling-generation.test.mjs index fedd1e290..3047c03a9 100644 --- a/scripts/browser-display-signaling-generation.test.mjs +++ b/scripts/browser-display-signaling-generation.test.mjs @@ -80,12 +80,19 @@ function controllerFixture({ audio = true, offerer = 'engine' } = {}) { const peers = [], calls = [], recovered = [], focused = [], timers = new Map(); const state = { supported: true, active: true, owner: 1, fetch: null, validation: async () => [], description: null }; class Peer extends Events { - connectionState = 'new'; iceConnectionState = 'new'; signalingState = 'stable'; + connectionState = 'new'; iceConnectionState = 'new'; iceGatheringState = 'complete'; signalingState = 'stable'; added = []; descriptions = []; constructor() { super(); peers.push(this); } addTransceiver() {} async setRemoteDescription(value) { this.descriptions.push(value); if (state.description) await state.description(this); } - async setLocalDescription(value) { this.localDescription = value; } + async setLocalDescription(value) { + this.localDescription = value; + if (state.localIce) { + for (const candidate of state.localIce) { + this.emit('icecandidate', { candidate: { toJSON: () => candidate } }); + } + } + } async createAnswer() { return { type: 'answer', sdp: 'v=0\r\ns=answer\r\n' }; } async createOffer() { return { type: 'offer', sdp: 'v=0\r\ns=offer\r\n' }; } async addIceCandidate(value) { this.added.push(value); } @@ -101,6 +108,7 @@ function controllerFixture({ audio = true, offerer = 'engine' } = {}) { normalizeDisplayIceServers: value => value || [], normalizeEngineCandidate: value => value, normalizeIceCandidateForRuntime: value => value, sdpHasOnlyRelayCandidates: () => true, stripTrickleCandidatesFromSdp: value => value, + waitForLocalAnswerIce: async () => {}, validateRuntimeLaunchTurn: (...args) => state.validation(...args), }); vm.runInContext(source.slice(source.indexOf('const WEBRTC_CONNECT_TIMEOUT_MS')).replace('export function createBrowserRemoteDisplay', 'function createBrowserRemoteDisplay'), context); @@ -163,7 +171,8 @@ for (const channel of ['video', 'audio']) for (const outcome of ['success', 'fai } return ack(body); }; - const poll = f.pollIds()[channel === 'video' ? 0 : 1]; f.timers.get(poll).callback(); await started.promise; + for (const id of f.pollIds()) f.timers.get(id).callback(); + await started.promise; await f.controller.connect(f.display('b'), {}); const polls = f.pollIds(); if (outcome === 'success') pending.resolve(f.ack({ type: 'end_of_candidates', display_generation: generation('a') })); else pending.reject(new Error('old poll failed')); @@ -271,10 +280,58 @@ for (const outcome of ['success', 'failure']) test(`closing retained owner stops if (outcome === 'success') pending.resolve(f.ack({ type: 'answer', display_generation: generation('a') })); else pending.reject(new Error('answer failed during close')); await connecting; - assert.equal(f.peers.length, 1); assert.equal(f.focused.length, 0); assert.equal(f.pollIds().length, 0); + assert.equal(f.peers.length, 2); assert.equal(f.focused.length, 0); assert.equal(f.pollIds().length, 0); assert.deepEqual(f.recovered, []); f.controller.close(); }); +for (const owner of ['current', 'replaced']) { + test(`queued local ICE send rejection recovers only the current owner: ${owner}`, async () => { + const f = controllerFixture({ audio: false }); + const pending = deferred(), started = deferred(); + const unhandled = []; + const onUnhandled = reason => unhandled.push(reason); + process.on('unhandledRejection', onUnhandled); + f.state.localIce = [ice('queued')]; + f.state.fetch = (body, ack) => { + if (body.display_generation === generation('a') && body.type === 'candidate') { + started.resolve(); + return pending.promise; + } + return ack(body); + }; + try { + const connecting = f.controller.connect(f.display('a'), {}); + await started.promise; + if (owner === 'replaced') await f.controller.connect(f.display('b'), {}); + pending.reject(new Error('queued ice failed')); + await connecting; + await flush(); + await flush(); + if (owner === 'current') { + assert.equal(f.recovered.length, 1); + assert.equal(f.recovered[0][0], 'queued ice failed'); + assert.equal(f.recovered[0][1].failureKind, 'signaling'); + } else { + assert.deepEqual(f.recovered, []); + } + assert.deepEqual(unhandled, []); + } finally { + process.off('unhandledRejection', onUnhandled); + f.controller.close(); + } + }); +} + +test('connect starts the audio peer before the first video frame', async () => { + const f = controllerFixture(); + await f.controller.connect(f.display('a'), {}); + await flush(); + await flush(); + assert.equal(f.peers.length, 2); + assert.equal(f.controller.isTrackReady(), false); + f.controller.close(); +}); + test('viewer activity guard is captured for the owner at connect', async () => { const f = controllerFixture(); await f.controller.connect(f.display('a'), {}); f.state.owner += 1; // Another owner is active; that does not authorize the existing peer. diff --git a/scripts/browser-entropy-check.mjs b/scripts/browser-entropy-check.mjs index db440d695..17d89ea7c 100644 --- a/scripts/browser-entropy-check.mjs +++ b/scripts/browser-entropy-check.mjs @@ -56,6 +56,7 @@ const browserManifest = read("capsules/browser/capsule.json"); const browserCapsuleManifest = JSON.parse(browserManifest); const browser = read("capsules/browser/browser/index.html"); const browserMain = read("capsules/browser/browser/browser.js"); +const browserRestoreBoot = read("capsules/browser/browser/browser-restore-boot.js"); const browserJs = readAll([ "capsules/browser/browser/browser.js", "capsules/browser/browser/browser-clipboard.js", @@ -225,9 +226,12 @@ const browserVmGuestControlBridge = read( const browserVmGuestControlBridgeSmoke = read( "scripts/browser-vm-guest-control-bridge-smoke.sh", ); -const browserVzEngineSupervisor = read( +const browserVzEngineSupervisor = readAll([ "elastos/crates/elastos-vz/src/bin/browser-vz-engine-supervisor.rs", -); + "elastos/crates/elastos-vz/src/ffi/delegate.rs", + "elastos/crates/elastos-vz/src/provider.rs", + "elastos/crates/elastos-vz/src/vm.rs", +]); const browserVmTargetStage = read("scripts/build/stage-browser-vm-target.sh"); const browserVmTargetStageSmoke = read( "scripts/build/stage-browser-vm-target-smoke.sh", @@ -390,11 +394,40 @@ assertNoForbidden(browserJs, "Browser UI", [ assert( browserJs.includes("normalizeUrl") && browserJs.includes("streamTargetForUrl") && + browserJs.includes("const requestedStartupUrl = params.get(\"url\")") && + browserJs.includes("if (!requestedStartupUrl)") && + !browserJs.includes("return requestRuntimeOpen(initialUrl, { history: \"replace\" });") && browserJs.includes("recoverableRuntimePage") && browserJs.includes("elastos.browser.cleanup-handle/v1") && browserJs.includes("elastos.browser.close-request/v2") && + browserJs.includes("the profile save failed") && + browserJs.includes("response.profile_durability") && !browserJs.includes("runtime_close") && !browserJs.includes("sessionStorage") && + !browserRestoreBoot.includes("sessionStorage") && + browser.includes("browser-restore-boot.js?v=browser-20260915c") && + browserRestoreBoot.includes("function waitForReadyAttach") && + browserRestoreBoot.includes("function attachFromKnownUrl") && + browser.includes('rel="modulepreload"') && + browserRestoreBoot.includes("__elastosBrowserRestoreBoot") && + browserRestoreBoot.includes('generation.slice("display:".length)') && + browserJs.includes("__elastosBrowserRestoreBoot") && + browserJs.includes("function startRecoverableDisplayAttach") && + browserJs.includes("keepalive: true") && + browserJs.includes('pending.state === "pending" || pending.state === "ready"') && + !browserJs.includes("prepareRecoverableDisplayAttach") && + browserJs.includes("const connecting = connectRemoteDisplay") && + browserRemoteDisplay.includes("const startAudioPeerAfterVideoIce = () => {") && + browserRemoteDisplay.includes("emitMediaDiagnostic(\"viewer_peer_created\", \"video\");\n startAudioPeerAfterVideoIce();") && + browserRemoteDisplay.includes("startAudioPeerAfterVideoIce();\n return;") && + !browserRemoteDisplay.includes("const audioConnecting = expectsAudio") && + !browserRemoteDisplay.includes("await audioConnecting") && + browserRemoteDisplay.includes("postAnswerWithQueuedLocalIce") && + !browserRemoteDisplay.includes("waitForLocalAnswerIce") && + browserRemoteDisplay.includes("const flushing = queued.splice(0).map((candidate) =>") && + browserRemoteDisplay.includes("signalOne(candidate).catch((error) => {") && + browserRemoteDisplay.includes("recoverPeerSignal(nextPeer, error)") && + browserRemoteDisplay.includes("void Promise.all(flushing)") && browserJs.includes("Only http and https addresses") && browserJs.includes("/api/apps/browser/open") && browserJs.includes("/api/apps/browser/summary") && @@ -530,6 +563,9 @@ assert( browserEngineAdapter.includes( "elastos.browser.engine-cleanup-result/v2", ) && + browserEngineAdapter.includes( + "ordinary_isolated_close_and_replay_keep_failed_profile_durability", + ) && browserEngineAdapter.includes("elastos.adapter-ipc/v1") && browserEngineAdapter.includes("runtime_stream_path") && browserEngineAdapter.includes("elastos.browser.engine.launch-request/v1") && @@ -1131,6 +1167,7 @@ assert( setupSourceHome.includes("browser-source-home-config.mjs") && setupSourceHome.includes("browser-vm-engine-supervisor.mjs") && setupSourceHome.includes("browser-vm-control-service.mjs") && + !setupSourceHome.includes("browser-vm-webrtc-apply-gate.mjs") && setupSourceHome.includes("browser-vm-remote-vz-launcher.mjs") && setupSourceHome.includes("browser-vm-local-crosvm-launcher.mjs") && setupSourceHome.includes("browser-vm-prepare-rootfs-pool.mjs") && @@ -1343,6 +1380,8 @@ assert( browserVmArtifactPreflightSmoke.includes("elastos.browser.vm-artifact-preflight-smoke/v1") && browserVmArtifactPreflightSmoke.includes("ELASTOS_BROWSER_VM_STAGED_ROOTFS") && browserVmControlService.includes("elastos.browser.vm-control-service.config/v1") && + !browserVmControlService.includes("createGuestWebrtcApplyGate") && + !browserVmControlService.includes("guestWebrtcApplyGate.apply") && browserVmControlService.includes("Browser VM control service accepts only chromium_microvm") && browserVmControlService.includes("Browser VM control service requires webrtc_remote_display") && browserVmControlService.includes('const expectedMediaTransport = "runtime_relay"') && @@ -1352,6 +1391,30 @@ assert( browserVmEngineSupervisor.includes("launcher: controlServiceArtifactFingerprints(config.launcher_program)") && browserVmControlService.includes("persistent_launcher") && browserVmControlService.includes("runPersistentProgram") && + browserVmControlService.includes("persistentLauncherReadyLine") && + browserVmControlService.includes( + 'parsed?.schema === "elastos.browser.engine.supervisor-result/v1"', + ) && + browserVmControlService.includes("settleDispatchedTransportLaunchWithoutBinding") && + browserVmControlService.includes("helperLogVzLaunchEffects") && + browserVmControlService.includes("launch_reconciliation_log_effects") && + browserVmControlService.includes( + "launchSettlement?.state === LAUNCH_SETTLEMENT_TERMINAL", + ) && + browserVmControlService.includes("? { terminal_record: true }") && + browserVmControlService.includes( + "record.terminal_cleanup_receipt === undefined ||", + ) && + browserVmControlService.includes("durableReceiptIsSafe") && + browserVmControlService.includes( + "if (current?.state === LAUNCH_SETTLEMENT_TERMINAL)", + ) && + browserVmControlService.includes("profile_durability") && + gatewayBrowserApi.includes("browser_profile_durability_from_receipt") && + gatewayBrowserRouteTests.includes( + "test_browser_ordinary_close_and_replay_keep_failed_profile_durability", + ) && + browserVmControlService.includes("terminateOwnedSupervisorPid") && browserVmControlService.includes("terminatePersistentLauncher") && !browserVmControlService.includes("sameLaunchIdentity") && !browserVmControlService.includes("launch_replacing") && @@ -1362,7 +1425,26 @@ assert( browserVmControlService.includes("single_active_page_non_reusable_profile") && browserVmControlService.includes("max_active_pages") && browserVmControlService.includes("Browser VM active page capacity reached") && + browserVmControlService.includes("vm_retirement_defers_guest_page_close") && + browserVmControlService.includes("supervisor_owns_guest_profile_flush") && browserVmControlService.includes("page_close_forced_vm_retirement") && + browserVmControlService.includes("exact owned native settlement arrives") && + browserVmControlService.includes("turnListenerPortAbsent") && + browserVmControlService.includes("turnRelayPortsAbsent") && + browserVmControlService.includes("reuseAddr: false") && + browserVmControlService.includes("termination?.graceful === true") && + browserVmControlService.includes("supervisorOwnedVsockBridgesAbsent") && + browserVmControlService.includes("ownedTurnDelayedTerminal") && + browserVmControlService.includes("vzLaunchNonPortAbsenceProved") && + browserVmControlService.includes("parseVzLaunchSettlementFromStderr") && + browserVmControlService.includes("delayed_turn_port_absence") && + !browserVmControlService.includes("const ordinaryBridgeAbsent = !fs.existsSync(") && + !browserVmControlService.includes("observedOwnerAbsent") && + !browserVmControlService.includes("restart_observation") && + !browserVmControlService.includes("function tcpEndpointAbsent") && + !browserVmControlService.includes( + "Browser VZ exact launcher did not produce a graceful terminal exit", + ) && browserVmControlService.includes("per_launch_vm_target") && browserVmControlService.includes("elastos.browser.vz-launch-settlement/v1") && browserVmControlService.includes("validateVzLaunchSettlementForLaunch") && @@ -1385,6 +1467,45 @@ assert( browserVmControlServiceSettlementSmoke.includes("verify-typed-restart") && browserVmControlServiceSettlementSmoke.includes("did_not_act cleanup_pending terminal_post_effect_cleanup") && browserVmControlServiceSettlementSmoke.includes("substituted transport settlement escaped cleanup ownership") && + browserVmControlServiceSettlementSmoke.includes("ungraceful-transport-close") && + browserVmControlServiceSettlementSmoke.includes( + "ungraceful post-exit close synthesized terminal TURN process absence", + ) && + browserVmControlServiceSettlementSmoke.includes( + "occupied UDP relay port did not retain cleanup ownership", + ) && + browserVmControlServiceSettlementSmoke.includes( + "Runtime stream sockets remaining after owned launcher cleanup", + ) && + browserVmControlServiceSettlementSmoke.includes("polluted-stdout-ready") && + browserVmControlServiceSettlementSmoke.includes("polluted-stdout-hold") && + browserVmControlServiceSettlementSmoke.includes("delayed-turn-child-present") && + browserVmControlServiceSettlementSmoke.includes( + "delayed turn with child_absent=false became terminal", + ) && + browserVmControlServiceSettlementSmoke.includes( + "verify-delayed-turn-child-present-restart", + ) && + browserVmControlServiceSettlementSmoke.includes("delayed-turn-port-only") && + browserVmControlServiceSettlementSmoke.includes( + "verify-delayed-turn-port-only-restart", + ) && + browserVmControlServiceSettlementSmoke.includes("log-started-paths-gone-owner-alive") && + browserVmControlServiceSettlementSmoke.includes( + "log-started-owner-exits-without-native-settlement", + ) && + browserVmControlServiceSettlementSmoke.includes( + "logged started launch with a live owner became terminal", + ) && + browserVmControlServiceSettlementSmoke.includes("verify-failed-flush-durability-restart") && + browserVmControlServiceSettlementSmoke.includes( + "failed durability close lost the data-save failure", + ) && + browserVmControlServiceSettlementSmoke.includes("Allocated inode: 728") && + browserVmControlServiceSettlementSmoke.includes("verify-transport-restart") && + browserVmControlServiceSettlementSmoke.includes( + "surviving transport owner was synthesized terminal after restart", + ) && browserVmEngineContractSmoke.includes("elastos.browser.vm-engine-contract-smoke/v1") && browserVmRemoteControlPreflightSmoke.includes("elastos.browser.vm-remote-control-preflight-smoke/v1") && browserVmRemoteControlPreflightSmoke.includes("remote_vm_control_socket") && @@ -1466,6 +1587,47 @@ assert( browserVzEngineSupervisor.includes("UNIX_SOCKET_PATH_BUDGET") && browserVzEngineSupervisor.includes("validate_unix_socket_path_budget") && browserVzEngineSupervisor.includes("/tmp/evzs") && + browserVzEngineSupervisor.includes("stage=shutdown_wait") && + browserVzEngineSupervisor.includes("sender_pid") && + browserVzEngineSupervisor.includes("stage=shutdown_wait_sender") && + browserVzEngineSupervisor.includes("stage=vz_delegate_exit") && + browserVzEngineSupervisor.includes("Browser VZ guest or framework stopped") && + browserVzEngineSupervisor.includes("wait_for_guest_exit") && + browserVzEngineSupervisor.includes("take_guest_exit_receiver") && + browserVzEngineSupervisor.includes("format_delegate_exit_reason") && + browserVzEngineSupervisor.includes("truncate_delegate_exit_message") && + browserVzEngineSupervisor.includes("is_char_boundary") && + !browserVzEngineSupervisor.includes("message.truncate(512)") && + browserVzEngineSupervisor.includes("request_guest_profile_disk_flush") && + browserVzEngineSupervisor.includes("guest_profile_disk_flush_accepted") && + browserVzEngineSupervisor.includes("settle_failed_profile_flush") && + browserVzEngineSupervisor.includes("settlement_with_absence_and_profile_durability") && + !browserVzEngineSupervisor.includes("retain_failed_profile_flush_durability") && + browserVzEngineSupervisor.includes( + "failed_profile_flush_keeps_truthful_child_absence_and_failed_durability", + ) && + browserVzEngineSupervisor.includes("leftover_writers_killed") && + browserVzEngineSupervisor.includes("profile_durability") && + browserVzEngineSupervisor.includes("stage=guest_profile_flush") && + browserVzEngineSupervisor.includes("overlay_guest_selkies_control_service") && + browserVzEngineSupervisor.includes("stage=guest_control_overlay") && + browserVzEngineSupervisor.includes("overlay_guest_webrtc_send_diagnostic") && + browserVzEngineSupervisor.includes("stage=guest_webrtc_send_overlay") && + browserVzEngineSupervisor.includes("ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY") && + browserVzEngineSupervisor.includes("ELASTOS_BROWSER_WEBRTC_SEND_GSTWEBRTC_SHA256") && + browserVzEngineSupervisor.includes("ELASTOS_BROWSER_WEBRTC_SEND_SELKIES_START_SHA256") && + browserVzEngineSupervisor.includes("webrtc_send_overlay_requested") && + browserVzEngineSupervisor.includes(".stdout(Stdio::null())") && + browserSelkiesControlService.includes("flushAndUnmountBrowserProfileDisk") && + browserSelkiesControlService.includes("quitGuestProfileDiskWriters") && + browserSelkiesControlService.includes("graceMs = 4000") && + browserSelkiesControlService.includes("browserProfileDiskDurability") && + browserSelkiesControlService.includes("leftover_writers_killed") && + browserSelkiesControlService.includes("shutdown_result") && + browserSelkiesControlService.includes("profile_disk_unmounted") && + browserSelkiesControlService.includes("Browser.close") && + browserVmControlService.includes("stderr_tail") && + browserVmControlService.includes("looksLikeRejectedVzMediaDiagnostic") && browserVzEngineSupervisor.includes("Browser VZ launcher requires adapter_ipc.runtime_stream_path") && browserVzEngineSupervisor.includes("validate_runtime_stream_socket_path") && browserVzEngineSupervisor.includes("launch_requires_runtime_owned_stream_path_for_egress") && @@ -1479,6 +1641,8 @@ assert( browserVzEngineSupervisor.includes("VzLaunchOwner") && browserVzEngineSupervisor.includes("TurnCleanupEvidence") && browserVzEngineSupervisor.includes("LaunchTurnStartError") && + browserVzEngineSupervisor.includes("wait_for_owned_turn_ports_absent") && + browserVzEngineSupervisor.includes("signal_owned_turn_process_group") && browserVzEngineSupervisor.includes("child_absent: self.terminate_and_reap()") && browserVzEngineSupervisor.includes("TurnCleanupEvidence::Indeterminate => false") && browserVzEngineSupervisor.includes("elastos.browser.vz-launch-settlement/v1") && @@ -1572,6 +1736,11 @@ assert( browserVmTargetStage.includes("found_media_iface") && browserVmTargetStage.includes('[ -n "$found_media_iface" ] && break') && browserVmTargetStage.includes("patch_selkies_relay_policy") && + browserVmTargetStage.includes("patch_selkies_display_renegotiate") && + browserVmTargetStage.includes("elastos_display_renegotiate") && + browserVmTargetStage.includes("request_display_offer") && + browserVmTargetStage.includes("recreating display pipeline for a new viewer") && + browserVmTargetStage.includes("_elastos_audio_only") && !browserVmTargetStage.includes("_elastos_raw_caps_with_framerate") && !browserVmTargetStage.includes("fraction_replacement") && browserVmTargetStage.includes("/run/elastos/browser-ice-transport-policy") && @@ -1838,7 +2007,22 @@ assert( ) && browserStyle.includes('.browser-status[data-visible="true"][data-copyable="true"]') && browserStyle.includes(".browser-status-copy") && - browser.includes("browser.js?v=browser-20260907c") && + browser.includes("browser.js?v=browser-20260915b") && + !browser.includes("browser.js?v=browser-20260915a") && + !browser.includes("browser.js?v=browser-20260914f") && + !browser.includes("browser.js?v=browser-20260914e") && + !browser.includes("browser.js?v=browser-20260914d") && + !browser.includes("browser.js?v=browser-20260914c") && + !browser.includes("browser.js?v=browser-20260913i") && + !browser.includes("browser.js?v=browser-20260913h") && + !browser.includes("browser.js?v=browser-20260913g") && + !browser.includes("browser.js?v=browser-20260913f") && + !browser.includes("browser.js?v=browser-20260913e") && + !browser.includes("browser.js?v=browser-20260913d") && + !browser.includes("browser.js?v=browser-20260913c") && + !browser.includes("browser.js?v=browser-20260913b") && + !browser.includes("browser.js?v=browser-20260913a") && + !browser.includes("browser.js?v=browser-20260907c") && !browser.includes("browser.js?v=browser-20260731a") && !browser.includes("browser.js?v=browser-20260730a") && !browser.includes("browser.js?v=browser-20260728a") && @@ -1900,7 +2084,14 @@ assert( ); assert( - browserJs.includes("browser-remote-display.js?v=browser-20260907c") && + browserJs.includes("browser-remote-display.js?v=browser-20260915d") && + !browserJs.includes("browser-remote-display.js?v=browser-20260915a") && + !browserJs.includes("browser-remote-display.js?v=browser-20260914c") && + !browserJs.includes("browser-remote-display.js?v=browser-20260913h") && + !browserJs.includes("browser-remote-display.js?v=browser-20260913g") && + !browserJs.includes("browser-remote-display.js?v=browser-20260913f") && + !browserJs.includes("browser-remote-display.js?v=browser-20260913e") && + !browserJs.includes("browser-remote-display.js?v=browser-20260907c") && !browserJs.includes("browser-remote-display.js?v=browser-20260730b") && !browserJs.includes("browser-remote-display.js?v=browser-20260728a") && !browserJs.includes("browser-remote-display.js?v=browser-20260727a") && @@ -2013,6 +2204,10 @@ assert( !releaseRuntimePageForUnloadBlock.includes("currentPage = null") && !releaseRuntimePageForUnloadBlock.includes("publishRuntimePageForHost(null)") && !releaseRuntimePageForUnloadBlock.includes("closeRemoteDisplay()") && + !releaseRuntimePageForUnloadBlock.includes("prepareRecoverableDisplayAttach") && + !releaseRuntimePageForUnloadBlock.includes("startRecoverableDisplayAttach") && + !releaseRuntimePageForUnloadBlock.includes("display_attach") && + !releaseRuntimePageForUnloadBlock.includes("keepalive") && failRuntimeOwnedPageBlock.includes("closeRemoteDisplay();") && !failRuntimeOwnedPageBlock.includes("currentPage = null") && !failRuntimeOwnedPageBlock.includes("publishRuntimePageForHost(null)") && @@ -2200,6 +2395,13 @@ assert( "Browser guest raster is fixed at 1920x1080", ) && !browserSelkiesControlService.includes("function resizeBrowserPage") && + browserSelkiesControlService.includes("waitForRenegotiatedOffer") && + browserSelkiesControlService.includes("settleRenegotiatedOffer") && + browserSelkiesControlService.includes("Selkies relay ICE for the new offer") && + browserSelkiesControlService.includes("relayIceCandidates") && + browserSelkiesControlService.includes("rememberIceCandidate") && + browserSelkiesControlService.includes("this.remoteCandidateHistory = [];") && + browserSelkiesControlService.includes("relayIceCandidates(") && browserSelkiesControlService.includes("function mediaKindsForSdp") && !browserSelkiesControlService.includes("isSelkiesAudioUnavailable") && !browserSelkiesControlService.includes("audio_offer_unavailable") && diff --git a/scripts/browser-profile-disk-flush.test.mjs b/scripts/browser-profile-disk-flush.test.mjs new file mode 100644 index 000000000..328db2afb --- /dev/null +++ b/scripts/browser-profile-disk-flush.test.mjs @@ -0,0 +1,194 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { + BROWSER_PROFILE_DISK_MOUNT, + browserProfileDiskDurability, + flushAndUnmountBrowserProfileDisk, + guestProcessIdsHoldingProfileDisk, + quitGuestProfileDiskWriters, +} from "./browser-selkies-control-service.mjs"; + +test("flush treats a missing profile mount as already unmounted", () => { + const commands = []; + const result = flushAndUnmountBrowserProfileDisk({ + exec: (command, args = []) => { + commands.push([command, ...args]); + return { status: 0 }; + }, + readMounts: () => "ext4 / rw 0 0\n", + }); + assert.deepEqual(commands, [["sync"]]); + assert.deepEqual(result, { + flushed: true, + unmounted: true, + already_absent: true, + }); +}); + +test("flush syncs then unmounts the guest profile disk", () => { + const commands = []; + const result = flushAndUnmountBrowserProfileDisk({ + exec: (command, args = []) => { + commands.push([command, ...args]); + return { status: 0 }; + }, + readMounts: () => `/dev/vdb ${BROWSER_PROFILE_DISK_MOUNT} ext4 rw,noatime 0 0\n`, + }); + assert.deepEqual(commands, [["sync"], ["umount", BROWSER_PROFILE_DISK_MOUNT]]); + assert.deepEqual(result, { + flushed: true, + unmounted: true, + already_absent: false, + }); +}); + +test("flush propagates a readMounts exception", () => { + assert.throws( + () => + flushAndUnmountBrowserProfileDisk({ + exec: () => ({ status: 0 }), + readMounts: () => { + throw new Error("proc mounts are unreadable"); + }, + }), + /proc mounts are unreadable/, + ); +}); + +test("flush fails closed when umount fails", () => { + assert.throws( + () => + flushAndUnmountBrowserProfileDisk({ + exec: (command) => ({ status: command === "umount" ? 1 : 0 }), + readMounts: () => `/dev/vdb ${BROWSER_PROFILE_DISK_MOUNT} ext4 rw 0 0\n`, + umountAttempts: 1, + }), + /Browser profile disk umount failed/, + ); +}); + +test("flush retries a busy umount until the profile disk releases", () => { + let umounts = 0; + const sleeps = []; + const result = flushAndUnmountBrowserProfileDisk({ + exec: (command) => { + if (command === "umount") { + umounts += 1; + return { status: umounts < 3 ? 1 : 0 }; + } + return { status: 0 }; + }, + readMounts: () => `/dev/vdb ${BROWSER_PROFILE_DISK_MOUNT} ext4 rw 0 0\n`, + sleep: (ms) => sleeps.push(ms), + umountAttempts: 4, + umountRetryMs: 250, + }); + assert.equal(umounts, 3); + assert.deepEqual(sleeps, [250, 250]); + assert.deepEqual(result, { + flushed: true, + unmounted: true, + already_absent: false, + }); +}); + +test("guest writer discovery skips the control service pid", () => { + const ids = guestProcessIdsHoldingProfileDisk({ + listProc: () => ["1", "22", "33"], + readCmdline: (pid) => { + if (pid === "1") return "init\0"; + if (pid === "22") return "/opt/elastos/bin/browser-native-proxy-engine\0"; + return `--user-data-dir=${BROWSER_PROFILE_DISK_MOUNT}/profiles/key\0`; + }, + selfPid: 33, + }); + assert.deepEqual(ids, [22]); +}); + +test("guest writer quit waits for Browser.close exit before TERM", async () => { + const commands = []; + const leftover = await quitGuestProfileDiskWriters({ + exec: (command, args = []) => { + commands.push([command, ...args]); + return { status: 0 }; + }, + listProc: () => ["9"], + readCmdline: () => "browser-native-proxy-engine\0", + pidAlive: () => false, + nowMs: (() => { + let now = 0; + return () => { + now += 10; + return now; + }; + })(), + sleep: () => {}, + graceMs: 4000, + timeoutMs: 1, + }); + assert.deepEqual(leftover, []); + assert.deepEqual(commands, []); +}); + +test("guest writer quit keeps the persist grace after writers already exited", async () => { + const sleeps = []; + let now = 0; + const leftover = await quitGuestProfileDiskWriters({ + exec: () => ({ status: 0 }), + listProc: () => ["9"], + readCmdline: () => "browser-native-proxy-engine\0", + pidAlive: () => false, + nowMs: () => now, + sleep: (ms) => { + sleeps.push(ms); + now += ms; + }, + graceMs: 4000, + timeoutMs: 1, + }); + assert.deepEqual(leftover, []); + assert.equal(sleeps.reduce((sum, value) => sum + value, 0), 4000); + assert.ok(now >= 4000); +}); + +test("guest writer quit sends TERM then KILL for leftover pids", async () => { + const commands = []; + const leftover = await quitGuestProfileDiskWriters({ + exec: (command, args = []) => { + commands.push([command, ...args]); + return { status: 0 }; + }, + listProc: () => ["9"], + readCmdline: () => "browser-native-proxy-engine\0", + pidAlive: () => true, + nowMs: (() => { + let now = 0; + return () => { + now += 5000; + return now; + }; + })(), + sleep: () => {}, + graceMs: 0, + timeoutMs: 1, + }); + assert.deepEqual(leftover, [9]); + assert.deepEqual(commands, [ + ["kill", "-TERM", "9"], + ["kill", "-KILL", "9"], + ]); + assert.deepEqual( + browserProfileDiskDurability( + { flushed: true, unmounted: true, already_absent: false }, + leftover, + ), + { + ok: false, + profile_disk_flushed: false, + profile_disk_unmounted: true, + leftover_writers_killed: true, + error: + "Guest profile disk writers required SIGKILL; application writes are unproven", + }, + ); +}); diff --git a/scripts/browser-selkies-control-service.mjs b/scripts/browser-selkies-control-service.mjs index 1c362cda2..95a704e59 100755 --- a/scripts/browser-selkies-control-service.mjs +++ b/scripts/browser-selkies-control-service.mjs @@ -1,4 +1,5 @@ #!/usr/bin/env node +import { spawnSync } from "node:child_process"; import crypto from "node:crypto"; import fs from "node:fs"; import http from "node:http"; @@ -18,6 +19,7 @@ const DISPLAY_CONTROL_HTTP_STATUS = Object.freeze({ display_attach_failed: 503, display_attach_uncertain: 503, }); +const DISPLAY_RENEGOTIATE_SIGNAL = JSON.stringify({ elastos_display_renegotiate: true }); const HOSTED_PRODUCT_OPEN_SCHEMA = "elastos.browser.hosted-product.open/v1"; const VM_GUEST_OPEN_SCHEMA = "elastos.browser.vm-guest.open/v1"; const VM_LOG_DIR = "/var/log/elastos"; @@ -762,6 +764,7 @@ function normalizeWalletBridge(wallet) { accounts.some((account) => account.account_id === wallet.default_account_id) ? wallet.default_account_id : accounts[0]?.account_id || ""; + const consumerMediation = walletUsesConsumerMediation(wallet); return { accounts, default_chain_namespace: defaultChain, @@ -775,13 +778,29 @@ function normalizeWalletBridge(wallet) { transaction_broadcast_url: typeof wallet?.transaction_broadcast_url === "string" ? wallet.transaction_broadcast_url : "", approval_status_url: typeof wallet?.approval_status_url === "string" ? wallet.approval_status_url : "", - home_token: typeof wallet?.home_token === "string" ? wallet.home_token : "", + home_token: consumerMediation + ? "" + : typeof wallet?.home_token === "string" + ? wallet.home_token + : "", principal_id: typeof wallet?.principal_id === "string" ? wallet.principal_id : "", session_id: typeof wallet?.session_id === "string" ? wallet.session_id : "", launch_id: typeof wallet?.launch_id === "string" ? wallet.launch_id : "", + schema: typeof wallet?.schema === "string" ? wallet.schema : "", + resolution: typeof wallet?.resolution === "string" ? wallet.resolution : "", + page_id: typeof wallet?.page_id === "string" ? wallet.page_id : "", + lifecycle_generation: + typeof wallet?.lifecycle_generation === "string" ? wallet.lifecycle_generation : "", }; } +function walletUsesConsumerMediation(wallet) { + return ( + wallet?.schema === "elastos.browser.wallet-consumer-mediation/v1" && + wallet.resolution === "consumer_runtime" + ); +} + function chainNamespaceToDecimal(namespace) { const [, value] = String(namespace || "").split(":"); const parsed = Number(value || ""); @@ -1824,6 +1843,92 @@ function nowIso() { return new Date().toISOString(); } +function isSdpOffer(message) { + return message?.sdp?.type === "offer" && typeof message.sdp.sdp === "string"; +} + +function isRelayIce(message) { + return Boolean(message?.ice) && summarizeIceCandidate(message.ice)?.type === "relay"; +} + +function relayIceCandidates(list) { + return (Array.isArray(list) ? list : []).filter( + (candidate) => summarizeIceCandidate(candidate)?.type === "relay", + ); +} + +function rememberIceCandidate(candidates, history, ice) { + if (!ice) { + return; + } + const exists = (Array.isArray(history) ? history : []).some((candidate) => ( + candidate?.candidate === ice.candidate && + candidate?.sdpMLineIndex === ice.sdpMLineIndex && + candidate?.sdpMid === ice.sdpMid + )); + if (exists) { + return; + } + if (Array.isArray(candidates)) { + candidates.push(ice); + } + if (Array.isArray(history)) { + history.push(ice); + } +} + +async function waitForRenegotiatedOffer(waitFor, clearHistory, rememberIce) { + let offer = await waitFor(isSdpOffer, "Selkies SDP offer"); + for (;;) { + const next = await waitFor( + (message) => isSdpOffer(message) || isRelayIce(message), + "Selkies relay ICE for the new offer", + ); + if (isSdpOffer(next)) { + clearHistory(); + offer = next; + continue; + } + rememberIce(next.ice); + return offer; + } +} + +function takeQueuedMessage(messages, predicate) { + const index = (Array.isArray(messages) ? messages : []).findIndex(predicate); + if (index < 0) { + return null; + } + const [message] = messages.splice(index, 1); + return message; +} + +async function settleRenegotiatedOffer(offer, waitFor, rememberIce, takeQueuedOffer, getHistory) { + let current = offer; + let sawNewer = false; + for (;;) { + const newer = takeQueuedOffer(); + if (newer && isSdpOffer(newer)) { + current = newer; + sawNewer = true; + continue; + } + const candidates = relayIceCandidates(getHistory()); + if (candidates.length > 0 || !sawNewer) { + return { offer: current, candidates }; + } + const next = await waitFor( + (message) => isSdpOffer(message) || isRelayIce(message), + "Selkies relay ICE for the new offer", + ); + if (isSdpOffer(next)) { + current = next; + continue; + } + rememberIce(next.ice); + } +} + function summarizeIceCandidate(candidate) { const line = String(candidate?.candidate || "").trim(); if (!line) { @@ -2320,20 +2425,36 @@ export class SelkiesPage { this.checkDisplayAttachmentDeadline(); }; current(); - await socket.connect(this.config.connectTimeoutMs); - current(); - socket.sendText("HELLO client " + JSON.stringify({ client_type: "controller", client_slot: 3, client_strict_viewer: false })); - await this.waitForAudio((message) => message.kind === "hello", "Selkies audio HELLO"); - current(); - socket.sendText("SESSION server"); - const session = await this.waitForAudio((message) => message.kind === "session_ok", "Selkies audio SESSION_OK"); - current(); - this.audioServerPeerId = session.serverPeerId; - this.audioSignalingEnvelope = "peer_routed"; - const offer = await this.waitForAudio( - (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", - "Selkies audio SDP offer", - ); + const reuse = socket.closed !== true && (this.audioServerPeerId || socket.legacyHelloAccepted); + if (reuse) { + socket.sendText(DISPLAY_RENEGOTIATE_SIGNAL); + } else { + await socket.connect(this.config.connectTimeoutMs); + current(); + socket.sendText("HELLO client " + JSON.stringify({ client_type: "controller", client_slot: 3, client_strict_viewer: false })); + await this.waitForAudio((message) => message.kind === "hello", "Selkies audio HELLO"); + current(); + socket.sendText("SESSION server"); + } + if (!reuse) { + const session = await this.waitForAudio((message) => message.kind === "session_ok", "Selkies audio SESSION_OK"); + current(); + this.audioServerPeerId = session.serverPeerId; + this.audioSignalingEnvelope = "peer_routed"; + } + const offer = reuse + ? await waitForRenegotiatedOffer( + (predicate, label) => this.waitForAudio(predicate, label), + () => { + this.audioRemoteCandidates = []; + this.audioRemoteCandidateHistory = []; + }, + (ice) => rememberIceCandidate(this.audioRemoteCandidates, this.audioRemoteCandidateHistory, ice), + ) + : await this.waitForAudio( + (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", + "Selkies audio SDP offer", + ); current(); return offer; } @@ -2345,20 +2466,34 @@ export class SelkiesPage { this.checkDisplayAttachmentDeadline(); }; current(); - await socket.connect(this.config.connectTimeoutMs); - current(); - const helloMeta = Buffer.from(JSON.stringify({ - res: `${this.config.displaySurface.stream.width}x${this.config.displaySurface.stream.height}`, - scale: displaySize.scale || 1, - })).toString("base64"); - socket.sendText(`HELLO 3 ${helloMeta}`); - await this.waitForAudio((message) => message.kind === "hello", "legacy Selkies audio HELLO"); - socket.legacyHelloAccepted = true; + const reuse = socket.closed !== true && (this.audioServerPeerId || socket.legacyHelloAccepted); + if (reuse) { + socket.sendText(DISPLAY_RENEGOTIATE_SIGNAL); + } else { + await socket.connect(this.config.connectTimeoutMs); + current(); + const helloMeta = Buffer.from(JSON.stringify({ + res: `${this.config.displaySurface.stream.width}x${this.config.displaySurface.stream.height}`, + scale: displaySize.scale || 1, + })).toString("base64"); + socket.sendText(`HELLO 3 ${helloMeta}`); + await this.waitForAudio((message) => message.kind === "hello", "legacy Selkies audio HELLO"); + socket.legacyHelloAccepted = true; + } current(); - const offer = await this.waitForAudio( - (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", - "legacy Selkies audio SDP offer", - ); + const offer = reuse + ? await waitForRenegotiatedOffer( + (predicate, label) => this.waitForAudio(predicate, label), + () => { + this.audioRemoteCandidates = []; + this.audioRemoteCandidateHistory = []; + }, + (ice) => rememberIceCandidate(this.audioRemoteCandidates, this.audioRemoteCandidateHistory, ice), + ) + : await this.waitForAudio( + (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", + "legacy Selkies audio SDP offer", + ); current(); this.audioServerPeerId = offer.from || "2"; this.audioSignalingEnvelope = "raw_json"; @@ -2372,20 +2507,36 @@ export class SelkiesPage { this.checkDisplayAttachmentDeadline(); }; current(); - await socket.connect(this.config.connectTimeoutMs); - current(); - socket.sendText("HELLO client " + JSON.stringify({ client_type: "controller", client_slot: 1, client_strict_viewer: false })); - await this.waitFor((message) => message.kind === "hello", "Selkies HELLO"); - current(); - socket.sendText("SESSION server"); - const session = await this.waitFor((message) => message.kind === "session_ok", "Selkies SESSION_OK"); - current(); - this.serverPeerId = session.serverPeerId; - this.signalingEnvelope = "peer_routed"; - const offer = await this.waitFor( - (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", - "Selkies SDP offer", - ); + const reuse = socket.closed !== true && (this.serverPeerId || socket.legacyHelloAccepted); + if (reuse) { + socket.sendText(DISPLAY_RENEGOTIATE_SIGNAL); + } else { + await socket.connect(this.config.connectTimeoutMs); + current(); + socket.sendText("HELLO client " + JSON.stringify({ client_type: "controller", client_slot: 1, client_strict_viewer: false })); + await this.waitFor((message) => message.kind === "hello", "Selkies HELLO"); + current(); + socket.sendText("SESSION server"); + } + if (!reuse) { + const session = await this.waitFor((message) => message.kind === "session_ok", "Selkies SESSION_OK"); + current(); + this.serverPeerId = session.serverPeerId; + this.signalingEnvelope = "peer_routed"; + } + const offer = reuse + ? await waitForRenegotiatedOffer( + (predicate, label) => this.waitFor(predicate, label), + () => { + this.remoteCandidates = []; + this.remoteCandidateHistory = []; + }, + (ice) => rememberIceCandidate(this.remoteCandidates, this.remoteCandidateHistory, ice), + ) + : await this.waitFor( + (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", + "Selkies SDP offer", + ); current(); return offer; } @@ -2397,20 +2548,34 @@ export class SelkiesPage { this.checkDisplayAttachmentDeadline(); }; current(); - await socket.connect(this.config.connectTimeoutMs); - current(); - const helloMeta = Buffer.from(JSON.stringify({ - res: `${this.config.displaySurface.stream.width}x${this.config.displaySurface.stream.height}`, - scale: displaySize.scale || 1, - })).toString("base64"); - socket.sendText(`HELLO 1 ${helloMeta}`); - await this.waitFor((message) => message.kind === "hello", "legacy Selkies HELLO"); - socket.legacyHelloAccepted = true; + const reuse = socket.closed !== true && (this.serverPeerId || socket.legacyHelloAccepted); + if (reuse) { + socket.sendText(DISPLAY_RENEGOTIATE_SIGNAL); + } else { + await socket.connect(this.config.connectTimeoutMs); + current(); + const helloMeta = Buffer.from(JSON.stringify({ + res: `${this.config.displaySurface.stream.width}x${this.config.displaySurface.stream.height}`, + scale: displaySize.scale || 1, + })).toString("base64"); + socket.sendText(`HELLO 1 ${helloMeta}`); + await this.waitFor((message) => message.kind === "hello", "legacy Selkies HELLO"); + socket.legacyHelloAccepted = true; + } current(); - const offer = await this.waitFor( - (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", - "legacy Selkies SDP offer", - ); + const offer = reuse + ? await waitForRenegotiatedOffer( + (predicate, label) => this.waitFor(predicate, label), + () => { + this.remoteCandidates = []; + this.remoteCandidateHistory = []; + }, + (ice) => rememberIceCandidate(this.remoteCandidates, this.remoteCandidateHistory, ice), + ) + : await this.waitFor( + (message) => message.sdp?.type === "offer" && typeof message.sdp.sdp === "string", + "legacy Selkies SDP offer", + ); current(); this.serverPeerId = offer.from || "1"; this.signalingEnvelope = "raw_json"; @@ -2490,6 +2655,8 @@ export class SelkiesPage { } if (parsed?.sdp?.type === "offer") { this.signalingStats.selkies_offers_received += 1; + this.remoteCandidates = []; + this.remoteCandidateHistory = []; } if (parsed?.ice) { this.remoteCandidates.push(parsed.ice); @@ -2511,6 +2678,8 @@ export class SelkiesPage { } if (parsed?.sdp?.type === "offer") { this.signalingStats.selkies_offers_received += 1; + this.audioRemoteCandidates = []; + this.audioRemoteCandidateHistory = []; } if (parsed?.ice) { this.audioRemoteCandidates.push(parsed.ice); @@ -2524,6 +2693,38 @@ export class SelkiesPage { } } + canReuseDisplaySignaling() { + return Boolean( + this.ws && + this.audioWs && + !this.videoClosed && + !this.audioClosed && + this.ws.closed !== true && + this.audioWs.closed !== true && + (this.serverPeerId || this.ws.legacyHelloAccepted) && + (this.audioServerPeerId || this.audioWs.legacyHelloAccepted) + ); + } + + rejectStaleDisplayWaiters() { + for (const waiter of this.waiters) { + clearTimeout(waiter.timer); + waiter.reject(new Error(`Selkies WebSocket closed while waiting for ${waiter.label}`)); + } + this.waiters = []; + this.messages = []; + this.remoteCandidates = []; + this.remoteCandidateHistory = []; + for (const waiter of this.audioWaiters) { + clearTimeout(waiter.timer); + waiter.reject(new Error(`Selkies audio WebSocket closed while waiting for ${waiter.label}`)); + } + this.audioWaiters = []; + this.audioMessages = []; + this.audioRemoteCandidates = []; + this.audioRemoteCandidateHistory = []; + } + flushWaiters() { for (const waiter of [...this.waiters]) { const matchIndex = this.messages.findIndex(waiter.predicate); @@ -2657,9 +2858,14 @@ export class SelkiesPage { }; const prepare = Promise.resolve().then(async () => { current(); - retire(); - this.resetSignaling(); - this.resetAudioSignaling(); + const reusedSignaling = this.canReuseDisplaySignaling(); + if (reusedSignaling) { + this.rejectStaleDisplayWaiters(); + } else { + retire(); + this.resetSignaling(); + this.resetAudioSignaling(); + } const size = browserDisplayMetrics(this.config); const negotiate = async (channel, protocol) => { for (;;) { @@ -2690,19 +2896,59 @@ export class SelkiesPage { ]); current(); if (this.videoClosed || this.audioClosed) throw new Error("Browser display signaling closed"); + let videoBound = { + offer: videoOffer, + candidates: relayIceCandidates(this.remoteCandidateHistory), + }; + let audioBound = { + offer: audioOffer, + candidates: relayIceCandidates(this.audioRemoteCandidateHistory), + }; + if (reusedSignaling) { + videoBound = await settleRenegotiatedOffer( + videoOffer, + (predicate, label) => this.waitFor(predicate, label), + (ice) => rememberIceCandidate(this.remoteCandidates, this.remoteCandidateHistory, ice), + () => takeQueuedMessage(this.messages, isSdpOffer), + () => this.remoteCandidateHistory, + ); + current(); + audioBound = await settleRenegotiatedOffer( + audioOffer, + (predicate, label) => this.waitForAudio(predicate, label), + (ice) => rememberIceCandidate(this.audioRemoteCandidates, this.audioRemoteCandidateHistory, ice), + () => takeQueuedMessage(this.audioMessages, isSdpOffer), + () => this.audioRemoteCandidateHistory, + ); + current(); + } + if (this.videoClosed || this.audioClosed) throw new Error("Browser display signaling closed"); const generation = `display:${crypto.randomBytes(16).toString("hex")}`; - const display = this.supervisorResult(videoOffer.sdp.sdp, this.browserPage, this.wallet, audioOffer.sdp.sdp).display_session; + const display = this.supervisorResult( + videoBound.offer.sdp.sdp, + this.browserPage, + this.wallet, + audioBound.offer.sdp.sdp, + ).display_session; + const initial_offer = { + ...display.initial_offer, + candidates: videoBound.candidates, + }; + const audio_offer = { + ...display.audio_offer, + candidates: audioBound.candidates, + }; const result = { schema: "elastos.browser.display-attach-result/v1", page_id: this.pageId, request_id: attachment.requestId, previous_display_generation: attachment.previousGeneration, display_generation: generation, - initial_offer: display.initial_offer, - audio_offer: display.audio_offer, + initial_offer, + audio_offer, }; this.displayGeneration = generation; - this.displaySession = { ...display, display_generation: generation }; + this.displaySession = { ...display, display_generation: generation, initial_offer, audio_offer }; this.displayAvailable = true; return result; }); @@ -2806,9 +3052,11 @@ export class SelkiesPage { } ack(type) { - const candidates = type === "answer" - ? this.remoteCandidateHistory.slice() - : this.remoteCandidates.splice(0); + const candidates = relayIceCandidates( + type === "answer" + ? this.remoteCandidateHistory.slice() + : this.remoteCandidates.splice(0), + ); if (type === "answer") { this.remoteCandidates = []; } @@ -2824,9 +3072,11 @@ export class SelkiesPage { } ackAudio(type) { - const candidates = type === "answer" - ? this.audioRemoteCandidateHistory.slice() - : this.audioRemoteCandidates.splice(0); + const candidates = relayIceCandidates( + type === "answer" + ? this.audioRemoteCandidateHistory.slice() + : this.audioRemoteCandidates.splice(0), + ); if (type === "answer") { this.audioRemoteCandidates = []; } @@ -2842,12 +3092,16 @@ export class SelkiesPage { } close() { + this.closeAndPersist().catch(() => {}); + } + + async closeAndPersist() { this.markClosed(); this.ws?.close(); this.markAudioClosed(); this.audioWs?.close(); - if (this.browserPage?.target_id) { - closeBrowserPage(this.config.browserControl, this.browserPage).catch(() => {}); + if (this.browserPage) { + await closeBrowserPage(this.config.browserControl, this.browserPage).catch(() => {}); } } @@ -3277,6 +3531,11 @@ async function walletRuntimeFetchJson( url, { method = "GET", body = null, timeoutMs: requestedTimeoutMs = null } = {}, ) { + if (walletUsesConsumerMediation(runtime.wallet)) { + throw walletRuntimeHttpError( + "Consumer Wallet mediation is bound for this Browser session. Wallet Bus stays on the consumer Runtime.", + ); + } if (!url || !runtime.wallet.home_token) { throw walletRuntimeHttpError("Runtime wallet endpoint is unavailable for this Browser session."); } @@ -3409,23 +3668,26 @@ export function walletRuntimeRequestedAccount(runtime, rawBody, { required = tru if (!/^eip155:\d+$/.test(requestedChain)) { throw walletRuntimeHttpError("Runtime wallet request chain is invalid.", 4902); } - const managed = (candidate) => candidate.proof_type === "managed_evm"; + const selectable = (candidate) => + candidate.proof_type === "managed_evm" || + (typeof candidate.connector_id === "string" && candidate.connector_id); const account = runtime.wallet.accounts.find( (candidate) => - managed(candidate) && + selectable(candidate) && candidate.chain_namespace === requestedChain && candidate.account_id === runtime.wallet.default_account_id, ) || runtime.wallet.accounts.find( - (candidate) => managed(candidate) && candidate.chain_namespace === requestedChain, + (candidate) => + selectable(candidate) && candidate.chain_namespace === requestedChain, ); if (!account) { if (!required) { return null; } throw walletRuntimeHttpError( - `No Runtime-managed wallet account is available for ${requestedChain}.`, + `No principal-owned wallet account is available for ${requestedChain}.`, 4902, ); } @@ -5507,6 +5769,179 @@ function validateOpenRequest(body) { return launch; } +export const BROWSER_PROFILE_DISK_MOUNT = "/var/lib/elastos/browser-profile-disk"; + +function defaultExec(command, args = []) { + return spawnSync(command, args, { encoding: "utf8" }); +} + +function defaultReadMounts() { + return fs.readFileSync("/proc/mounts", "utf8"); +} + +function defaultListProc() { + return fs.readdirSync("/proc"); +} + +function defaultReadCmdline(pid) { + return fs.readFileSync(`/proc/${pid}/cmdline`, "utf8"); +} + +function defaultPidAlive(pid) { + try { + fs.accessSync(`/proc/${pid}`); + return true; + } catch { + return false; + } +} + +export function guestProcessIdsHoldingProfileDisk({ + listProc = defaultListProc, + readCmdline = defaultReadCmdline, + mountPoint = BROWSER_PROFILE_DISK_MOUNT, + selfPid = process.pid, +} = {}) { + const ids = []; + let names; + try { + names = listProc(); + } catch { + return ids; + } + for (const name of names) { + if (!/^\d+$/.test(name)) continue; + const pid = Number(name); + if (pid === selfPid) continue; + let cmdline = ""; + try { + cmdline = readCmdline(name); + } catch { + continue; + } + if ( + cmdline.includes("browser-native-proxy-engine") || + cmdline.includes(`--user-data-dir=${mountPoint}`) + ) { + ids.push(pid); + } + } + return ids; +} + +export function signalGuestProfileDiskWriters(pids, { exec = defaultExec, signal = "TERM" } = {}) { + for (const pid of pids) { + exec("kill", [`-${signal}`, String(pid)]); + } +} + +export function flushAndUnmountBrowserProfileDisk({ + exec = defaultExec, + readMounts = defaultReadMounts, + mountPoint = BROWSER_PROFILE_DISK_MOUNT, + sleep = (ms) => { + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); + }, + umountAttempts = 8, + umountRetryMs = 250, +} = {}) { + const syncResult = exec("sync", []); + if (syncResult.status !== 0) { + throw new Error("Browser profile disk sync failed"); + } + const mounts = readMounts(); + if (!mounts.includes(` ${mountPoint} `)) { + return { flushed: true, unmounted: true, already_absent: true }; + } + const attempts = Math.max(1, Number(umountAttempts) || 1); + for (let attempt = 0; attempt < attempts; attempt += 1) { + const umount = exec("umount", [mountPoint]); + if (umount.status === 0) { + return { flushed: true, unmounted: true, already_absent: false }; + } + if (attempt + 1 < attempts) { + sleep(umountRetryMs); + } + } + throw new Error("Browser profile disk umount failed"); +} + +export async function quitGuestProfileDiskWriters({ + exec = defaultExec, + listProc = defaultListProc, + readCmdline = defaultReadCmdline, + pidAlive = defaultPidAlive, + nowMs = () => Date.now(), + sleep = (ms) => { + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); + }, + graceMs = 4000, + timeoutMs = 5000, +} = {}) { + const pids = guestProcessIdsHoldingProfileDisk({ listProc, readCmdline }); + const graceDeadline = nowMs() + Math.max(0, graceMs); + while (nowMs() < graceDeadline) { + sleep(Math.min(50, Math.max(0, graceDeadline - nowMs()))); + } + let leftover = pids.filter((pid) => pidAlive(pid)); + if (leftover.length === 0) { + return leftover; + } + signalGuestProfileDiskWriters(leftover, { exec, signal: "TERM" }); + const deadline = nowMs() + timeoutMs; + while (leftover.some((pid) => pidAlive(pid)) && nowMs() < deadline) { + sleep(50); + } + leftover = leftover.filter((pid) => pidAlive(pid)); + if (leftover.length > 0) { + signalGuestProfileDiskWriters(leftover, { exec, signal: "KILL" }); + } + return leftover; +} + +export function browserProfileDiskDurability(profile, leftoverWriterPids = []) { + const leftover = Array.isArray(leftoverWriterPids) ? leftoverWriterPids : []; + const leftoverKilled = leftover.length > 0; + if (leftoverKilled) { + return { + ok: false, + profile_disk_flushed: false, + profile_disk_unmounted: profile?.unmounted === true, + leftover_writers_killed: true, + error: + "Guest profile disk writers required SIGKILL; application writes are unproven", + }; + } + return { + ok: profile?.flushed === true && profile?.unmounted === true, + profile_disk_flushed: profile?.flushed === true, + profile_disk_unmounted: profile?.unmounted === true, + leftover_writers_killed: false, + }; +} + +async function quitChromiumBrowser(browserControl) { + if (!browserControl) { + return; + } + try { + const version = await fetchBrowserControlJson(browserControl, "/json/version"); + const wsUrl = version?.webSocketDebuggerUrl; + if (!wsUrl) { + return; + } + const cdp = new CdpClient(wsUrl, 5000, { retainEvents: false }); + await cdp.connect(5000); + try { + await cdp.request("Browser.close", {}, 5000); + } finally { + cdp.close(); + } + } catch { + // Chromium may already be gone. Process-group kill still flushes or fails closed. + } +} + async function main() { const config = readConfig(); if (fs.existsSync(config.controlSocketPath)) { @@ -5520,11 +5955,9 @@ async function main() { const markSessionClosed = () => { lastSessionClosedAt = Date.now(); }; - const closeActivePages = () => { + const closeActivePages = async () => { const pageIds = [...pages.keys()]; - for (const page of pages.values()) { - page.close(); - } + await Promise.all([...pages.values()].map((page) => page.closeAndPersist())); pages.clear(); if (pageIds.length > 0) { markSessionClosed(); @@ -5542,10 +5975,31 @@ async function main() { const url = new URL(req.url, "http://browser-engine"); if (req.method === "POST" && url.pathname === "/shutdown") { logControlEvent("request", { method: req.method, path: url.pathname }); - closeActivePages(); + const closedPageIds = await closeActivePages(); + await quitChromiumBrowser(config.browserControl); + const leftover = await quitGuestProfileDiskWriters(); + let profile; + try { + profile = flushAndUnmountBrowserProfileDisk(); + } catch (error) { + profile = { + flushed: false, + unmounted: false, + error: error instanceof Error ? error.message : String(error), + }; + } + const durability = { + ...browserProfileDiskDurability(profile, leftover), + ...(profile?.error ? { error: profile.error } : {}), + }; + logControlEvent("shutdown_result", { + closed_page_ids: closedPageIds, + ...durability, + }); httpJson(res, 200, { schema: "elastos.browser.selkies-control.shutdown/v1", - ok: true, + closed_page_ids: closedPageIds, + ...durability, }); setTimeout(() => { server.close(() => { @@ -5835,7 +6289,7 @@ async function main() { return; } if (req.method === "POST" && op === "close") { - page.close(); + await page.closeAndPersist(); pages.delete(pageId); markSessionClosed(); httpJson(res, 200, { schema: "elastos.browser.close-result/v1", page_id: pageId, closed: true }); diff --git a/scripts/browser-startup-readiness.test.mjs b/scripts/browser-startup-readiness.test.mjs index ea13af263..9ba906126 100644 --- a/scripts/browser-startup-readiness.test.mjs +++ b/scripts/browser-startup-readiness.test.mjs @@ -6,13 +6,13 @@ import vm from "node:vm"; const source = readFileSync(new URL("../capsules/browser/browser/browser.js", import.meta.url), "utf8"); const html = readFileSync(new URL("../capsules/browser/browser/index.html", import.meta.url), "utf8"); const input = html.match(/]*\bid="browser-url"[^>]*>/)?.[0]; -const startup = source.slice(source.lastIndexOf("const initialUrl =")); +const startup = source.slice(source.lastIndexOf("const requestedStartupUrl =")); function deferred() { let resolve, reject; const promise = new Promise((ok, fail) => { resolve = ok; reject = fail; }); return { promise, resolve, reject }; } -function start({ expired = false } = {}) { +function start({ expired = false, params = new URLSearchParams() } = {}) { const summary = deferred(), open = deferred(); const address = { disabled: /\sdisabled(?:\s|=|\/?>)/.test(input), value: "" }; const calls = [], messages = []; @@ -20,7 +20,7 @@ function start({ expired = false } = {}) { const setLoading = loading => { readiness.loading = loading; address.disabled = loading; }; const settled = vm.runInNewContext(startup, { unloadCleanupStarted: false, homeWindowCloseInFlight: false, homeWindowTerminalCloseConfirmed: false, - params: new URLSearchParams(), DEFAULT_URL: "https://ela.city/", addressInput: address, + params, DEFAULT_URL: "https://ela.city/", addressInput: address, updateNavState() {}, setLoading, fetchBrowserSummary: () => summary.promise, restoreRuntimePageViewer: async () => false, requestRuntimeOpen: async url => { calls.push(url); setLoading(true); try { await open.promise; } finally { setLoading(false); } }, @@ -35,12 +35,21 @@ test("address entry waits for its controller before the Browser module starts", assert.match(input, /\sdisabled(?:\s|=|\/?>)/); }); -test("a pending initial summary cannot advertise an address field that default launch will overwrite", async () => { +test("a Home window without a URL keeps the address field for the first ordinary submit", async () => { const state = start(); assert.equal(state.address.disabled, true); assert.equal(state.readiness.loading, true); assert.deepEqual(state.calls, []); state.summary.resolve({}); + await state.settled; + assert.deepEqual(state.calls, []); + assert.equal(state.address.disabled, false); +}); + +test("a launch URL still opens after the initial summary", async () => { + const state = start({ params: new URLSearchParams("url=https://ela.city/") }); + assert.deepEqual(state.calls, []); + state.summary.resolve({}); await new Promise(setImmediate); assert.deepEqual(state.calls, ["https://ela.city/"]); assert.equal(state.address.disabled, true); diff --git a/scripts/browser-viewer-metrics-read.test.mjs b/scripts/browser-viewer-metrics-read.test.mjs index 7e451a549..2121d8995 100644 --- a/scripts/browser-viewer-metrics-read.test.mjs +++ b/scripts/browser-viewer-metrics-read.test.mjs @@ -80,5 +80,27 @@ test('signaling evidence preserves only allowlisted stage metadata', () => { sdp: 'private', candidate: 'private', url: 'private', token: 'private' }), { signal_type: 'display_attach', error_code: 'display_attach_failed' }); assert.deepEqual(browserViewerSignalMetadata({ schema: 'elastos.browser.display-attach-result/v1', initial_offer: { sdp: 'private' } }), { attached: true }); + assert.deepEqual(browserViewerSignalMetadata({ + schema: 'elastos.browser.display-attach-result/v1', + initial_offer: { + sdp: 'private', + candidates: [ + { candidate: 'candidate:1 1 UDP 1 203.0.113.5 3478 typ relay' }, + { candidate: 'candidate:2 1 UDP 1 192.0.2.8 9 typ host' }, + ], + }, + audio_offer: { sdp: 'private', candidates: [] }, + }), { + attached: true, + initial_offer_ice: { candidate_count: 2, relay_candidate_count: 1 }, + audio_offer_ice: { candidate_count: 0, relay_candidate_count: 0 }, + }); + assert.deepEqual(browserViewerSignalMetadata({ + attached: true, + initial_offer_ice: { candidate_count: 1, relay_candidate_count: 1 }, + }), { + attached: true, + initial_offer_ice: { candidate_count: 1, relay_candidate_count: 1 }, + }); assert.deepEqual(browserViewerSignalMetadata({ type: ['answer'], code: ['display_attach_busy'], attached: 'true' }), {}); }); diff --git a/scripts/browser-viewer-restore.test.mjs b/scripts/browser-viewer-restore.test.mjs index 54af73e24..2978f5215 100644 --- a/scripts/browser-viewer-restore.test.mjs +++ b/scripts/browser-viewer-restore.test.mjs @@ -23,10 +23,11 @@ function summary() { } function harness() { const calls = [], failures = [], requests = []; - const state = vm.createContext({ crypto, currentPage: null, currentPageGeneration: 0, nextPageGeneration: 1, + const state = vm.createContext({ crypto, URLSearchParams, AbortSignal, currentPage: null, currentPageGeneration: 0, nextPageGeneration: 1, restoredViewerOwner: null, unloadCleanupStarted: false, relaunchRequested: false, homeWindowCloseInFlight: false, homeWindowTerminalCloseConfirmed: false, pendingHomeWindowCloseDelivery: null, runtimePageCleanup: { status: () => null }, + displayAttachRetryIdentity: null, window: {}, globalThis: null, runtimeOwnershipTerminallyAbsent: true, selectedBrowserEngineId: '', currentBrowserEngineId: '', selectedRemoteExitId: '', currentRemoteExitId: '', currentDisplayMode: '', currentView: null, lastPageStatus: null, runtimePageOwner, sameRuntimePageOwner, @@ -52,12 +53,158 @@ function harness() { actual_url: 'https://current.invalid/form', title: 'Retained form', direct_network: false, display_session: { mode: 'webrtc_remote_display', width: 1280, height: 720, source: 'diagnostic-status', ice_servers: [{ credential_present: true }] } }; }, }); - for (const name of ['currentRuntimePageOwner', 'runtimeViewerOwnerActive', 'recoverableRuntimePage', 'fetchPageStatus', 'attachRecoveredDisplay', 'restoreRuntimePageViewer', 'settleRemoteDisplayFailure']) { + state.globalThis = state; + for (const name of ['displayAttachRequestId', 'viewerDisplayAttachTimeoutMs', 'reuseDisplayAttachment', 'terminalDisplayAttachment', 'allocateDisplayAttachRetryId', 'markDisplayAttachRetryNeeded', 'displayAttachRequestForRecovery', 'currentRuntimePageOwner', 'runtimeViewerOwnerActive', 'recoverableRuntimePage', 'fetchPageStatus', 'attachRecoveredDisplay', 'restoreRuntimePageViewer', 'settleRemoteDisplayFailure']) { vm.runInContext(declaration(name), state); } return { state, calls, failures, requests, restore: value => state.restoreRuntimePageViewer(value) }; } +test('display attach request id is the generation hex', () => { + const state = vm.createContext({}); + vm.runInContext(declaration('displayAttachRequestId'), state); + assert.equal(state.displayAttachRequestId('display:' + '1'.repeat(32)), '1'.repeat(32)); + assert.equal(state.displayAttachRequestId('display:not-hex'), 'not-hex'); + assert.equal(state.displayAttachRequestId(''), ''); + assert.equal(state.displayAttachRequestId(null), ''); +}); + +test('unload release leaves display attach to a dedicated starter', () => { + const unload = declaration('releaseRuntimePageForUnload'); + assert.match(unload, /unloadCleanupStarted = true/); + assert.doesNotMatch(unload, /display_attach/); + assert.doesNotMatch(unload, /keepalive/); + assert.doesNotMatch(unload, /startRecoverableDisplayAttach/); + assert.doesNotMatch(source, /prepareRecoverableDisplayAttach/); +}); + +test('unload attach posts the current generation with keepalive and skips Home close', () => { + const posts = []; + const state = vm.createContext({ + launchToken: 'token-one', + recoverableDisplayAttachStarted: false, + currentPage: { + page_id: 'page-one', + display_session: { + mode: 'webrtc_remote_display', + display_generation: 'display:' + '1'.repeat(32), + }, + }, + browserSummary: { engine_adapter: { display_attach_supported: true } }, + homeWindowCloseInFlight: false, + homeWindowTerminalCloseConfirmed: false, + PRODUCT_DISPLAY_MODE: 'webrtc_remote_display', + console: { info() {} }, + fetch: (path, options) => { + posts.push({ path, options }); + return Promise.resolve(); + }, + }); + vm.runInContext(declaration('displayAttachRequestId'), state); + vm.runInContext(declaration('startRecoverableDisplayAttach'), state); + state.startRecoverableDisplayAttach(); + assert.equal(posts.length, 1); + assert.equal(posts[0].path, '/api/apps/browser/pages/page-one/webrtc'); + assert.equal(posts[0].options.method, 'POST'); + assert.equal(posts[0].options.keepalive, true); + assert.equal(posts[0].options.headers['x-elastos-home-token'], 'token-one'); + assert.deepEqual(JSON.parse(posts[0].options.body), { + type: 'display_attach', + request_id: '1'.repeat(32), + display_generation: 'display:' + '1'.repeat(32), + }); + state.startRecoverableDisplayAttach(); + assert.equal(posts.length, 1); + state.recoverableDisplayAttachStarted = false; + state.homeWindowCloseInFlight = true; + state.startRecoverableDisplayAttach(); + assert.equal(posts.length, 1); +}); + +test('boot attach uses the generation hex when no pending request exists', () => { + const boot = readFileSync(new URL('../capsules/browser/browser/browser-restore-boot.js', import.meta.url), 'utf8'); + assert.match(boot, /generation\.slice\("display:"\.length\)/); + assert.match(boot, /function waitForReadyAttach/); + assert.match(boot, /pending\.state === "ready"/); + assert.match(source, /__elastosBrowserRestoreBoot/); +}); + +test('recoverable display query writes page identity and clears it', () => { + const hrefs = []; + const location = { + href: 'http://localhost/apps/browser/?browser_instance=b#home_token=t', + pathname: '/apps/browser/', + search: '?browser_instance=b', + hash: '#home_token=t', + }; + const state = vm.createContext({ + location, + history: { + state: null, + replaceState(_s, _t, next) { + hrefs.push(next); + const url = new URL(next, 'http://localhost'); + location.href = url.href; + location.pathname = url.pathname; + location.search = url.search; + location.hash = url.hash; + }, + }, + URL, + }); + vm.runInContext(declaration('persistRecoverableDisplayQuery'), state); + state.persistRecoverableDisplayQuery({ + page_id: 'page-one', + display_session: { display_generation: 'display:' + '1'.repeat(32) }, + }); + assert.match(hrefs[0], /page_id=page-one/); + assert.match(hrefs[0], /display_generation=display%3A11111111111111111111111111111111/); + state.persistRecoverableDisplayQuery(null); + assert.equal(hrefs.at(-1).includes('page_id'), false); + assert.equal(hrefs.at(-1).includes('display_generation'), false); +}); + +test('failed boot attach falls back to a webrtc POST', async () => { + const h = harness(); + h.state.window = { + __elastosBrowserRestoreBoot: { + attachPromise: Promise.reject(new Error('boot attach failed')), + }, + }; + await h.restore(summary()); + assert.deepEqual(h.requests, ['/api/apps/browser/pages/page-one/status', '/api/apps/browser/pages/page-one/webrtc']); + assert.equal(h.calls.find(row => row[0] === 'connect')[3].initial_offer.sdp, 'fresh-video'); + assert.equal(h.state.window.__elastosBrowserRestoreBoot.attachPromise, null); +}); + +test('boot attach promise is consumed once and skips a second webrtc POST', async () => { + const h = harness(); + const request = { + type: 'display_attach', + request_id: 'a'.repeat(32), + display_generation: 'display:' + '1'.repeat(32), + }; + const result = { + schema: 'elastos.browser.display-attach-result/v1', + page_id: 'page-one', + request_id: request.request_id, + previous_display_generation: request.display_generation, + display_generation: 'display:' + '2'.repeat(32), + initial_offer: { schema: 'elastos.browser.webrtc-offer/v1', type: 'offer', sdp: 'boot-video' }, + audio_offer: { schema: 'elastos.browser.webrtc-offer/v1', type: 'offer', sdp: 'boot-audio' }, + }; + h.state.window = { + __elastosBrowserRestoreBoot: { + attachPromise: Promise.resolve({ request, result, page_id: 'page-one' }), + }, + }; + await h.restore(summary()); + assert.deepEqual(h.requests, ['/api/apps/browser/pages/page-one/status']); + const connection = h.calls.find(row => row[0] === 'connect'); + assert.equal(connection[3].initial_offer.sdp, 'boot-video'); + assert.equal(h.state.window.__elastosBrowserRestoreBoot.attachPromise, null); +}); + test('startup restores exact owner, fresh URL, and Runtime-selected services without opening or closing', async () => { const h = harness(), original = summary(); assert.equal(await h.restore(original), true); @@ -91,8 +238,9 @@ test('restored page becomes visible to Home after its fresh address is applied', } finally { pending.resolve(); await restoring; } const url = h.calls.findIndex(row => row[0] === 'url'); const publish = h.calls.findIndex(row => row[0] === 'publish'); + const connect = h.calls.findIndex(row => row[0] === 'connect'); assert.ok(url >= 0 && publish > url); - assert.ok(h.calls.findIndex(row => row[0] === 'connect') > publish); + assert.ok(connect >= 0); }); test('Automatic Engine and local Exit remain the Runtime-selected empty values', async () => { @@ -169,11 +317,10 @@ for (const owned of [true, false]) test(`actual startup selects ${owned ? 'resto Object.assign(h.state, { params: new URLSearchParams(), DEFAULT_URL: 'https://default.invalid/', addressInput: {}, fetchBrowserSummary: async () => value, requestRuntimeOpen: async url => opens.push(url), isAuthoritySessionError: () => false, friendlyOpenError: error => error.message }); - await vm.runInContext(source.slice(source.lastIndexOf('const initialUrl =')), h.state); + await vm.runInContext(source.slice(source.lastIndexOf('const requestedStartupUrl =')), h.state); assert.deepEqual(opens, owned ? [] : ['https://default.invalid/']); }); - test('summary authority failure reaches startup renewal without a replacement open', async () => { const h = harness(), renewals = [], opens = [], error = Object.assign(new Error('Browser authority expired'), { status: 401 }); Object.assign(h.state, { browserSummaryPromise: null, browserSummary: null, browserInstanceId: 'instance-one', @@ -184,7 +331,7 @@ test('summary authority failure reaches startup renewal without a replacement op friendlyOpenError: value => value.message, requestHomeRelaunch: value => { renewals.push(value); return true; } }); vm.runInContext(declaration('fetchBrowserSummary'), h.state); - await vm.runInContext(source.slice(source.lastIndexOf('const initialUrl =')), h.state); + await vm.runInContext(source.slice(source.lastIndexOf('const requestedStartupUrl =')), h.state); assert.deepEqual(renewals, ['Browser authority expired']); assert.deepEqual(opens, []); assert.deepEqual(h.failures, []); assert.equal(h.state.browserSummaryPromise, null); @@ -205,7 +352,7 @@ for (const flag of ['unloadCleanupStarted', 'homeWindowCloseInFlight', 'homeWind Object.assign(h.state, { params: new URLSearchParams(), DEFAULT_URL: 'https://default.invalid/', addressInput: {}, fetchBrowserSummary: async () => wait.promise, requestRuntimeOpen: async url => opens.push(url), isAuthoritySessionError: () => false, friendlyOpenError: error => error.message }); - const startup = vm.runInContext(source.slice(source.lastIndexOf('const initialUrl =')), h.state); + const startup = vm.runInContext(source.slice(source.lastIndexOf('const requestedStartupUrl =')), h.state); h.state[flag] = true; wait.resolve(value); await startup; assert.deepEqual(opens, []); assert.deepEqual(h.failures, []); assert.equal(h.state.currentPage, null); @@ -295,7 +442,7 @@ for (const flag of ['unloadCleanupStarted', 'homeWindowCloseInFlight', 'homeWind fetchBrowserSummary: async () => wait.promise, requestRuntimeOpen: async () => assert.fail('fresh open'), isAuthoritySessionError: () => true, friendlyOpenError: error => error.message, requestHomeRelaunch: value => { renewals.push(value); return true; } }); - const startup = vm.runInContext(source.slice(source.lastIndexOf('const initialUrl =')), h.state); + const startup = vm.runInContext(source.slice(source.lastIndexOf('const requestedStartupUrl =')), h.state); h.state[flag] = true; const before = h.calls.length; wait.reject(new Error('late expired authority')); await startup; assert.deepEqual(renewals, []); assert.deepEqual(h.failures, []); assert.equal(h.calls.length, before); @@ -323,7 +470,9 @@ for (const stage of ['status', 'attachment']) for (const closeState of ['in-flig if (outcome === 'failure') wait.reject(new Error('late close-time failure')); else wait.resolve(); assert.equal(await pending, true); assert.equal(attaches, 1); // Both requests started before close began. - assert.equal(h.calls.length, before); assert.deepEqual(h.failures, []); + if (stage === 'attachment') assert.equal(h.calls.length, before); + assert.equal(h.calls.some(row => row[0] === 'publish'), false); + assert.deepEqual(h.failures, []); assert.equal(h.state.currentPage.display_session.display_generation, 'display:' + '1'.repeat(32)); assert.equal(h.state.currentPage.runtime_cleanup.id, 'cleanup-one'); }); @@ -347,9 +496,11 @@ test('restore starts attachment while fresh status is pending and publishes only assert.deepEqual(started, ['status', 'attachment']); attachment.resolve(); await new Promise(resolve => setImmediate(resolve)); - assert.equal(h.calls.some(row => ['publish', 'connect'].includes(row[0])), false); + assert.equal(h.calls.some(row => row[0] === 'publish'), false); + assert.equal(h.calls.filter(row => row[0] === 'connect').length, 1); } finally { status.resolve(); attachment.resolve(); await pending; } assert.equal(h.calls.filter(row => row[0] === 'connect').length, 1); + assert.equal(h.calls.filter(row => row[0] === 'publish').length, 1); }); for (const field of ['schema', 'page_id', 'direct_network']) test(`invalid concurrent status ${field} cannot publish or connect`, async () => { @@ -360,7 +511,8 @@ for (const field of ['schema', 'page_id', 'direct_network']) test(`invalid concu return result; }; await assert.rejects(h.restore(summary())); - assert.equal(h.calls.some(row => ['publish', 'connect'].includes(row[0])), false); + assert.equal(h.calls.some(row => row[0] === 'publish'), false); + assert.equal(h.calls.some(row => row[0] === 'connect'), true); assert.equal(h.state.currentPage.runtime_cleanup.id, 'cleanup-one'); }); @@ -379,3 +531,168 @@ test('attachment failure while status is pending cannot publish after late valid assert.equal(h.calls.some(row => ['publish', 'connect'].includes(row[0])), false); assert.equal(h.state.currentPage.display_session.display_generation, 'display:' + '1'.repeat(32)); }); + +function failedAttachment(errorCode = 'display_attach_failed') { + return { + schema: 'elastos.browser.display-attachment/v1', + state: 'failed', + error_code: errorCode, + request_id: '1'.repeat(32), + previous_display_generation: 'display:' + '1'.repeat(32), + }; +} + +test('a terminal attach failure allocates a fresh request id instead of replaying the cached id', async () => { + const h = harness(), value = summary(), calls = [], fetch = h.state.fetchJson; + value.sessions.recoverable_page.display_attachment = failedAttachment(); + h.state.fetchJson = async (path, options) => { if (options?.body) calls.push(options.body); return fetch(path, options); }; + await h.restore(value); + assert.equal(calls.length, 1); + assert.match(calls[0].request_id, /^[a-f0-9]{32}$/); + assert.notEqual(calls[0].request_id, '1'.repeat(32)); + assert.equal(calls[0].display_generation, 'display:' + '1'.repeat(32)); +}); + +test('uncertain attach work keeps the same request id', async () => { + const h = harness(), value = summary(), calls = [], fetch = h.state.fetchJson; + value.sessions.recoverable_page.display_attachment = failedAttachment('display_attach_uncertain'); + h.state.fetchJson = async (path, options) => { if (options?.body) calls.push(options.body); return fetch(path, options); }; + await h.restore(value); + assert.equal(calls[0].request_id, '1'.repeat(32)); +}); + +test('boot and main viewer share one retry identity after a terminal failure', async () => { + const h = harness(), value = summary(), calls = [], fetch = h.state.fetchJson; + const shared = 'c'.repeat(32); + value.sessions.recoverable_page.display_attachment = failedAttachment(); + h.state.window.__elastosBrowserRestoreBoot = { + retryIdentity: { page_id: 'page-one', generation: 'display:' + '1'.repeat(32), request_id: shared }, + }; + h.state.fetchJson = async (path, options) => { if (options?.body) calls.push(options.body); return fetch(path, options); }; + await h.restore(value); + assert.equal(calls[0].request_id, shared); +}); + +test('viewer attach timeout stops a hung restore POST', async () => { + const h = harness(); + h.state.VIEWER_DISPLAY_ATTACH_TIMEOUT_MS = 20; + h.state.fetchJson = async (path, options) => { + if (options?.body?.type === 'display_attach') { + await new Promise((_, reject) => { + const fail = () => reject(Object.assign(new Error('The operation was aborted'), { name: 'AbortError' })); + if (options.signal?.aborted) fail(); + else options.signal?.addEventListener('abort', fail, { once: true }); + }); + } + return { schema: 'elastos.browser.page-status/v1', page_id: 'page-one', actual_url: 'https://current.invalid/form', + title: 'Retained form', direct_network: false, + display_session: { mode: 'webrtc_remote_display', width: 1280, height: 720, source: 'diagnostic-status', ice_servers: [{ credential_present: true }] } }; + }; + await assert.rejects(h.restore(summary()), /aborted|could not restore|interrupted/i); + assert.equal(h.calls.some(row => row[0] === 'connect'), false); + assert.equal(h.state.currentPage.runtime_cleanup.id, 'cleanup-one'); +}); + +test('reload after a terminal Engine failure recovers with a new request id', async () => { + const cached = '1'.repeat(32); + const seen = []; + const h = harness(), first = summary(), second = summary(); + second.sessions.recoverable_page.display_attachment = failedAttachment(); + const fetch = h.state.fetchJson; + h.state.fetchJson = async (path, options) => { + if (options?.body?.type === 'display_attach') { + seen.push(options.body.request_id); + if (options.body.request_id === cached) { + throw Object.assign(new Error('Browser display attachment failed'), { + payload: { code: 'display_attach_failed' }, + }); + } + } + return fetch(path, options); + }; + await assert.rejects(h.restore(first), /display attachment failed/); + assert.deepEqual(seen, [cached]); + h.state.currentPage = null; + h.state.restoredViewerOwner = null; + await h.restore(second); + assert.equal(seen.length, 2); + assert.notEqual(seen[1], cached); + assert.equal(h.calls.filter(row => row[0] === 'connect').length, 1); +}); + +function bootContext(summaries, harness = {}) { + const posts = []; + let summaryIndex = 0; + const cachedId = '1'.repeat(32); + const context = { + location: { search: '?browser_instance=b&page_id=page-one&display_generation=display%3A' + cachedId, hash: '#home_token=t' }, + URLSearchParams, crypto, AbortController, setTimeout, clearTimeout, Date, + console: { info() {} }, + window: {}, + fetch(url, fetchOptions) { + if (String(url).includes('/summary')) { + if (harness.hangSummary) return new Promise(() => {}); + const body = summaries[Math.min(summaryIndex, summaries.length - 1)]; + summaryIndex += 1; + return Promise.resolve({ ok: true, json: async () => body }); + } + const request = JSON.parse(fetchOptions.body); + posts.push(request); + if (harness.failCachedId && request.request_id === cachedId) { + return Promise.resolve({ ok: false, json: async () => ({}) }); + } + return Promise.resolve({ + ok: true, + json: async () => ({ + schema: 'elastos.browser.display-attach-result/v1', + page_id: 'page-one', + request_id: request.request_id, + previous_display_generation: request.display_generation, + display_generation: 'display:' + '2'.repeat(32), + initial_offer: { schema: 'elastos.browser.webrtc-offer/v1', type: 'offer', sdp: 'boot-video' }, + audio_offer: { schema: 'elastos.browser.webrtc-offer/v1', type: 'offer', sdp: 'boot-audio' }, + }), + }); + }, + }; + context.window = context; + vm.runInContext(readFileSync(new URL('../capsules/browser/browser/browser-restore-boot.js', import.meta.url), 'utf8'), vm.createContext(context)); + return { context, posts }; +} + +function bootSummary(attachment) { + return { + engine_adapter: { display_attach_supported: true }, + sessions: { recoverable_page: { + state: 'active', page_id: 'page-one', + engine_page: { display_session: { mode: 'webrtc_remote_display', display_generation: 'display:' + '1'.repeat(32) } }, + display_attachment: attachment, + } }, + }; +} + +test('known URL attach posts before a delayed summary returns', async () => { + const { context, posts } = bootContext([bootSummary(null)], { hangSummary: true }); + const attached = await context.window.__elastosBrowserRestoreBoot.attachPromise; + assert.equal(posts.length, 1); + assert.equal(posts[0].request_id, '1'.repeat(32)); + assert.equal(attached.request.request_id, posts[0].request_id); +}); + +test('restore boot posts a fresh request id after a terminal attach failure', async () => { + const { context, posts } = bootContext([bootSummary(failedAttachment())], { failCachedId: true }); + const attached = await context.window.__elastosBrowserRestoreBoot.attachPromise; + assert.equal(posts.length, 2); + assert.equal(posts[0].request_id, '1'.repeat(32)); + assert.notEqual(posts[1].request_id, '1'.repeat(32)); + assert.match(posts[1].request_id, /^[a-f0-9]{32}$/); + assert.equal(attached.request.request_id, posts[1].request_id); + assert.equal(context.window.__elastosBrowserRestoreBoot.retryIdentity.request_id, posts[1].request_id); +}); + +test('restore boot reuses an uncertain request id', async () => { + const { posts } = bootContext([bootSummary(failedAttachment('display_attach_uncertain'))]); + await new Promise(resolve => setImmediate(resolve)); + await new Promise(resolve => setImmediate(resolve)); + assert.equal(posts[0].request_id, '1'.repeat(32)); +}); diff --git a/scripts/browser-vm-control-service-settlement-smoke.sh b/scripts/browser-vm-control-service-settlement-smoke.sh index 34c935780..a3c145ca3 100755 --- a/scripts/browser-vm-control-service-settlement-smoke.sh +++ b/scripts/browser-vm-control-service-settlement-smoke.sh @@ -13,6 +13,12 @@ cleanup() { kill "$service_pid" >/dev/null 2>&1 || true wait "$service_pid" 2>/dev/null || true fi + if [[ -n "${OWNER_PID_FILE:-}" && -f "${OWNER_PID_FILE:-}" ]]; then + owner_pid="$(tr -d '[:space:]' < "$OWNER_PID_FILE")" + if [[ "$owner_pid" =~ ^[0-9]+$ ]]; then + kill "$owner_pid" >/dev/null 2>&1 || true + fi + fi shopt -s nullglob for pid_file in "$proof_dir"/*.pid; do child_pid="$(tr -d '[:space:]' < "$pid_file")" @@ -27,6 +33,7 @@ trap cleanup EXIT fake_launcher="$tmp_dir/fake-settlement-launcher.mjs" cat > "$fake_launcher" <<'NODE' #!/usr/bin/env node +import { spawn } from "node:child_process"; import fs from "node:fs"; import http from "node:http"; @@ -39,6 +46,48 @@ const suffix = launch.stream_id.replace(/[^A-Za-z0-9_-]/g, "_"); const pageId = launch.page_id || `page:settlement-${suffix}`; const controlSocketPath = `${proofDir}/${suffix}.sock`; const pidPath = `${proofDir}/${suffix}.pid`; +if (process.env.POLLUTE_STDOUT === "1") { + process.stdout.write("Allocated inode: 728\n"); +} +if (process.env.HOLD_AFTER_POLLUTE === "1" && launch.transport_authority) { + const writeHoldSettlement = () => { + process.stderr.write(`${JSON.stringify({ + schema: "elastos.browser.vz-launch-settlement/v1", + state: "terminal_post_effect_cleanup", + message: "polluted stdout hold reaped", + binding_hash: launch.transport_authority.binding_hash, + generation: launch.transport_authority.generation, + page_id: launch.transport_authority.page_id, + vm_id: launch.transport_authority.vm_id, + stream_id: launch.transport_authority.egress.stream_id, + media_stream_id: launch.transport_authority.media.stream_id, + effects: { + session_directory: true, + control_socket: true, + ordinary_stream_bridge: true, + media_stream_bridge: true, + turn_process: true, + supervisor_child: true, + vm: true, + }, + absence: { + child_absent: true, + supervisor_child_absent: true, + control_socket_absent: true, + route_absent: true, + turn_listener_absent: true, + turn_relay_ports_absent: true, + ordinary_stream_bridge_absent: true, + media_stream_bridge_absent: true, + session_directory_absent: true, + vm_absent: true, + }, + })}\n`); + process.exit(0); + }; + process.once("SIGTERM", writeHoldSettlement); + setInterval(() => {}, 60_000); +} else { const typedFailure = process.env.TYPED_TRANSPORT_FAILURE; if (typedFailure && launch.transport_authority) { const acted = typedFailure !== "did_not_act"; @@ -86,6 +135,28 @@ fs.writeFileSync(pidPath, `${process.pid}\n`, { mode: 0o600 }); if (process.env.FAIL_TRANSPORT_LAUNCH === "1" && launch.transport_authority) { process.exit(23); } +if (process.env.LOG_STARTED_OWNER_HOLD === "1" && launch.transport_authority) { + process.stdout.write("Allocated inode: 728\n"); + process.stderr.write(`${JSON.stringify({ + schema: "elastos.browser.media-diagnostic/v1", + event: "turn_process_started", + binding_hash: launch.transport_authority.binding_hash, + generation: launch.transport_authority.generation, + page_id: launch.transport_authority.page_id, + vm_id: launch.transport_authority.vm_id, + media_stream_id: launch.transport_authority.media.stream_id, + ordinal: 0, + })}\n`); + const holder = spawn(process.execPath, ["-e", "setInterval(() => {}, 60000)"], { + detached: true, + stdio: "ignore", + }); + holder.unref(); + if (process.env.OWNER_PID_FILE) { + fs.writeFileSync(process.env.OWNER_PID_FILE, `${holder.pid}\n`, { mode: 0o600 }); + } + process.exit(23); +} if (process.env.MEDIA_DIAGNOSTIC_SMOKE === "1" && launch.transport_authority) { const diagnostic = { schema: "elastos.browser.media-diagnostic/v1", @@ -219,7 +290,53 @@ guest.listen(controlSocketPath, () => { direct_network: false, }, })}\n`); + const afterReady = process.env.TYPED_TRANSPORT_AFTER_READY; + if (afterReady && launch.transport_authority) { + const childAbsent = process.env.TYPED_TRANSPORT_CHILD_ABSENT !== "0"; + const delayedPorts = process.env.TYPED_TRANSPORT_DELAYED_PORTS === "1"; + const acted = afterReady !== "did_not_act"; + const profileDurability = process.env.TYPED_TRANSPORT_PROFILE_DURABILITY || ""; + process.stderr.write(`${JSON.stringify({ + schema: "elastos.browser.vz-launch-settlement/v1", + state: afterReady, + message: delayedPorts + ? "injected delayed-port-only settlement" + : `injected after-ready ${afterReady}`, + binding_hash: launch.transport_authority.binding_hash, + generation: launch.transport_authority.generation, + page_id: launch.transport_authority.page_id, + vm_id: launch.transport_authority.vm_id, + stream_id: launch.transport_authority.egress.stream_id, + media_stream_id: launch.transport_authority.media.stream_id, + effects: { + session_directory: acted, + control_socket: acted, + ordinary_stream_bridge: acted, + media_stream_bridge: acted, + turn_process: acted, + supervisor_child: acted, + vm: acted, + }, + absence: { + child_absent: childAbsent, + supervisor_child_absent: childAbsent, + control_socket_absent: true, + route_absent: true, + turn_listener_absent: delayedPorts ? false : true, + turn_relay_ports_absent: delayedPorts ? false : true, + ordinary_stream_bridge_absent: true, + media_stream_bridge_absent: true, + session_directory_absent: true, + vm_absent: childAbsent, + }, + ...(profileDurability + ? { profile_durability: profileDurability } + : {}), + })}\n`); + process.exit(1); + } }); +} NODE chmod +x "$fake_launcher" @@ -255,6 +372,9 @@ const config = { shutdown_timeout_ms: 1000, }; if (shutdownProgram) config.shutdown_program = shutdownProgram; +if (/^[0-9a-f]{64}$/.test(process.env.CONFIG_FINGERPRINT || "")) { + config.config_fingerprint = process.env.CONFIG_FINGERPRINT; +} process.stdout.write(JSON.stringify(config)); NODE } @@ -272,8 +392,17 @@ start_service() { LAUNCH_MARKER_PATH="$launch_marker" \ FAIL_TRANSPORT_LAUNCH="${FAIL_TRANSPORT_LAUNCH:-}" \ TYPED_TRANSPORT_FAILURE="${TYPED_TRANSPORT_FAILURE:-}" \ + TYPED_TRANSPORT_AFTER_READY="${TYPED_TRANSPORT_AFTER_READY:-}" \ + TYPED_TRANSPORT_CHILD_ABSENT="${TYPED_TRANSPORT_CHILD_ABSENT:-}" \ + TYPED_TRANSPORT_DELAYED_PORTS="${TYPED_TRANSPORT_DELAYED_PORTS:-}" \ + TYPED_TRANSPORT_PROFILE_DURABILITY="${TYPED_TRANSPORT_PROFILE_DURABILITY:-}" \ TYPED_TRANSPORT_SUBSTITUTE="${TYPED_TRANSPORT_SUBSTITUTE:-}" \ + LOG_STARTED_OWNER_HOLD="${LOG_STARTED_OWNER_HOLD:-}" \ + OWNER_PID_FILE="${OWNER_PID_FILE:-}" \ + ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG="${ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG:-}" \ MEDIA_DIAGNOSTIC_SMOKE="${MEDIA_DIAGNOSTIC_SMOKE:-}" \ + POLLUTE_STDOUT="${POLLUTE_STDOUT:-}" \ + HOLD_AFTER_POLLUTE="${HOLD_AFTER_POLLUTE:-}" \ "$node_bin" "$repo_root/scripts/browser-vm-control-service.mjs" \ > "$tmp_dir/${label}.out" 2> "$tmp_dir/${label}.err" & service_pid=$! @@ -295,9 +424,11 @@ stop_service() { client="$tmp_dir/settlement-client.mjs" cat > "$client" <<'NODE' import crypto from "node:crypto"; +import dgram from "node:dgram"; import fs from "node:fs"; import http from "node:http"; import net from "node:net"; +import path from "node:path"; const socketPath = process.env.CONTROL_SOCKET; const streamId = process.env.STREAM_ID; @@ -549,6 +680,24 @@ async function waitFor(predicate, message) { throw new Error(message); } +async function waitForReconcile(id, check, message) { + const deadline = Date.now() + 5000; + let last = null; + while (Date.now() < deadline) { + last = await reconcile(id); + if (check(last)) return last; + await new Promise((resolve) => setTimeout(resolve, 25)); + } + throw new Error(`${message}: ${JSON.stringify(last)}`); +} + +function removeSessionDir(page) { + const sessionDir = page?.isolation?.session_dir; + if (typeof sessionDir === "string" && sessionDir.startsWith("/")) { + fs.rmSync(sessionDir, { recursive: true, force: true }); + } +} + async function requireBindingRejected(page, label, mutate) { const body = runtimeSerializedCloseBody(page); mutate(body.runtime_cleanup); @@ -662,6 +811,28 @@ if (process.env.PHASE === "binding-equality") { if (afterExit.state !== "cleanup_pending") { throw new Error(`launcher exit synthesized terminal cleanup: ${JSON.stringify(afterExit)}`); } + let afterExitStatus = null; + const exitDeadline = Date.now() + 5000; + while (Date.now() < exitDeadline) { + afterExitStatus = await request("GET", "/status"); + if ( + afterExitStatus.active_pages === 0 && + afterExitStatus.capacity_available === true && + afterExitStatus.pending_cleanup_pages === 1 + ) { + break; + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } + if ( + afterExitStatus?.active_pages !== 0 || + afterExitStatus?.capacity_available !== true || + afterExitStatus?.pending_cleanup_pages !== 1 + ) { + throw new Error( + `launcher exit kept live capacity: ${JSON.stringify(afterExitStatus)}`, + ); + } const second = await request("POST", "/shutdown", closeBody(page)); if ( second.terminal !== true || @@ -681,6 +852,240 @@ if (process.env.PHASE === "binding-equality") { if (alreadyAbsent.terminal !== true || alreadyAbsent.already_absent !== true) { throw new Error(`terminal retry lost already-absent proof: ${JSON.stringify(alreadyAbsent)}`); } +} else if (process.env.PHASE === "exit-then-reopen") { + const first = await request("POST", "/pages", openBody(streamId)); + const live = await request("GET", "/status"); + if ( + live.active_pages !== 1 || + live.capacity_available !== false || + live.pending_cleanup_pages !== 0 + ) { + throw new Error(`live page did not occupy capacity: ${JSON.stringify(live)}`); + } + process.kill(first.process.pid, "SIGTERM"); + await waitFor( + () => !processAlive(first.process.pid), + "exit-then-reopen child did not exit", + ); + let afterExit = null; + const exitDeadline = Date.now() + 5000; + while (Date.now() < exitDeadline) { + afterExit = await request("GET", "/status"); + if ( + afterExit.active_pages === 0 && + afterExit.capacity_available === true && + afterExit.pending_cleanup_pages === 1 + ) { + break; + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } + if ( + afterExit?.active_pages !== 0 || + afterExit?.capacity_available !== true || + afterExit?.pending_cleanup_pages !== 1 + ) { + throw new Error( + `VM exit did not free live capacity: ${JSON.stringify(afterExit)}`, + ); + } + const pending = await reconcile(streamId); + if (pending.state !== "cleanup_pending") { + throw new Error(`VM exit synthesized terminal cleanup: ${JSON.stringify(pending)}`); + } + const closed = await request("POST", "/shutdown", closeBody(first)); + if ( + closed.terminal !== true || + Object.values(closed.effects || {}).some((value) => value !== true) + ) { + throw new Error(`post-exit close was not terminal: ${JSON.stringify(closed)}`); + } + const afterClose = await request("GET", "/status"); + if ( + afterClose.active_pages !== 0 || + afterClose.pending_cleanup_pages !== 0 || + afterClose.capacity_available !== true + ) { + throw new Error(`terminal close left pending capacity: ${JSON.stringify(afterClose)}`); + } + const reopenId = `${streamId}-reopen`; + const second = await request("POST", "/pages", openBody(reopenId)); + const reopened = await request("GET", "/status"); + if ( + reopened.active_pages !== 1 || + reopened.capacity_available !== false || + second.page_id === first.page_id + ) { + throw new Error(`fresh open after VM exit failed: ${JSON.stringify({ second, reopened })}`); + } + const secondClose = await request("POST", "/shutdown", closeBody(second)); + if (secondClose.terminal !== true) { + throw new Error(`reopened page close was not terminal: ${JSON.stringify(secondClose)}`); + } +} else if (process.env.PHASE === "ungraceful-transport-close") { + const first = await request("POST", "/pages", openBody(streamId)); + process.kill(first.process.pid, "SIGKILL"); + await waitFor( + () => !processAlive(first.process.pid), + "ungraceful-transport-close child did not exit", + ); + let afterExit = null; + const exitDeadline = Date.now() + 5000; + while (Date.now() < exitDeadline) { + afterExit = await request("GET", "/status"); + if ( + afterExit.active_pages === 0 && + afterExit.capacity_available === true && + afterExit.pending_cleanup_pages === 1 + ) { + break; + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } + if ( + afterExit?.active_pages !== 0 || + afterExit?.capacity_available !== true || + afterExit?.pending_cleanup_pages !== 1 + ) { + throw new Error( + `ungraceful VM exit did not free live capacity: ${JSON.stringify(afterExit)}`, + ); + } + const pending = await reconcile(streamId); + if (pending.state !== "cleanup_pending") { + throw new Error( + `ungraceful VM exit synthesized terminal cleanup: ${JSON.stringify(pending)}`, + ); + } + const closed = await requestRaw("POST", "/shutdown", closeBody(first)); + if ( + closed.status !== 400 || + !String(closed.body.error || "").includes("turn_process_absent") + ) { + throw new Error( + `ungraceful post-exit close synthesized terminal TURN process absence: ${JSON.stringify(closed)}`, + ); + } + const afterClose = await request("GET", "/status"); + if ( + afterClose.active_pages !== 0 || + afterClose.pending_cleanup_pages !== 1 || + afterClose.capacity_available !== true + ) { + throw new Error( + `ungraceful post-exit close released pending ownership: ${JSON.stringify(afterClose)}`, + ); + } +} else if (process.env.PHASE === "exit-then-reopen-before-cleanup") { + if (transportEnabled) { + throw new Error("exit-then-reopen-before-cleanup is a local same-profile phase"); + } + const first = await request("POST", "/pages", openBody(streamId)); + const live = await request("GET", "/status"); + if ( + live.active_pages !== 1 || + live.capacity_available !== false || + live.pending_cleanup_pages !== 0 + ) { + throw new Error(`live page did not occupy capacity: ${JSON.stringify(live)}`); + } + process.kill(first.process.pid, "SIGTERM"); + await waitFor( + () => !processAlive(first.process.pid), + "exit-then-reopen-before-cleanup child did not exit", + ); + let afterExit = null; + const exitDeadline = Date.now() + 5000; + while (Date.now() < exitDeadline) { + afterExit = await request("GET", "/status"); + if ( + afterExit.active_pages === 0 && + afterExit.capacity_available === true && + afterExit.pending_cleanup_pages === 1 && + afterExit.pending_cleanup_page_ids?.[0] === first.page_id + ) { + break; + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } + if ( + afterExit?.active_pages !== 0 || + afterExit?.capacity_available !== true || + afterExit?.pending_cleanup_pages !== 1 || + afterExit?.pending_cleanup_page_ids?.[0] !== first.page_id + ) { + throw new Error( + `VM exit did not free live capacity: ${JSON.stringify(afterExit)}`, + ); + } + const pending = await reconcile(streamId); + if (pending.state !== "cleanup_pending") { + throw new Error(`VM exit synthesized terminal cleanup: ${JSON.stringify(pending)}`); + } + const reopenId = `${streamId}-reopen`; + const second = await request("POST", "/pages", openBody(reopenId)); + const overlapped = await request("GET", "/status"); + if ( + second.page_id === first.page_id || + overlapped.active_pages !== 1 || + overlapped.capacity_available !== false || + overlapped.pending_cleanup_pages !== 1 || + !overlapped.page_ids?.includes(second.page_id) || + overlapped.page_ids?.includes(first.page_id) || + !overlapped.pending_cleanup_page_ids?.includes(first.page_id) || + overlapped.pending_cleanup_page_ids?.includes(second.page_id) + ) { + throw new Error( + `same-profile reopen revived the old pending page: ${JSON.stringify({ second, overlapped })}`, + ); + } + const oldStatus = await requestRaw( + "GET", + `/pages/${encodeURIComponent(first.page_id)}/status`, + ); + if ( + oldStatus.status !== 404 || + !String(oldStatus.body.error || "").includes("cleanup is pending") + ) { + throw new Error( + `old pending page did not stay cleanup-owned: ${JSON.stringify(oldStatus)}`, + ); + } + if (!processAlive(second.process.pid) || !fs.existsSync(second.control_socket_path)) { + throw new Error("new same-profile page was not intact before old cleanup"); + } + const closedOld = await request("POST", "/shutdown", closeBody(first)); + if ( + closedOld.terminal !== true || + Object.values(closedOld.effects || {}).some((value) => value !== true) + ) { + throw new Error(`old-owner cleanup was not terminal: ${JSON.stringify(closedOld)}`); + } + const afterOldClose = await request("GET", "/status"); + if ( + afterOldClose.active_pages !== 1 || + afterOldClose.capacity_available !== false || + afterOldClose.pending_cleanup_pages !== 0 || + !afterOldClose.page_ids?.includes(second.page_id) || + afterOldClose.page_ids?.includes(first.page_id) + ) { + throw new Error( + `old-owner cleanup disturbed the new page: ${JSON.stringify(afterOldClose)}`, + ); + } + if (!processAlive(second.process.pid) || !fs.existsSync(second.control_socket_path)) { + throw new Error("old-owner cleanup retired the new same-profile VM"); + } + const terminalOld = await reconcile(streamId); + if (terminalOld.state !== "terminal_post_effect_cleanup") { + throw new Error( + `old-owner cleanup did not persist terminal state: ${JSON.stringify(terminalOld)}`, + ); + } + const secondClose = await request("POST", "/shutdown", closeBody(second)); + if (secondClose.terminal !== true) { + throw new Error(`new page close was not terminal: ${JSON.stringify(secondClose)}`); + } } else if (process.env.PHASE === "open-for-restart") { const page = await request("POST", "/pages", openBody(streamId)); fs.writeFileSync(process.env.PAGE_FILE, JSON.stringify(page), { mode: 0o600 }); @@ -737,6 +1142,46 @@ if (process.env.PHASE === "binding-equality") { if ((await reconcile(streamId)).state !== "cleanup_pending") { throw new Error("stale process identity did not remain pending"); } +} else if (process.env.PHASE === "verify-transport-restart") { + const page = JSON.parse(fs.readFileSync(process.env.PAGE_FILE, "utf8")); + issuedTransportAuthority = page.transport_authority || null; + const pending = await reconcile(streamId); + if ( + pending.state !== "cleanup_pending" || + pending.cleanup_binding?.page_id !== page.page_id || + pending.supervisor_result !== undefined + ) { + throw new Error( + `transport restart did not retain only its exact durable cleanup binding: ${JSON.stringify(pending)}`, + ); + } + if (!processAlive(page.process.pid)) { + throw new Error("transport restart lost its surviving owned launcher"); + } + if (fs.existsSync(page.control_socket_path)) { + fs.unlinkSync(page.control_socket_path); + } + const surviving = await requestRaw( + "POST", + "/shutdown", + runtimeSerializedCloseBody(page), + ); + if ( + surviving.status !== 400 || + !String(surviving.body.error || "").includes( + "exact owned launcher unavailable", + ) + ) { + throw new Error( + `surviving transport owner was synthesized terminal after restart: ${JSON.stringify(surviving)}`, + ); + } + if (!processAlive(page.process.pid)) { + throw new Error("transport restart cleanup disturbed the surviving owned launcher"); + } + if ((await reconcile(streamId)).state !== "cleanup_pending") { + throw new Error("surviving transport owner did not remain pending"); + } } else if (process.env.PHASE === "capacity") { const before = fs.readFileSync(process.env.JOURNAL_PATH); const beforeHash = crypto.createHash("sha256").update(before).digest("hex"); @@ -842,6 +1287,59 @@ if (process.env.PHASE === "binding-equality") { unexpectedTurnListener.close(resolve), ); + const unexpectedUdpRelay = dgram.createSocket({ + type: "udp4", + reuseAddr: false, + }); + await new Promise((resolve, reject) => { + unexpectedUdpRelay.once("error", reject); + unexpectedUdpRelay.bind( + { + address: page.transport_authority.turn.relay_host, + port: page.transport_authority.turn.relay_port_min, + exclusive: true, + }, + resolve, + ); + }); + const occupiedRelay = await requestRaw( + "POST", + "/shutdown", + closeBody(page), + ); + if ( + occupiedRelay.status !== 400 || + !String(occupiedRelay.body.error || "").includes( + "turn_relay_ports_absent", + ) + ) { + throw new Error( + `occupied UDP relay port did not retain cleanup ownership: ${JSON.stringify(occupiedRelay)}`, + ); + } + await new Promise((resolve) => unexpectedUdpRelay.close(resolve)); + + const listenUnix = (path) => + new Promise((resolve, reject) => { + try { + fs.unlinkSync(path); + } catch (error) { + if (error?.code !== "ENOENT") { + reject(error); + return; + } + } + const server = net.createServer(); + server.once("error", reject); + server.listen(path, () => resolve(server)); + }); + const runtimeEgress = await listenUnix( + page.transport_authority.egress.runtime_socket_path, + ); + const runtimeMedia = await listenUnix( + page.transport_authority.media.runtime_socket_path, + ); + const terminal = await request("POST", "/shutdown", closeBody(page)); const requiredEffects = [ "transport_session_absent", @@ -877,6 +1375,16 @@ if (process.env.PHASE === "binding-equality") { `transport terminal cleanup was not durable: ${JSON.stringify(durable)}`, ); } + if ( + !fs.existsSync(page.transport_authority.egress.runtime_socket_path) || + !fs.existsSync(page.transport_authority.media.runtime_socket_path) + ) { + throw new Error( + "Runtime stream sockets remaining after owned launcher cleanup were unlinked by helper", + ); + } + await new Promise((resolve) => runtimeEgress.close(resolve)); + await new Promise((resolve) => runtimeMedia.close(resolve)); const journalAfter = fs.readFileSync(process.env.JOURNAL_PATH, "utf8"); if ( journalAfter.includes('"auth_secret":') || @@ -1004,6 +1512,342 @@ if (process.env.PHASE === "binding-equality") { `substituted transport settlement escaped cleanup ownership: ${JSON.stringify(durable)}`, ); } +} else if (process.env.PHASE === "polluted-stdout-ready") { + const page = await request("POST", "/pages", openBody(streamId)); + if (page.schema !== "elastos.browser.engine.supervisor-result/v1") { + throw new Error( + `polluted stdout hid the supervisor result: ${JSON.stringify(page)}`, + ); + } + const close = await request("POST", "/shutdown", closeBody(page)); + if (close.terminal !== true) { + throw new Error(`polluted-stdout ready close was not terminal: ${JSON.stringify(close)}`); + } +} else if (process.env.PHASE === "polluted-stdout-hold") { + const failed = await requestRaw("POST", "/pages", openBody(streamId)); + if ( + failed.status !== 400 || + failed.body.launch_settlement_result?.state !== + "terminal_post_effect_cleanup" || + failed.body.launch_settlement_result?.binding_hash !== + issuedTransportAuthority.binding_hash + ) { + throw new Error( + `polluted stdout hold did not reap a typed settlement: ${JSON.stringify(failed)}`, + ); + } + const durable = await reconcile(streamId); + if ( + durable.state !== "terminal_post_effect_cleanup" || + durable.launch_settlement_result?.binding_hash !== + issuedTransportAuthority.binding_hash + ) { + throw new Error( + `polluted stdout hold settlement was not durable: ${JSON.stringify(durable)}`, + ); + } +} else if (process.env.PHASE === "delayed-turn-child-present") { + const page = await request("POST", "/pages", openBody(streamId)); + issuedTransportAuthority = page.transport_authority || null; + if (process.env.PAGE_FILE) { + fs.writeFileSync(process.env.PAGE_FILE, JSON.stringify(page)); + } + await waitFor( + () => !processAlive(page.process.pid), + "after-ready child with child_absent=false did not exit", + ); + const pending = await waitForReconcile( + streamId, + (record) => + record.state === "cleanup_pending" && + record.cleanup_binding?.page_id === page.page_id && + record.launch_settlement_result?.absence?.child_absent === false && + record.launch_settlement_result?.effects?.turn_process === true, + "after-ready child_absent=false settlement was not durable", + ); + removeSessionDir(page); + const closed = await requestRaw( + "POST", + "/shutdown", + runtimeSerializedCloseBody(page), + ); + if (closed.status !== 400 || closed.body?.terminal === true) { + throw new Error( + `delayed turn with child_absent=false became terminal: ${JSON.stringify(closed)}`, + ); + } + const durable = await reconcile(streamId); + if ( + durable.state !== "cleanup_pending" || + durable.launch_settlement_result?.absence?.child_absent !== false || + durable.launch_settlement_result?.binding_hash !== + pending.launch_settlement_result?.binding_hash + ) { + throw new Error( + `delayed turn with child_absent=false did not remain pending: ${JSON.stringify(durable)}`, + ); + } +} else if (process.env.PHASE === "verify-delayed-turn-child-present-restart") { + const page = JSON.parse(fs.readFileSync(process.env.PAGE_FILE, "utf8")); + issuedTransportAuthority = page.transport_authority || null; + removeSessionDir(page); + const pending = await reconcile(streamId); + if ( + pending.state !== "cleanup_pending" || + pending.cleanup_binding?.page_id !== page.page_id || + pending.launch_settlement_result?.absence?.child_absent !== false + ) { + throw new Error( + `restart lost child_absent=false cleanup ownership: ${JSON.stringify(pending)}`, + ); + } + const closed = await requestRaw( + "POST", + "/shutdown", + runtimeSerializedCloseBody(page), + ); + if (closed.status !== 400 || closed.body?.terminal === true) { + throw new Error( + `restart delayed turn with child_absent=false became terminal: ${JSON.stringify(closed)}`, + ); + } + if ((await reconcile(streamId)).state !== "cleanup_pending") { + throw new Error("restart delayed turn with child_absent=false did not remain pending"); + } +} else if (process.env.PHASE === "delayed-turn-port-only") { + const page = await request("POST", "/pages", openBody(streamId)); + issuedTransportAuthority = page.transport_authority || null; + if (process.env.PAGE_FILE) { + fs.writeFileSync(process.env.PAGE_FILE, JSON.stringify(page)); + } + await waitFor( + () => !processAlive(page.process.pid), + "delayed-port-only child did not exit", + ); + await waitForReconcile( + streamId, + (record) => + record.state === "cleanup_pending" && + record.cleanup_binding?.page_id === page.page_id && + record.launch_settlement_result?.absence?.child_absent === true && + record.launch_settlement_result?.absence?.turn_listener_absent === false && + record.launch_settlement_result?.absence?.turn_relay_ports_absent === false && + (!process.env.TYPED_TRANSPORT_PROFILE_DURABILITY || + record.profile_durability === + process.env.TYPED_TRANSPORT_PROFILE_DURABILITY || + record.launch_settlement_result?.profile_durability === + process.env.TYPED_TRANSPORT_PROFILE_DURABILITY), + "delayed-port-only settlement was not durable", + ); +} else if (process.env.PHASE === "verify-delayed-turn-port-only-restart") { + const page = JSON.parse(fs.readFileSync(process.env.PAGE_FILE, "utf8")); + issuedTransportAuthority = page.transport_authority || null; + removeSessionDir(page); + const pending = await reconcile(streamId); + if ( + pending.state !== "cleanup_pending" || + pending.launch_settlement_result?.absence?.child_absent !== true || + pending.launch_settlement_result?.absence?.turn_listener_absent !== false + ) { + throw new Error( + `restart lost delayed-port-only cleanup ownership: ${JSON.stringify(pending)}`, + ); + } + const closed = await request("POST", "/shutdown", runtimeSerializedCloseBody(page)); + if ( + closed.terminal !== true || + closed.delayed_turn_port_absence !== true + ) { + throw new Error( + `delayed-port-only restart close was not delayed terminal: ${JSON.stringify(closed)}`, + ); + } + const durable = await reconcile(streamId); + if ( + durable.state !== "terminal_post_effect_cleanup" || + durable.terminal_cleanup_receipt?.delayed_turn_port_absence !== true + ) { + throw new Error( + `delayed-port-only restart close was not durable: ${JSON.stringify(durable)}`, + ); + } +} else if (process.env.PHASE === "log-started-paths-gone-owner-alive") { + const failed = await requestRaw("POST", "/pages", openBody(streamId)); + if (failed.status !== 400) { + throw new Error( + `logged started launch did not fail closed: ${JSON.stringify(failed)}`, + ); + } + const ownerPid = Number( + String(fs.readFileSync(process.env.OWNER_PID_FILE, "utf8")).trim(), + ); + if (!Number.isInteger(ownerPid) || !processAlive(ownerPid)) { + throw new Error("logged started owner is not alive before reconcile"); + } + const digest = String(issuedTransportAuthority.binding_hash) + .replace(/^sha256:/, "") + .toLowerCase(); + const segment = digest.slice(0, 32); + const sessionDir = path.join( + process.env.ELASTOS_BROWSER_VM_ROOT || "/tmp/evzs", + `vz-${segment}`, + ); + const socketDir = path.join( + process.env.ELASTOS_BROWSER_VM_SOCKET_ROOT || "/tmp/evzrc", + segment, + ); + fs.mkdirSync(sessionDir, { recursive: true, mode: 0o700 }); + fs.mkdirSync(socketDir, { recursive: true, mode: 0o700 }); + fs.writeFileSync(path.join(socketDir, "c.sock"), ""); + fs.rmSync(sessionDir, { recursive: true, force: true }); + fs.rmSync(socketDir, { recursive: true, force: true }); + const durable = await reconcile(streamId); + if (durable.state !== "cleanup_pending") { + throw new Error( + `logged started launch with a live owner became terminal: ${JSON.stringify(durable)}`, + ); + } + if (!processAlive(ownerPid)) { + throw new Error("logged started owner was reaped during manufactured settlement"); + } +} else if (process.env.PHASE === "log-started-owner-exits-without-native-settlement") { + const failed = await requestRaw("POST", "/pages", openBody(streamId)); + if (failed.status !== 400) { + throw new Error( + `logged started launch did not fail closed: ${JSON.stringify(failed)}`, + ); + } + const digest = String(issuedTransportAuthority.binding_hash) + .replace(/^sha256:/, "") + .toLowerCase(); + const segment = digest.slice(0, 32); + const sessionDir = path.join( + process.env.ELASTOS_BROWSER_VM_ROOT || "/tmp/evzs", + `vz-${segment}`, + ); + const socketDir = path.join( + process.env.ELASTOS_BROWSER_VM_SOCKET_ROOT || "/tmp/evzrc", + segment, + ); + const controlSocket = path.join(socketDir, "c.sock"); + fs.mkdirSync(sessionDir, { recursive: true, mode: 0o700 }); + fs.mkdirSync(socketDir, { recursive: true, mode: 0o700 }); + fs.writeFileSync( + path.join(socketDir, "owner.json"), + `${JSON.stringify({ + schema: "elastos.browser.vz-socket-owner/v1", + binding_hash: issuedTransportAuthority.binding_hash, + generation: issuedTransportAuthority.generation, + page_id: issuedTransportAuthority.page_id, + vm_id: issuedTransportAuthority.vm_id, + stream_id: issuedTransportAuthority.egress.stream_id, + media_stream_id: issuedTransportAuthority.media.stream_id, + })}\n`, + { mode: 0o600 }, + ); + const holder = net.createServer(); + await new Promise((resolve, reject) => { + holder.once("error", reject); + holder.listen(controlSocket, resolve); + }); + const live = await reconcile(streamId); + if ( + live.state !== "cleanup_pending" || + live.launch_settlement_result?.state === "terminal_post_effect_cleanup" + ) { + throw new Error( + `once-live owner manufactured a native settlement: ${JSON.stringify(live)}`, + ); + } + await new Promise((resolve) => holder.close(resolve)); + fs.rmSync(sessionDir, { recursive: true, force: true }); + fs.rmSync(socketDir, { recursive: true, force: true }); + const durable = await reconcile(streamId); + if ( + durable.state !== "cleanup_pending" || + durable.launch_settlement_result?.state === "terminal_post_effect_cleanup" + ) { + throw new Error( + `exited owner without a native settlement became terminal: ${JSON.stringify(durable)}`, + ); + } +} else if (process.env.PHASE === "native-terminal-then-ordinary-close") { + const page = await request("POST", "/pages", openBody(streamId)); + issuedTransportAuthority = page.transport_authority || null; + await waitFor( + () => !processAlive(page.process.pid), + "native terminal child did not exit", + ); + await waitForReconcile( + streamId, + (record) => + record.state === "terminal_post_effect_cleanup" && + record.launch_settlement_result?.state === + "terminal_post_effect_cleanup" && + record.launch_settlement_result?.absence?.child_absent === true && + record.terminal_cleanup_receipt === undefined && + (record.profile_durability === "failed" || + record.launch_settlement_result?.profile_durability === "failed"), + "native terminal settlement was not durable before ordinary close", + ); + const closed = await request("POST", "/shutdown", runtimeSerializedCloseBody(page)); + if ( + closed.terminal !== true || + closed.effects?.child_absent !== true || + closed.profile_durability !== "failed" + ) { + throw new Error( + `ordinary close after native terminal lost failed durability or child absence: ${JSON.stringify(closed)}`, + ); + } + const durable = await reconcile(streamId); + if ( + durable.state !== "terminal_post_effect_cleanup" || + durable.launch_settlement_result?.state !== + "terminal_post_effect_cleanup" || + durable.profile_durability !== "failed" || + durable.terminal_cleanup_receipt?.effects?.child_absent !== true || + durable.terminal_cleanup_receipt?.profile_durability !== "failed" + ) { + throw new Error( + `ordinary close after native terminal did not retain settlement and receipt: ${JSON.stringify(durable)}`, + ); + } +} else if (process.env.PHASE === "verify-failed-flush-durability-restart") { + const page = JSON.parse(fs.readFileSync(process.env.PAGE_FILE, "utf8")); + issuedTransportAuthority = page.transport_authority || null; + removeSessionDir(page); + const pending = await reconcile(streamId); + if ( + pending.state !== "cleanup_pending" || + pending.launch_settlement_result?.absence?.child_absent !== true || + (pending.profile_durability !== "failed" && + pending.launch_settlement_result?.profile_durability !== "failed") + ) { + throw new Error( + `restart lost failed profile durability: ${JSON.stringify(pending)}`, + ); + } + const closed = await request("POST", "/shutdown", runtimeSerializedCloseBody(page)); + if ( + closed.terminal !== true || + closed.delayed_turn_port_absence !== true || + closed.profile_durability !== "failed" + ) { + throw new Error( + `failed durability close lost the data-save failure: ${JSON.stringify(closed)}`, + ); + } + const durable = await reconcile(streamId); + if ( + durable.state !== "terminal_post_effect_cleanup" || + durable.profile_durability !== "failed" || + durable.terminal_cleanup_receipt?.effects?.child_absent !== true + ) { + throw new Error( + `failed durability reconcile lost truthful child absence: ${JSON.stringify(durable)}`, + ); + } } else { throw new Error(`unknown settlement phase: ${process.env.PHASE}`); } @@ -1025,6 +1869,40 @@ PHASE="cleanup-retry" \ "$node_bin" "$client" stop_service +reopen_socket="$tmp_dir/reopen-control.sock" +start_service "$reopen_socket" "" "reopen-service" +CONTROL_SOCKET="$reopen_socket" \ +STREAM_ID="stream:settlement-exit-then-reopen" \ +PHASE="exit-then-reopen" \ + "$node_bin" "$client" +stop_service + +reopen_transport_socket="$tmp_dir/reopen-transport-control.sock" +start_service "$reopen_transport_socket" "" "reopen-transport-service" +TRANSPORT=1 \ +CONTROL_SOCKET="$reopen_transport_socket" \ +STREAM_ID="stream:settlement-exit-then-reopen-transport" \ +PHASE="exit-then-reopen" \ + "$node_bin" "$client" +stop_service + +ungraceful_transport_socket="$tmp_dir/ungraceful-transport-control.sock" +start_service "$ungraceful_transport_socket" "" "ungraceful-transport-service" +TRANSPORT=1 \ +CONTROL_SOCKET="$ungraceful_transport_socket" \ +STREAM_ID="stream:settlement-ungraceful-transport-close" \ +PHASE="ungraceful-transport-close" \ + "$node_bin" "$client" +stop_service + +reopen_before_cleanup_socket="$tmp_dir/reopen-before-cleanup-control.sock" +start_service "$reopen_before_cleanup_socket" "" "reopen-before-cleanup-service" +CONTROL_SOCKET="$reopen_before_cleanup_socket" \ +STREAM_ID="stream:settlement-exit-then-reopen-before-cleanup" \ +PHASE="exit-then-reopen-before-cleanup" \ + "$node_bin" "$client" +stop_service + restart_socket="$tmp_dir/restart-control.sock" restart_page="$tmp_dir/restart-page.json" start_service "$restart_socket" "" "restart-service-first" @@ -1045,6 +1923,49 @@ PHASE="verify-restart" \ "$node_bin" "$client" stop_service +transport_restart_socket="$tmp_dir/transport-restart-control.sock" +transport_restart_page="$tmp_dir/transport-restart-page.json" +start_service "$transport_restart_socket" "" "transport-restart-service-first" +TRANSPORT=1 \ +CONTROL_SOCKET="$transport_restart_socket" \ +STREAM_ID="stream:settlement-transport-restart" \ +PAGE_FILE="$transport_restart_page" \ +PHASE="open-for-restart" \ + "$node_bin" "$client" +kill -KILL "$service_pid" +wait "$service_pid" 2>/dev/null || true +service_pid="" +rm -f "$transport_restart_socket" +start_service "$transport_restart_socket" "" "transport-restart-service-second" +TRANSPORT=1 \ +CONTROL_SOCKET="$transport_restart_socket" \ +STREAM_ID="stream:settlement-transport-restart" \ +PAGE_FILE="$transport_restart_page" \ +PHASE="verify-transport-restart" \ + "$node_bin" "$client" +stop_service + +fingerprint_socket="$tmp_dir/fingerprint-control.sock" +fingerprint_page="$tmp_dir/fingerprint-page.json" +start_service "$fingerprint_socket" "" "fingerprint-service-first" +CONTROL_SOCKET="$fingerprint_socket" \ +STREAM_ID="stream:settlement-fingerprint-restart" \ +PAGE_FILE="$fingerprint_page" \ +PHASE="open-for-restart" \ + "$node_bin" "$client" +kill -KILL "$service_pid" +wait "$service_pid" 2>/dev/null || true +service_pid="" +rm -f "$fingerprint_socket" +CONFIG_FINGERPRINT="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" \ +start_service "$fingerprint_socket" "" "fingerprint-service-second" +CONTROL_SOCKET="$fingerprint_socket" \ +STREAM_ID="stream:settlement-fingerprint-restart" \ +PAGE_FILE="$fingerprint_page" \ +PHASE="verify-restart" \ + "$node_bin" "$client" +stop_service + capacity_socket="$tmp_dir/capacity-control.sock" capacity_journal="${capacity_socket}.launch-reconciliations.json" launch_marker="$tmp_dir/capacity-launcher-ran" @@ -1169,4 +2090,184 @@ if grep -q 'must-not-reach-control-service-log' "$tmp_dir/transport-service.err" exit 1 fi +pollute_ready_socket="$tmp_dir/pollute-ready-control.sock" +POLLUTE_STDOUT=1 \ + start_service "$pollute_ready_socket" "" "pollute-ready-service" +CONTROL_SOCKET="$pollute_ready_socket" \ +STREAM_ID="stream:polluted-stdout-ready" \ +PHASE="polluted-stdout-ready" \ +TRANSPORT=1 \ + "$node_bin" "$client" +stop_service +unset POLLUTE_STDOUT + +pollute_hold_socket="$tmp_dir/pollute-hold-control.sock" +pollute_hold_journal="${pollute_hold_socket}.launch-reconciliations.json" +POLLUTE_STDOUT=1 \ +HOLD_AFTER_POLLUTE=1 \ + start_service "$pollute_hold_socket" "" "pollute-hold-service" +CONTROL_SOCKET="$pollute_hold_socket" \ +STREAM_ID="stream:polluted-stdout-hold" \ +JOURNAL_PATH="$pollute_hold_journal" \ +PHASE="polluted-stdout-hold" \ +TRANSPORT=1 \ + "$node_bin" "$client" +stop_service +unset POLLUTE_STDOUT +unset HOLD_AFTER_POLLUTE + +delayed_child_socket="$tmp_dir/delayed-turn-child-present-control.sock" +delayed_child_journal="${delayed_child_socket}.launch-reconciliations.json" +delayed_child_page="$tmp_dir/delayed-turn-child-present-page.json" +TYPED_TRANSPORT_AFTER_READY="cleanup_pending" \ +TYPED_TRANSPORT_CHILD_ABSENT="0" \ + start_service "$delayed_child_socket" "" "delayed-turn-child-present-service" +CONTROL_SOCKET="$delayed_child_socket" \ +STREAM_ID="stream:delayed-turn-child-present" \ +JOURNAL_PATH="$delayed_child_journal" \ +PAGE_FILE="$delayed_child_page" \ +PHASE="delayed-turn-child-present" \ +TRANSPORT=1 \ + "$node_bin" "$client" +kill -KILL "$service_pid" +wait "$service_pid" 2>/dev/null || true +service_pid="" +rm -f "$delayed_child_socket" +TYPED_TRANSPORT_AFTER_READY="cleanup_pending" \ +TYPED_TRANSPORT_CHILD_ABSENT="0" \ + start_service "$delayed_child_socket" "" "delayed-turn-child-present-restart-service" +CONTROL_SOCKET="$delayed_child_socket" \ +STREAM_ID="stream:delayed-turn-child-present" \ +JOURNAL_PATH="$delayed_child_journal" \ +PAGE_FILE="$delayed_child_page" \ +PHASE="verify-delayed-turn-child-present-restart" \ +TRANSPORT=1 \ + "$node_bin" "$client" +stop_service +unset TYPED_TRANSPORT_AFTER_READY +unset TYPED_TRANSPORT_CHILD_ABSENT + +delayed_port_socket="$tmp_dir/delayed-turn-port-only-control.sock" +delayed_port_journal="${delayed_port_socket}.launch-reconciliations.json" +delayed_port_page="$tmp_dir/delayed-turn-port-only-page.json" +TYPED_TRANSPORT_AFTER_READY="cleanup_pending" \ +TYPED_TRANSPORT_DELAYED_PORTS="1" \ + start_service "$delayed_port_socket" "" "delayed-turn-port-only-service" +CONTROL_SOCKET="$delayed_port_socket" \ +STREAM_ID="stream:delayed-turn-port-only" \ +JOURNAL_PATH="$delayed_port_journal" \ +PAGE_FILE="$delayed_port_page" \ +PHASE="delayed-turn-port-only" \ +TRANSPORT=1 \ + "$node_bin" "$client" +kill -KILL "$service_pid" +wait "$service_pid" 2>/dev/null || true +service_pid="" +rm -f "$delayed_port_socket" +TYPED_TRANSPORT_AFTER_READY="cleanup_pending" \ +TYPED_TRANSPORT_DELAYED_PORTS="1" \ + start_service "$delayed_port_socket" "" "delayed-turn-port-only-restart-service" +CONTROL_SOCKET="$delayed_port_socket" \ +STREAM_ID="stream:delayed-turn-port-only" \ +JOURNAL_PATH="$delayed_port_journal" \ +PAGE_FILE="$delayed_port_page" \ +PHASE="verify-delayed-turn-port-only-restart" \ +TRANSPORT=1 \ + "$node_bin" "$client" +stop_service +unset TYPED_TRANSPORT_AFTER_READY +unset TYPED_TRANSPORT_DELAYED_PORTS + +failed_flush_socket="$tmp_dir/failed-flush-durability-control.sock" +failed_flush_journal="${failed_flush_socket}.launch-reconciliations.json" +failed_flush_page="$tmp_dir/failed-flush-durability-page.json" +TYPED_TRANSPORT_AFTER_READY="cleanup_pending" \ +TYPED_TRANSPORT_DELAYED_PORTS="1" \ +TYPED_TRANSPORT_PROFILE_DURABILITY="failed" \ + start_service "$failed_flush_socket" "" "failed-flush-durability-service" +CONTROL_SOCKET="$failed_flush_socket" \ +STREAM_ID="stream:failed-flush-durability" \ +JOURNAL_PATH="$failed_flush_journal" \ +PAGE_FILE="$failed_flush_page" \ +PHASE="delayed-turn-port-only" \ +TRANSPORT=1 \ +TYPED_TRANSPORT_PROFILE_DURABILITY="failed" \ + "$node_bin" "$client" +kill -KILL "$service_pid" +wait "$service_pid" 2>/dev/null || true +service_pid="" +rm -f "$failed_flush_socket" +TYPED_TRANSPORT_AFTER_READY="cleanup_pending" \ +TYPED_TRANSPORT_DELAYED_PORTS="1" \ +TYPED_TRANSPORT_PROFILE_DURABILITY="failed" \ + start_service "$failed_flush_socket" "" "failed-flush-durability-restart-service" +CONTROL_SOCKET="$failed_flush_socket" \ +STREAM_ID="stream:failed-flush-durability" \ +JOURNAL_PATH="$failed_flush_journal" \ +PAGE_FILE="$failed_flush_page" \ +PHASE="verify-failed-flush-durability-restart" \ +TRANSPORT=1 \ +TYPED_TRANSPORT_PROFILE_DURABILITY="failed" \ + "$node_bin" "$client" +stop_service +unset TYPED_TRANSPORT_AFTER_READY +unset TYPED_TRANSPORT_DELAYED_PORTS +unset TYPED_TRANSPORT_PROFILE_DURABILITY + +native_close_socket="$tmp_dir/native-terminal-ordinary-close-control.sock" +native_close_journal="${native_close_socket}.launch-reconciliations.json" +TYPED_TRANSPORT_AFTER_READY="terminal_post_effect_cleanup" \ +TYPED_TRANSPORT_PROFILE_DURABILITY="failed" \ + start_service "$native_close_socket" "" "native-terminal-ordinary-close-service" +CONTROL_SOCKET="$native_close_socket" \ +STREAM_ID="stream:native-terminal-ordinary-close" \ +JOURNAL_PATH="$native_close_journal" \ +PHASE="native-terminal-then-ordinary-close" \ +TRANSPORT=1 \ +TYPED_TRANSPORT_PROFILE_DURABILITY="failed" \ + "$node_bin" "$client" +stop_service +unset TYPED_TRANSPORT_AFTER_READY +unset TYPED_TRANSPORT_PROFILE_DURABILITY + +log_started_socket="$tmp_dir/log-started-owner-alive-control.sock" +log_started_journal="${log_started_socket}.launch-reconciliations.json" +log_started_owner="$tmp_dir/log-started-owner.pid" +LOG_STARTED_OWNER_HOLD=1 \ +OWNER_PID_FILE="$log_started_owner" \ +ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG="$tmp_dir/log-started-owner-alive-service.err" \ + start_service "$log_started_socket" "" "log-started-owner-alive-service" +CONTROL_SOCKET="$log_started_socket" \ +STREAM_ID="stream:log-started-owner-alive" \ +JOURNAL_PATH="$log_started_journal" \ +OWNER_PID_FILE="$log_started_owner" \ +PHASE="log-started-paths-gone-owner-alive" \ +TRANSPORT=1 \ + "$node_bin" "$client" +stop_service +if [[ -f "$log_started_owner" ]]; then + owner_pid="$(tr -d '[:space:]' < "$log_started_owner")" + if [[ "$owner_pid" =~ ^[0-9]+$ ]]; then + kill "$owner_pid" >/dev/null 2>&1 || true + fi +fi +unset LOG_STARTED_OWNER_HOLD +unset OWNER_PID_FILE +unset ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG + +log_exited_socket="$tmp_dir/log-started-owner-exited-control.sock" +log_exited_journal="${log_exited_socket}.launch-reconciliations.json" +LOG_STARTED_OWNER_HOLD=1 \ +ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG="$tmp_dir/log-started-owner-exited-service.err" \ + start_service "$log_exited_socket" "" "log-started-owner-exited-service" +CONTROL_SOCKET="$log_exited_socket" \ +STREAM_ID="stream:log-started-owner-exited" \ +JOURNAL_PATH="$log_exited_journal" \ +PHASE="log-started-owner-exits-without-native-settlement" \ +TRANSPORT=1 \ + "$node_bin" "$client" +stop_service +unset LOG_STARTED_OWNER_HOLD +unset ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG + printf '%s\n' '{"schema":"elastos.browser.vm-control-service-settlement-smoke/v1","ok":true}' diff --git a/scripts/browser-vm-control-service.mjs b/scripts/browser-vm-control-service.mjs index 4581f0b62..f9e0608b3 100755 --- a/scripts/browser-vm-control-service.mjs +++ b/scripts/browser-vm-control-service.mjs @@ -1,13 +1,13 @@ #!/usr/bin/env node import crypto from "node:crypto"; +import dgram from "node:dgram"; import fs from "node:fs"; import http from "node:http"; import net from "node:net"; import path from "node:path"; import process from "node:process"; -import { execFile, spawn } from "node:child_process"; +import { execFile, execFileSync, spawn } from "node:child_process"; import { isDeepStrictEqual } from "node:util"; - const CONFIG_ENV = "ELASTOS_BROWSER_VM_CONTROL_SERVICE_CONFIG"; let verifiedHostReadiness = null; @@ -57,7 +57,7 @@ async function engineReadiness(config) { if (identity && verifiedHostReadiness?.identity === identity) return verifiedHostReadiness.result; return new Promise((resolve) => { execFile(script, ["--host-readiness"], { - timeout: 8000, maxBuffer: 64 * 1024, + timeout: 30000, maxBuffer: 64 * 1024, env: { ...process.env, ELASTOS_BROWSER_VM_STAGED_ROOTFS: "", ...((process.env.ELASTOS_BROWSER_VM_PLATFORM?.startsWith("darwin-") ?? process.platform === "darwin") @@ -102,6 +102,20 @@ const VZ_TRANSPORT_SECRET_SCHEMA = "elastos.browser.vz-transport-secret/v1"; const VZ_LAUNCH_SETTLEMENT_SCHEMA = "elastos.browser.vz-launch-settlement/v1"; +const VZ_LAUNCH_SETTLEMENT_KEYS = [ + "schema", + "state", + "message", + "binding_hash", + "generation", + "page_id", + "vm_id", + "stream_id", + "media_stream_id", + "effects", + "absence", +]; +const VZ_LAUNCH_PROFILE_DURABILITY = ["failed", "unknown", "proved"]; const VZ_MEDIA_DIAGNOSTIC_SCHEMA = "elastos.browser.media-diagnostic/v1"; const VZ_MEDIA_DIAGNOSTIC_EVENTS = new Set([ @@ -132,6 +146,10 @@ const VZ_LAUNCH_ABSENCE_KEYS = [ "session_directory_absent", "vm_absent", ]; +const VZ_LAUNCH_PORT_ABSENCE_KEYS = [ + "turn_listener_absent", + "turn_relay_ports_absent", +]; const TERMINAL_CLEANUP_EFFECT_KEYS = [ "page_absent", "child_absent", @@ -234,6 +252,33 @@ function exactObjectKeys(value, keys) { ); } +function profileDurabilityIsSafe(value) { + return value === undefined || VZ_LAUNCH_PROFILE_DURABILITY.includes(value); +} + +function profileDurabilityFields(source) { + const durability = + source?.profile_durability || + source?.launch_settlement_result?.profile_durability; + if (!VZ_LAUNCH_PROFILE_DURABILITY.includes(durability)) { + return {}; + } + return { profile_durability: durability }; +} + +function vzLaunchSettlementKeysAreExact(settlement) { + if (!settlement || typeof settlement !== "object" || Array.isArray(settlement)) { + return false; + } + if (!VZ_LAUNCH_SETTLEMENT_KEYS.every((key) => Object.hasOwn(settlement, key))) { + return false; + } + return Object.keys(settlement).every( + (key) => + VZ_LAUNCH_SETTLEMENT_KEYS.includes(key) || key === "profile_durability", + ); +} + function canonicalJson(value) { if (Array.isArray(value)) return value.map(canonicalJson); if (value && typeof value === "object") { @@ -458,19 +503,8 @@ function validateVzTransportSecret(authority, secret) { function validateVzLaunchSettlement(settlement) { if ( - !exactObjectKeys(settlement, [ - "schema", - "state", - "message", - "binding_hash", - "generation", - "page_id", - "vm_id", - "stream_id", - "media_stream_id", - "effects", - "absence", - ]) || + !vzLaunchSettlementKeysAreExact(settlement) || + !profileDurabilityIsSafe(settlement.profile_durability) || settlement.schema !== VZ_LAUNCH_SETTLEMENT_SCHEMA || ![ LAUNCH_SETTLEMENT_DID_NOT_ACT, @@ -648,10 +682,20 @@ function controlServiceIdentityIsSafe(identity, controlSocketPath) { !/[\r\n\0]/.test(identity.control_socket_path) && identity.control_socket_path === controlSocketPath && (identity.config_fingerprint === null || + identity.config_fingerprint === undefined || /^[0-9a-f]{64}$/.test(identity.config_fingerprint)) ); } +function controlServiceIdentityMatches(left, right) { + return ( + controlServiceIdentityIsSafe(left, left?.control_socket_path) && + controlServiceIdentityIsSafe(right, right?.control_socket_path) && + left.service_id === right.service_id && + left.control_socket_path === right.control_socket_path + ); +} + function hostProcessBindingIsSafe(binding) { return ( binding?.schema === HOST_PROCESS_BINDING_SCHEMA && @@ -739,26 +783,148 @@ function exactCleanupEffects( }; } -function tcpEndpointAbsent(host, port, timeoutMs = 500) { +function closeBoundHandle(handle) { return new Promise((resolve) => { - const socket = net.createConnection({ host, port }); + try { + handle.close(() => resolve()); + } catch { + resolve(); + } + }); +} + +function tcpBindAbsent(host, port) { + return new Promise((resolve) => { + const server = net.createServer(); let settled = false; const finish = (absent) => { if (settled) return; settled = true; - socket.removeAllListeners(); - socket.destroy(); - resolve(absent); + server.removeAllListeners(); + server.close(() => resolve(absent)); }; - socket.setTimeout(timeoutMs); - socket.once("connect", () => finish(false)); - socket.once("timeout", () => finish(false)); - socket.once("error", (error) => - finish(error?.code === "ECONNREFUSED"), - ); + server.once("error", () => finish(false)); + server.listen({ host, port, exclusive: true }, () => finish(true)); + }); +} + +function bindTcpExclusive(host, port) { + return new Promise((resolve, reject) => { + const server = net.createServer(); + server.once("error", (error) => reject(error)); + server.listen({ host, port, exclusive: true }, () => resolve(server)); + }); +} + +function bindUdpExclusive(host, port) { + return new Promise((resolve, reject) => { + const socket = dgram.createSocket({ + type: net.isIP(host) === 6 ? "udp6" : "udp4", + reuseAddr: false, + }); + socket.once("error", (error) => reject(error)); + socket.bind({ address: host, port, exclusive: true }, () => resolve(socket)); }); } +async function turnListenerPortAbsent(turn) { + return tcpBindAbsent(turn.listen_host, turn.listen_port); +} + +async function turnRelayPortsAbsent(turn) { + const held = []; + try { + for (let port = turn.relay_port_min; port <= turn.relay_port_max; port += 1) { + let tcp; + let udp; + try { + tcp = await bindTcpExclusive(turn.relay_host, port); + } catch { + return false; + } + held.push(tcp); + try { + udp = await bindUdpExclusive(turn.relay_host, port); + } catch { + return false; + } + held.push(udp); + } + return true; + } finally { + await Promise.all(held.map(closeBoundHandle)); + } +} + +function supervisorOwnedVsockBridgesAbsent(sessionAbsent, termination) { + return ( + sessionAbsent === true && + (termination?.graceful === true || + termination?.terminal_record === true) + ); +} + +function hostProcessIsAlive(binding) { + const pid = binding?.pid; + if (!Number.isInteger(pid) || pid <= 1) { + return false; + } + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +function parseVzLaunchSettlementFromStderr(stderr) { + const error = launcherError("Browser VZ launcher settlement", stderr); + return error?.vz_launch_settlement || null; +} + +function vzLaunchNonPortAbsenceProved(absence) { + if (!absence || typeof absence !== "object" || Array.isArray(absence)) { + return false; + } + return VZ_LAUNCH_ABSENCE_KEYS.filter( + (key) => !VZ_LAUNCH_PORT_ABSENCE_KEYS.includes(key), + ).every((key) => absence[key] === true); +} + +function ownedTurnDelayedTerminal( + settlement, + binding, + sessionAbsent, + turnListenerAbsent, + relayPortsAbsent, +) { + if (!settlement || !binding) { + return false; + } + if (settlement.schema !== VZ_LAUNCH_SETTLEMENT_SCHEMA) { + return false; + } + if (settlement.effects?.turn_process !== true) { + return false; + } + if ( + settlement.page_id !== binding.page_id || + settlement.generation !== binding.generation + ) { + return false; + } + // effects.turn_process only records that TURN started. Native settlement + // absence must prove every non-port effect before the helper retries ports. + if (!vzLaunchNonPortAbsenceProved(settlement.absence)) { + return false; + } + return ( + sessionAbsent === true && + turnListenerAbsent === true && + relayPortsAbsent === true + ); +} + async function exactTransportCleanupEffects(binding, termination) { const authority = binding.transport_authority; if (!authority) return {}; @@ -768,22 +934,49 @@ async function exactTransportCleanupEffects(binding, termination) { authority, ); const sessionAbsent = !fs.existsSync(binding.isolation.session_dir); - const turnListenerAbsent = await tcpEndpointAbsent( - authority.turn.listen_host, - authority.turn.listen_port, + // Runtime owns egress/media runtime_socket_path listeners during official close. + // Supervisor connects to those paths and proves its own bridges by session + // absence plus a graceful or already-terminal owned launcher exit. + // Native browser-vz-engine-supervisor owns TURN child cleanup. + // Port binds prove only listen and declared TCP/UDP relay absence. + const probeTurnPorts = + binding.transport_receipt.effects.turn_launch_owned === true; + const turnListenerAbsent = probeTurnPorts + ? await turnListenerPortAbsent(authority.turn) + : true; + const relayPortsAbsent = probeTurnPorts + ? await turnRelayPortsAbsent(authority.turn) + : true; + const delayedTurnTerminal = ownedTurnDelayedTerminal( + termination?.vz_launch_settlement, + binding, + sessionAbsent, + turnListenerAbsent, + relayPortsAbsent, + ); + const turnProcessAbsent = + !probeTurnPorts || + termination?.graceful === true || + termination?.terminal_record === true || + delayedTurnTerminal; + const delayedBridgesAbsent = supervisorOwnedVsockBridgesAbsent( + sessionAbsent, + { + graceful: termination?.graceful === true, + terminal_record: + termination?.terminal_record === true || delayedTurnTerminal, + }, ); - const gracefulOwnerExit = termination?.graceful === true; - const ownedEffectsAbsent = gracefulOwnerExit && sessionAbsent; return { transport_session_absent: sessionAbsent, - turn_process_absent: ownedEffectsAbsent, - turn_listener_absent: ownedEffectsAbsent && turnListenerAbsent, - turn_relay_ports_absent: ownedEffectsAbsent && turnListenerAbsent, - ordinary_vsock_bridge_absent: ownedEffectsAbsent, - media_vsock_bridge_absent: ownedEffectsAbsent, - bootstrap_vsock_bridge_absent: ownedEffectsAbsent, + turn_process_absent: turnProcessAbsent, + turn_listener_absent: turnListenerAbsent, + turn_relay_ports_absent: relayPortsAbsent, + ordinary_vsock_bridge_absent: delayedBridgesAbsent, + media_vsock_bridge_absent: delayedBridgesAbsent, + bootstrap_vsock_bridge_absent: sessionAbsent, hibernation_state_absent: - ownedEffectsAbsent && + sessionAbsent && binding.transport_receipt.effects.hibernation_disabled === true, }; } @@ -907,7 +1100,7 @@ function requireExactRuntimeCleanupRecord( (binding.principal_id || "") === (launch.principal_id || "") && binding.control_socket_path === page.control_socket_path && binding.shutdown_socket_path === config.control_socket_path && - isDeepStrictEqual(binding.control_service, controlServiceIdentity) && + controlServiceIdentityMatches(binding.control_service, controlServiceIdentity) && isDeepStrictEqual(binding.isolation, page.isolation) && isDeepStrictEqual(binding.process, page.process) && isDeepStrictEqual( @@ -939,8 +1132,8 @@ function requireExactDurableCleanupRecord(store, binding) { !record || !record.cleanup_binding || !record.control_service || - !isDeepStrictEqual(record.control_service, store.control_service) || - !isDeepStrictEqual(binding.control_service, store.control_service) || + !controlServiceIdentityMatches(record.control_service, store.control_service) || + !controlServiceIdentityMatches(binding.control_service, store.control_service) || !launchIdentityMatchesCleanupBinding(record.launch, binding) || !isDeepStrictEqual(record.cleanup_binding, binding) || ![ @@ -1154,6 +1347,21 @@ function parseVzMediaDiagnostic(line, launch) { return diagnostic; } +function looksLikeRejectedVzMediaDiagnostic(line) { + const trimmed = String(line || "").trim(); + if ( + !trimmed.startsWith("{") || + !trimmed.includes("elastos.browser.media-diagnostic/v1") + ) { + return false; + } + try { + return JSON.parse(trimmed)?.schema === "elastos.browser.media-diagnostic/v1"; + } catch { + return false; + } +} + function logVzMediaDiagnostic(diagnostic) { logEvent("media_diagnostic", { component: "turn", @@ -1338,15 +1546,17 @@ function launchReconciliationRecordIsSafe(record) { launchSettlementIsSafe && record.launch_settlement_result?.state === LAUNCH_SETTLEMENT_TERMINAL; + const durableReceiptIsSafe = durableTerminalCleanupReceiptIsSafe( + record.terminal_cleanup_receipt, + launch, + record.cleanup_binding, + ); const terminalCleanupReceiptIsSafe = state === LAUNCH_SETTLEMENT_TERMINAL ? typedTerminalLaunchSettlement - ? record.terminal_cleanup_receipt === undefined - : durableTerminalCleanupReceiptIsSafe( - record.terminal_cleanup_receipt, - launch, - record.cleanup_binding, - ) + ? record.terminal_cleanup_receipt === undefined || + durableReceiptIsSafe + : durableReceiptIsSafe : record.terminal_cleanup_receipt === undefined; return ( record?.schema === @@ -1381,7 +1591,8 @@ function launchReconciliationRecordIsSafe(record) { (typeof effects.page_acquired === "boolean" && typeof effects.vm_acquired === "boolean")) && launchSettlementIsSafe && - terminalCleanupReceiptIsSafe + terminalCleanupReceiptIsSafe && + profileDurabilityIsSafe(record.profile_durability) ); } @@ -1407,6 +1618,9 @@ function durableLaunchReconciliationRecord(record) { durable.launch_settlement_result = record.launch_settlement_result; } + if (record.profile_durability !== undefined) { + durable.profile_durability = record.profile_durability; + } return durable; } @@ -1571,6 +1785,19 @@ function recordLaunchReconciliation( ) { next.cleanup_binding = current.cleanup_binding; } + if ( + next.launch_settlement_result === undefined && + current?.launch_settlement_result !== undefined && + current.launch_settlement_result.state === state + ) { + next.launch_settlement_result = current.launch_settlement_result; + } + if ( + next.profile_durability === undefined && + current?.profile_durability !== undefined + ) { + next.profile_durability = current.profile_durability; + } launchReconciliations.set(key, next); try { persistLaunchReconciliations(launchReconciliationStore); @@ -1583,7 +1810,99 @@ function recordLaunchReconciliation( } } -function reconcileLaunch(launchReconciliationStore, activePages, body) { +async function settleDispatchedTransportLaunchWithoutBinding( + launchReconciliationStore, + record, + _shutdownTimeoutMs, +) { + const launch = record?.launch; + if ( + record?.state !== LAUNCH_SETTLEMENT_PENDING || + record.cleanup_binding || + !launch?.transport_authority + ) { + return record; + } + const authority = validateVzTransportAuthority(launch.transport_authority); + const paths = vzTransportLaunchPaths(authority); + if (!paths) { + return record; + } + let owner = null; + if (fs.existsSync(paths.owner_path)) { + try { + owner = JSON.parse(fs.readFileSync(paths.owner_path, "utf8")); + } catch { + return record; + } + if (!ownerMatchesTransportLaunch(owner, launch)) { + return record; + } + } + const loggedEffects = helperLogVzLaunchEffects( + launch.page_id, + launch.lifecycle_generation, + launch.stream_id, + ); + if (loggedEffects) { + logEvent("launch_reconciliation_log_effects", { + stream_id: launch.stream_id, + page_id: launch.page_id, + generation: launch.lifecycle_generation, + effects: loggedEffects, + }); + } + if (record.launch_settlement_result) { + try { + const native = validateVzLaunchSettlementForLaunch( + record.launch_settlement_result, + launch, + ); + if (native.state === LAUNCH_SETTLEMENT_TERMINAL) { + return record; + } + const liveEffectsGone = + !fs.existsSync(paths.owner_path) && + !fs.existsSync(paths.session_dir) && + !fs.existsSync(paths.control_socket_path); + const nativeAbsent = + native.absence?.vm_absent === true && + native.absence?.session_directory_absent === true && + native.absence?.supervisor_child_absent === true; + if (liveEffectsGone && nativeAbsent) { + recordLaunchReconciliation( + launchReconciliationStore, + launch, + LAUNCH_SETTLEMENT_TERMINAL, + { + effects: { page_acquired: false, vm_acquired: false }, + launch_settlement_result: { + ...native, + state: LAUNCH_SETTLEMENT_TERMINAL, + absence: Object.fromEntries( + VZ_LAUNCH_ABSENCE_KEYS.map((key) => [key, true]), + ), + }, + }, + ); + return ( + launchReconciliationStore.records.get( + launchReconciliationKey( + launch.lifecycle_generation, + launch.stream_id, + ), + ) || record + ); + } + } catch { + return record; + } + } + // Keep cleanup_pending until an exact owned native settlement arrives. + return record; +} + +async function reconcileLaunch(launchReconciliationStore, activePages, body, shutdownTimeoutMs) { const launchReconciliations = launchReconciliationStore.records; if ( body?.schema !== "elastos.browser.vm-control-service.reconcile-launch/v1" || @@ -1683,11 +2002,16 @@ function reconcileLaunch(launchReconciliationStore, activePages, body) { "Browser VM launch reconciliation authority does not match the durable record", ); } + const settled = await settleDispatchedTransportLaunchWithoutBinding( + launchReconciliationStore, + record, + shutdownTimeoutMs, + ); return { - ...record, + ...(settled || record), responder_control_service: launchReconciliationStore.control_service, - ...(record.launch?.transport_authority - ? { transport_authority: record.launch.transport_authority } + ...((settled || record).launch?.transport_authority + ? { transport_authority: (settled || record).launch.transport_authority } : {}), }; } @@ -1713,6 +2037,8 @@ function markLaunchReconciliationTerminal( ...(terminalCleanupReceipt ? { terminal_cleanup_receipt: terminalCleanupReceipt } : {}), + ...profileDurabilityFields(terminalCleanupReceipt), + ...profileDurabilityFields(record), }, ); } @@ -1830,6 +2156,46 @@ function vmKeyHash(vmKey) { return crypto.createHash("sha256").update(String(vmKey || "")).digest("hex").slice(0, 16); } +function launcherChildIsLive(child) { + return Boolean(child) && child.exitCode == null && child.signalCode == null; +} + +function vmOwnsExactPage(vmRecord, pageId) { + return ( + Boolean(pageId) && + vmRecord?.pages instanceof Set && + vmRecord.pages.has(pageId) + ); +} + +function ownedVmRecord(record, pageId, activeVms) { + const vmRecord = record?.vm_key ? activeVms.get(record.vm_key) : null; + return vmOwnsExactPage(vmRecord, pageId) ? vmRecord : null; +} + +function pageHoldsLiveCapacity(pageId, record, activeVms) { + if (launcherChildIsLive(record?.launcher_child)) { + return true; + } + return launcherChildIsLive(ownedVmRecord(record, pageId, activeVms)?.launcher_child); +} + +function liveCapacityPageCount(activePages, activeVms) { + let count = 0; + for (const [pageId, record] of activePages.entries()) { + if (pageHoldsLiveCapacity(pageId, record, activeVms)) { + count += 1; + } + } + return count; +} + +function pendingCleanupPageIds(activePages, activeVms) { + return [...activePages.entries()] + .filter(([pageId, record]) => !pageHoldsLiveCapacity(pageId, record, activeVms)) + .map(([pageId]) => pageId); +} + const LIFECYCLE_PHASES = [ "CONTROL_READY", "ACQUIRING_SLOT", @@ -1947,11 +2313,14 @@ function lifecycleWarmVmRecord(vmKey, vmRecord, nowMs, config) { function lifecycleStatus(config, activePages, activeVms, pendingLaunches) { const nowMs = Date.now(); const maxActivePages = Number(config.max_active_pages ?? 1); + const livePages = [...activePages.entries()].filter(([pageId, record]) => + pageHoldsLiveCapacity(pageId, record, activeVms), + ); const sessions = [ ...[...pendingLaunches.entries()].map(([requestId, launch]) => lifecyclePendingLaunchRecord(requestId, launch, nowMs), ), - ...[...activePages.entries()].map(([pageId, record]) => + ...livePages.map(([pageId, record]) => lifecycleActivePageRecord(pageId, record, nowMs), ), ...[...activeVms.entries()] @@ -1962,7 +2331,7 @@ function lifecycleStatus(config, activePages, activeVms, pendingLaunches) { schema: "elastos.browser.lifecycle-status/v1", owner: "vm_control_service", phases: LIFECYCLE_PHASES, - capacity_available: activePages.size < maxActivePages, + capacity_available: liveCapacityPageCount(activePages, activeVms) < maxActivePages, sessions, redaction: { principal_id: "sha256-16", @@ -2454,6 +2823,246 @@ function vmSupervisorResultFromGuest(result, launch, vmRecord) { return normalized; } +function persistentLauncherReadyLine(stdout) { + for (const line of String(stdout).split(/\r?\n/)) { + const trimmed = line.trim(); + if (!trimmed) continue; + try { + const parsed = JSON.parse(trimmed); + if (parsed?.schema === "elastos.browser.engine.supervisor-result/v1") { + return trimmed; + } + } catch { + // Host tools such as debugfs may write stdout before the supervisor result. + } + } + return null; +} + +function attachLauncherFailureContext(error, child, stderr, stderrTail) { + const next = error instanceof Error ? error : new Error(String(error)); + next.launcher_child = child; + next.stderr = stderr; + next.stderr_tail = stderrTail; + return next; +} + +function launchedChildFromFailure(launcher, error) { + return launcher?.child || error?.launcher_child || null; +} + +function launchedStderrFromFailure(launcher, error) { + return ( + launcher?.stderr_tail?.text || + error?.stderr_tail?.text || + launcher?.stderr || + error?.stderr || + "" + ); +} + +function vzTransportLaunchPaths(authority) { + const digest = String(authority?.binding_hash || "") + .replace(/^sha256:/, "") + .toLowerCase(); + if (!/^[0-9a-f]{64}$/.test(digest)) { + return null; + } + const segment = digest.slice(0, 32); + const sessionRoot = process.env.ELASTOS_BROWSER_VM_ROOT || "/tmp/evzs"; + const socketRoot = process.env.ELASTOS_BROWSER_VM_SOCKET_ROOT || "/tmp/evzrc"; + const socketDir = path.join(socketRoot, segment); + return { + session_dir: path.join(sessionRoot, `vz-${segment}`), + socket_dir: socketDir, + control_socket_path: path.join(socketDir, "c.sock"), + owner_path: path.join(socketDir, "owner.json"), + }; +} + +function ownerMatchesTransportLaunch(owner, launch) { + return ( + owner?.schema === "elastos.browser.vz-socket-owner/v1" && + owner.page_id === launch.page_id && + owner.generation === launch.lifecycle_generation && + owner.vm_id === launch.vm_id && + owner.stream_id === launch.stream_id + ); +} + +function helperLogVzLaunchEffects(pageId, generation, streamId) { + const dataDir = process.env.ELASTOS_BROWSER_VM_DATA_DIR; + const logPath = + process.env.ELASTOS_BROWSER_VM_CONTROL_SERVICE_LOG || + (dataDir + ? path.join(dataDir, "logs/browser-vm-control-service.log") + : ""); + if (!logPath || !fs.existsSync(logPath)) { + return null; + } + let text = ""; + try { + const fd = fs.openSync(logPath, "r"); + try { + const size = fs.fstatSync(fd).size; + const start = Math.max(0, size - 2 * 1024 * 1024); + const buffer = Buffer.alloc(size - start); + fs.readSync(fd, buffer, 0, buffer.length, start); + text = buffer.toString("utf8"); + } finally { + fs.closeSync(fd); + } + } catch { + return null; + } + let sawSession = false; + let sawTurn = false; + for (const line of text.split(/\r?\n/)) { + if ( + !line.includes(pageId) && + !line.includes(generation) && + !line.includes(streamId) + ) { + continue; + } + try { + const event = JSON.parse(line); + if (event.page_id && event.page_id !== pageId) continue; + if (event.generation && event.generation !== generation) continue; + if (event.stream_id && event.stream_id !== streamId) continue; + if (event.event === "media_diagnostic") { + sawSession = true; + if ( + event.diagnostic_event === "turn_process_started" || + event.diagnostic_event === "turn_listener_ready" + ) { + sawTurn = true; + } + } + if ( + event.event === "launch_failed" && + String(event.error || "").includes("Allocated inode") + ) { + sawSession = true; + } + } catch { + // Ignore non-JSON log lines. + } + } + if (!sawSession) { + return null; + } + return { + session_directory: true, + control_socket: true, + ordinary_stream_bridge: true, + media_stream_bridge: true, + turn_process: sawTurn, + supervisor_child: true, + vm: true, + }; +} + +function supervisorPidOwningPath(socketPath) { + try { + const output = execFileSync("lsof", ["-t", "--", socketPath], { + encoding: "utf8", + timeout: 2000, + }); + const pids = [ + ...new Set( + String(output) + .trim() + .split(/\s+/) + .map((value) => Number(value)) + .filter((pid) => Number.isInteger(pid) && pid > 1 && pid !== process.pid), + ), + ]; + return pids.length === 1 ? pids[0] : null; + } catch { + return null; + } +} + +function terminateOwnedSupervisorPid(pid, timeoutMs) { + return new Promise((resolve, reject) => { + if (!Number.isInteger(pid) || pid <= 1) { + resolve({ already_exited: true, graceful: false, forced: false }); + return; + } + try { + process.kill(pid, 0); + } catch (error) { + if (error?.code === "ESRCH") { + resolve({ already_exited: true, graceful: false, forced: false }); + return; + } + reject(error); + return; + } + let forced = false; + let settled = false; + const finish = (result, error = null) => { + if (settled) return; + settled = true; + clearInterval(poll); + clearTimeout(killTimer); + if (error) reject(error); + else resolve(result); + }; + const poll = setInterval(() => { + try { + process.kill(pid, 0); + } catch (error) { + if (error?.code === "ESRCH") { + finish({ + already_exited: false, + graceful: !forced, + forced, + }); + } + } + }, 100); + const killTimer = setTimeout(() => { + forced = true; + try { + process.kill(pid, "SIGKILL"); + } catch (error) { + if (error?.code === "ESRCH") { + finish({ already_exited: false, graceful: false, forced: true }); + return; + } + finish(null, error); + return; + } + setTimeout(() => { + try { + process.kill(pid, 0); + finish( + null, + new Error("Browser VZ supervisor did not exit after its exact SIGKILL"), + ); + } catch (error) { + if (error?.code === "ESRCH") { + finish({ already_exited: false, graceful: false, forced: true }); + return; + } + finish(null, error); + } + }, Math.min(Math.max(timeoutMs, 100), 5000)); + }, timeoutMs); + try { + process.kill(pid, "SIGTERM"); + } catch (error) { + if (error?.code === "ESRCH") { + finish({ already_exited: true, graceful: false, forced: false }); + return; + } + finish(null, error); + } + }); +} + function runPersistentProgram(program, args, env, stdin, timeoutMs, signal, launch) { return new Promise((resolve, reject) => { let child; @@ -2471,6 +3080,7 @@ function runPersistentProgram(program, args, env, stdin, timeoutMs, signal, laun let stdout = ""; let stderr = ""; let diagnosticStderr = ""; + const stderrTail = { text: "" }; let phase = "running"; let timer; const clearSettlementTriggers = () => { @@ -2481,13 +3091,13 @@ function runPersistentProgram(program, args, env, stdin, timeoutMs, signal, laun if (phase === "settled") return; phase = "settled"; clearSettlementTriggers(); - reject(error); + reject(attachLauncherFailureContext(error, child, stderr, stderrTail)); }; const settleOk = (line) => { if (phase !== "running") return; phase = "settled"; clearSettlementTriggers(); - resolve({ stdout: line, stderr, child, owner_reaped: false }); + resolve({ stdout: line, stderr, child, owner_reaped: false, stderr_tail: stderrTail }); }; const terminateFor = (error) => { if (phase !== "running") return; @@ -2527,9 +3137,9 @@ function runPersistentProgram(program, args, env, stdin, timeoutMs, signal, laun signal?.addEventListener?.("abort", abortLaunch, { once: true }); child.stdout.on("data", (chunk) => { stdout += chunk.toString("utf8"); - const lines = stdout.split(/\r?\n/).filter(Boolean); - if (lines.length > 0) { - settleOk(lines[0]); + const ready = persistentLauncherReadyLine(stdout); + if (ready) { + settleOk(ready); } }); child.stderr.on("data", (chunk) => { @@ -2548,6 +3158,11 @@ function runPersistentProgram(program, args, env, stdin, timeoutMs, signal, laun const diagnostic = parseVzMediaDiagnostic(line, launch); if (diagnostic) { logVzMediaDiagnostic(diagnostic); + } else if (!looksLikeRejectedVzMediaDiagnostic(line)) { + stderrTail.text += `${line}\n`; + if (stderrTail.text.length > 8192) { + stderrTail.text = stderrTail.text.slice(-8192); + } } } }); @@ -2659,7 +3274,10 @@ function recordProvenLaunchFailure( error?.vz_launch_settlement?.effects?.vm ?? true, }, ...(error?.vz_launch_settlement - ? { launch_settlement_result: error.vz_launch_settlement } + ? { + launch_settlement_result: error.vz_launch_settlement, + ...profileDurabilityFields(error.vz_launch_settlement), + } : {}), }, ); @@ -2710,10 +3328,31 @@ async function settleDispatchedLaunchFailure( shutdownTimeoutMs, transportLaunch = false, ) { - if ( - transportLaunch && - !error?.vz_launch_settlement - ) { + const child = launchedChildFromFailure(launcher, error); + if (launcherChildIsLive(child)) { + try { + await terminatePersistentLauncher(child, shutdownTimeoutMs); + } catch (cleanupError) { + return launchSettlementError( + error, + LAUNCH_SETTLEMENT_PENDING, + cleanupError, + ); + } + } + const settlement = + error?.vz_launch_settlement || + parseVzLaunchSettlementFromStderr(launchedStderrFromFailure(launcher, error)); + if (settlement) { + return launchSettlementError( + Object.assign( + error instanceof Error ? error : new Error(String(error)), + { vz_launch_settlement: settlement }, + ), + settlement.state, + ); + } + if (transportLaunch) { return launchSettlementError( error, LAUNCH_SETTLEMENT_PENDING, @@ -2727,19 +3366,7 @@ async function settleDispatchedLaunchFailure( ) { return error; } - if (launcher?.child) { - try { - await terminatePersistentLauncher(launcher.child, shutdownTimeoutMs); - return launchSettlementError(error, LAUNCH_SETTLEMENT_TERMINAL); - } catch (cleanupError) { - return launchSettlementError( - error, - LAUNCH_SETTLEMENT_PENDING, - cleanupError, - ); - } - } - if (launcher?.owner_reaped === true) { + if (launcher?.owner_reaped === true || child) { return launchSettlementError(error, LAUNCH_SETTLEMENT_TERMINAL); } return launchSettlementError(error, LAUNCH_SETTLEMENT_PENDING); @@ -2781,8 +3408,9 @@ async function openPage( throw new Error(`Browser VM launch already in progress${busyStreamId ? ` for ${busyStreamId}` : ""}`); } const maxActivePages = Number(config.max_active_pages ?? 1); - if (activePages.size >= maxActivePages) { - throw new Error(`Browser VM active page capacity reached (${activePages.size}/${maxActivePages}); close a page before launching another page`); + const livePages = liveCapacityPageCount(activePages, activeVms); + if (livePages >= maxActivePages) { + throw new Error(`Browser VM active page capacity reached (${livePages}/${maxActivePages}); close a page before launching another page`); } await retireConflictingIdleVmsForProfile(config, activeVms, vmKey, profileLeaseKey); await retireNonReusableIdleVmsForSinglePageRuntime(config, activeVms, vmKey); @@ -3041,16 +3669,17 @@ async function openPage( supervisor_result: result, }, ); + const acquiredVmKey = vmKey || result.control_socket_path || null; activePages.set(result.page_id, { page: result, launch, - vm_key: vmKey, + vm_key: acquiredVmKey, launcher_child: launcher.child || null, process_binding: result.process, started_at: new Date(startedAt).toISOString(), }); - if (vmKey && result.control_socket_path) { - activeVms.set(vmKey, { + if (acquiredVmKey && result.control_socket_path) { + activeVms.set(acquiredVmKey, { control_socket_path: result.control_socket_path, isolation: result.isolation, launcher_child: launcher.child || null, @@ -3069,7 +3698,7 @@ async function openPage( const handleLauncherExit = (code, exitSignal) => { if (launcherExitHandled) return; launcherExitHandled = true; - const currentVm = vmKey ? activeVms.get(vmKey) : null; + const currentVm = acquiredVmKey ? activeVms.get(acquiredVmKey) : null; const affectedPages = currentVm?.launcher_child === launcher.child ? [...currentVm.pages] @@ -3078,15 +3707,28 @@ async function openPage( : [activePages.get(result.page_id)].filter( (record) => record?.launcher_child === launcher.child, ); + const stderrTail = String(launcher.stderr_tail?.text || "").trim(); + const launchSettlement = parseVzLaunchSettlementFromStderr(stderrTail); + const nextState = + launchSettlement?.state === LAUNCH_SETTLEMENT_TERMINAL + ? LAUNCH_SETTLEMENT_TERMINAL + : LAUNCH_SETTLEMENT_PENDING; for (const affected of affectedPages) { affected.cleanup_pending = true; + affected.stderr_tail = launcher.stderr_tail; try { recordLaunchReconciliation( launchReconciliations, affected.launch, - LAUNCH_SETTLEMENT_PENDING, + nextState, { effects: { page_acquired: true, vm_acquired: true }, + ...(launchSettlement + ? { + launch_settlement_result: launchSettlement, + ...profileDurabilityFields(launchSettlement), + } + : {}), }, ); } catch (error) { @@ -3096,17 +3738,24 @@ async function openPage( ); logEvent("launch_reconciliation_persist_failed", { stream_id: affected.launch.stream_id, - intended_state: LAUNCH_SETTLEMENT_PENDING, + intended_state: nextState, error: error instanceof Error ? error.message : String(error), }); } } + if (currentVm?.launcher_child === launcher.child && acquiredVmKey) { + clearIdleVmShutdown(currentVm); + activeVms.delete(acquiredVmKey); + } logEvent("launcher_exit", { request_id: requestId, page_id: result.page_id, code, signal: exitSignal, - settlement: LAUNCH_SETTLEMENT_PENDING, + settlement: nextState, + live_capacity_pages: liveCapacityPageCount(activePages, activeVms), + pending_cleanup_pages: pendingCleanupPageIds(activePages, activeVms).length, + ...(stderrTail ? { stderr_tail: stderrTail } : {}), }); }; launcher.child.once("exit", handleLauncherExit); @@ -3218,9 +3867,11 @@ async function shutdownPage( record, ); } - let vmKey = record?.vm_key || null; - let vmRecord = vmKey ? activeVms.get(vmKey) : null; - if (!vmRecord) { + let vmKey = null; + let vmRecord = ownedVmRecord(record, pageId, activeVms); + if (vmRecord) { + vmKey = record.vm_key; + } else { for (const [candidateKey, candidate] of activeVms.entries()) { if (candidate.pages.has(pageId)) { vmKey = candidateKey; @@ -3230,7 +3881,16 @@ async function shutdownPage( } } - const recordCleanupPending = () => + const recordCleanupPending = () => { + const current = launchReconciliations.records.get( + launchReconciliationKey( + runtimeCleanup.generation, + runtimeCleanup.stream_id, + ), + ); + if (current?.state === LAUNCH_SETTLEMENT_TERMINAL) { + return; + } recordLaunchReconciliation( launchReconciliations, durableRecord.launch, @@ -3240,6 +3900,7 @@ async function shutdownPage( cleanup_binding: runtimeCleanup, }, ); + }; const runShutdownProgram = async (page) => { if (!config.shutdown_program) return; const request = JSON.stringify({ @@ -3261,6 +3922,53 @@ async function shutdownPage( }; if (!record && !vmRecord) { + if ( + durableRecord.state === LAUNCH_SETTLEMENT_PENDING && + durableRecord.launch_settlement_result && + !hostProcessIsAlive(runtimeCleanup.process) + ) { + const delayedEffects = await exactTransportCleanupEffects( + runtimeCleanup, + { + graceful: false, + terminal_record: false, + vz_launch_settlement: durableRecord.launch_settlement_result, + }, + ); + const delayedUnresolved = Object.entries(delayedEffects) + .filter(([, value]) => value !== true) + .map(([key]) => key); + if (delayedUnresolved.length === 0) { + const receipt = terminalCleanupReceipt( + runtimeCleanup, + { + page_absent: true, + child_absent: true, + vm_absent: true, + route_absent: true, + socket_absent: true, + ...delayedEffects, + }, + { + already_absent: true, + delayed_turn_port_absence: true, + ...profileDurabilityFields(durableRecord), + ...profileDurabilityFields(durableRecord.launch_settlement_result), + }, + ); + markLaunchReconciliationTerminal( + launchReconciliations, + runtimeCleanup.generation, + runtimeCleanup.stream_id, + { + page_acquired: true, + vm_acquired: true, + }, + runtimeCleanup.transport_authority ? receipt : undefined, + ); + return receipt; + } + } if (durableRecord.state !== LAUNCH_SETTLEMENT_TERMINAL) { throw new Error( "Browser VM cleanup remains indeterminate after service restart: exact owned launcher unavailable", @@ -3280,7 +3988,11 @@ async function shutdownPage( socket_absent: true, ...transportEffects, }, - { already_absent: true }, + { + already_absent: true, + ...profileDurabilityFields(durableRecord), + ...profileDurabilityFields(durableRecord.launch_settlement_result), + }, ); } if ( @@ -3304,7 +4016,12 @@ async function shutdownPage( let closeError = null; const controlSocketPath = vmRecord?.control_socket_path || record?.page?.control_socket_path || ""; - if (controlSocketPath) { + if (controlSocketPath && runtimeCleanup.transport_authority) { + logEvent("vm_retirement_defers_guest_page_close", { + page_id: pageId, + reason: "supervisor_owns_guest_profile_flush", + }); + } else if (controlSocketPath) { try { await postJsonOverUnix( controlSocketPath, @@ -3348,14 +4065,6 @@ async function shutdownPage( "Browser VM exact owned launcher did not produce an exit receipt", ); } - if ( - runtimeCleanup.transport_authority && - launcherTermination?.graceful !== true - ) { - throw new Error( - "Browser VZ exact launcher did not produce a graceful terminal exit", - ); - } } catch (error) { cleanupErrors.push(error instanceof Error ? error.message : String(error)); } @@ -3386,9 +4095,30 @@ async function shutdownPage( `Browser VM cleanup remains indeterminate: ${externallyUnresolved.join(", ")}`, ); } + const launchSettlement = + parseVzLaunchSettlementFromStderr( + String( + record?.stderr_tail?.text || + record?.launcher_child?.stderr_tail?.text || + "", + ), + ) || + launchReconciliations.records.get( + launchReconciliationKey( + runtimeCleanup.generation, + runtimeCleanup.stream_id, + ), + )?.launch_settlement_result || + durableRecord.launch_settlement_result; const transportEffects = await exactTransportCleanupEffects( runtimeCleanup, - launcherTermination, + { + ...launcherTermination, + ...(launchSettlement?.state === LAUNCH_SETTLEMENT_TERMINAL + ? { terminal_record: true } + : {}), + vz_launch_settlement: launchSettlement, + }, ); const unresolvedTransport = Object.entries(transportEffects) .filter(([, value]) => value !== true) @@ -3420,22 +4150,37 @@ async function shutdownPage( { forced_vm_retirement: Boolean(closeError), ...(closeError ? { control_error: closeError } : {}), + ...profileDurabilityFields(durableRecord), + ...profileDurabilityFields(durableRecord.launch_settlement_result), + ...profileDurabilityFields(launchSettlement), }, ); markVmPendingLaunchReconciliationsTerminal( launchReconciliations, vmRecord, ); - markLaunchReconciliationTerminal( - launchReconciliations, - runtimeCleanup.generation, - runtimeCleanup.stream_id, - { - page_acquired: true, - vm_acquired: true, - }, - runtimeCleanup.transport_authority ? receipt : undefined, - ); + try { + markLaunchReconciliationTerminal( + launchReconciliations, + runtimeCleanup.generation, + runtimeCleanup.stream_id, + { + page_acquired: true, + vm_acquired: true, + }, + runtimeCleanup.transport_authority ? receipt : undefined, + ); + } catch (error) { + if (launchSettlement?.state !== LAUNCH_SETTLEMENT_TERMINAL) { + throw error; + } + logEvent("launch_reconciliation_persist_failed", { + stream_id: runtimeCleanup.stream_id, + page_id: pageId, + intended_state: LAUNCH_SETTLEMENT_TERMINAL, + error: error instanceof Error ? error.message : String(error), + }); + } return receipt; } @@ -3450,7 +4195,7 @@ function activePageGuestControl(activePages, activeVms, pageId) { if (record.cleanup_pending === true) { throw new Error("browser page cleanup is pending"); } - const vmRecord = record.vm_key ? activeVms.get(record.vm_key) : null; + const vmRecord = ownedVmRecord(record, pageId, activeVms); const controlSocketPath = vmRecord?.control_socket_path || record.page?.control_socket_path || ""; if (!controlSocketPath) { throw new Error("browser page guest control socket is unavailable"); @@ -3643,6 +4388,8 @@ function main() { return; } if (req.method === "GET" && url.pathname === "/status") { + const livePages = liveCapacityPageCount(activePages, activeVms); + const pendingIds = pendingCleanupPageIds(activePages, activeVms); sendJson(200, { schema: "elastos.browser.vm-control-service.status/v1", ok: true, @@ -3651,16 +4398,22 @@ function main() { uptime_ms: Math.max(0, Date.now() - serviceStartedAtMs), config_fingerprint: config.config_fingerprint || null, control_service: controlServiceIdentity, - active_pages: activePages.size, + active_pages: livePages, active_vms: activeVms.size, warm_vms: [...activeVms.values()].filter((record) => record.pages.size === 0).length, max_active_pages: Number(config.max_active_pages ?? 1), idle_vm_keepalive_ms: idleVmKeepaliveMs(config), reuse_idle_vms: idleVmReuseEnabled(config), hibernation_mode: config.hibernation_mode || "off", - capacity_available: activePages.size < Number(config.max_active_pages ?? 1), - page_ids: [...activePages.keys()], - active_stream_ids: [...activePages.values()].map((record) => record.page?.stream_id || record.launch?.stream_id || ""), + capacity_available: livePages < Number(config.max_active_pages ?? 1), + pending_cleanup_pages: pendingIds.length, + pending_cleanup_page_ids: pendingIds, + page_ids: [...activePages.keys()].filter((pageId) => + pageHoldsLiveCapacity(pageId, activePages.get(pageId), activeVms), + ), + active_stream_ids: [...activePages.entries()] + .filter(([pageId, record]) => pageHoldsLiveCapacity(pageId, record, activeVms)) + .map(([, record]) => record.page?.stream_id || record.launch?.stream_id || ""), pending_launches: pendingLaunches.size, pending_stream_ids: [...pendingLaunches.values()].map((launch) => launch.stream_id), lifecycle: lifecycleStatus(config, activePages, activeVms, pendingLaunches), @@ -3796,10 +4549,11 @@ function main() { if (req.method === "POST" && url.pathname === "/launches/reconcile") { sendJson( 200, - reconcileLaunch( + await reconcileLaunch( launchReconciliations, activePages, await readJsonBody(req), + Number(config.shutdown_timeout_ms ?? 30000), ), ); return; diff --git a/scripts/browser-vm-control-stdout-ready.test.mjs b/scripts/browser-vm-control-stdout-ready.test.mjs new file mode 100644 index 000000000..cb22a704a --- /dev/null +++ b/scripts/browser-vm-control-stdout-ready.test.mjs @@ -0,0 +1,27 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import test from "node:test"; + +const source = fs.readFileSync( + new URL("./browser-vm-control-service.mjs", import.meta.url), + "utf8", +); +const match = source.match( + /function persistentLauncherReadyLine\(stdout\) \{[\s\S]*?\n\}/, +); +assert.ok(match, "persistentLauncherReadyLine must exist"); +const persistentLauncherReadyLine = new Function( + `${match[0]}; return persistentLauncherReadyLine;`, +)(); + +test("persistent launcher ignores debugfs stdout before supervisor JSON", () => { + const result = { + schema: "elastos.browser.engine.supervisor-result/v1", + page_id: "page:ready", + }; + assert.equal( + persistentLauncherReadyLine(`Allocated inode: 728\n${JSON.stringify(result)}\n`), + JSON.stringify(result), + ); + assert.equal(persistentLauncherReadyLine("Allocated inode: 728\n"), null); +}); diff --git a/scripts/browser-wallet-account-consent-smoke.mjs b/scripts/browser-wallet-account-consent-smoke.mjs index 6cd6f84ef..bce7dea8a 100644 --- a/scripts/browser-wallet-account-consent-smoke.mjs +++ b/scripts/browser-wallet-account-consent-smoke.mjs @@ -58,6 +58,56 @@ assert.equal( accountId, ); +const connectorAccountId = "wallet:eip155:20:0x3333333333333333333333333333333333333333"; +const connectorRuntime = { + wallet: { + ...runtime.wallet, + default_account_id: connectorAccountId, + accounts: [ + { + account_id: connectorAccountId, + chain_namespace: "eip155:20", + address: "0x3333333333333333333333333333333333333333", + proof_type: "siwe", + connector_id: "wallet-metamask", + }, + ], + }, +}; +assert.equal( + walletRuntimeRequestedAccount(connectorRuntime, { chain_namespace: "eip155:20" }) + .connector_id, + "wallet-metamask", +); + +const bothRuntime = { + wallet: { + ...runtime.wallet, + default_account_id: connectorAccountId, + accounts: [ + { + account_id: accountId, + chain_namespace: "eip155:20", + address, + proof_type: "managed_evm", + }, + { + account_id: connectorAccountId, + chain_namespace: "eip155:20", + address: "0x3333333333333333333333333333333333333333", + proof_type: "siwe", + connector_id: "wallet-metamask", + }, + ], + }, +}; +assert.equal( + walletRuntimeRequestedAccount(bothRuntime, { chain_namespace: "eip155:20" }) + .connector_id, + "wallet-metamask", + "selected connector wins when a managed account is also present", +); + runtime.pendingAccountAccess.set("wallet-approval:consent", { executionContextId: context.executionContextId, origin: context.pageOrigin, diff --git a/scripts/browser-window-close-handshake.test.mjs b/scripts/browser-window-close-handshake.test.mjs index c6f1fbad7..73a9821a8 100644 --- a/scripts/browser-window-close-handshake.test.mjs +++ b/scripts/browser-window-close-handshake.test.mjs @@ -1295,12 +1295,17 @@ test("unexpected close errors return error receipts without implying terminal", test("iframe unload and refresh remain teardown-only", () => { const unloadBlock = extractFunction(browserSource, "releaseRuntimePageForUnload"); assert.doesNotMatch(unloadBlock, /closeRuntimePage\s*\(/); + assert.doesNotMatch(unloadBlock, /display_attach/); + assert.doesNotMatch(unloadBlock, /prepareRecoverableDisplayAttach/); + assert.doesNotMatch(unloadBlock, /startRecoverableDisplayAttach/); + assert.doesNotMatch(browserSource, /prepareRecoverableDisplayAttach/); + assert.match(browserSource, /function startRecoverableDisplayAttach/); assert.match( browserSource, - /window\.addEventListener\("beforeunload", \(\) => \{\s*releaseRuntimePageForUnload\(\);\s*\}\);/, + /window\.addEventListener\("beforeunload", \(\) => \{\s*startRecoverableDisplayAttach\(\);\s*releaseRuntimePageForUnload\(\);\s*\}\);/, ); assert.match( browserSource, - /window\.addEventListener\("pagehide", releaseRuntimePageForUnload\);/, + /window\.addEventListener\("pagehide", \(event\) => \{/, ); }); diff --git a/scripts/build/browser-webrtc-send-diagnostic.py b/scripts/build/browser-webrtc-send-diagnostic.py new file mode 100644 index 000000000..3694c357e --- /dev/null +++ b/scripts/build/browser-webrtc-send-diagnostic.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +"""Patch the in-image Selkies module for an opt-in webrtcbin sink experiment. + +Apply only to a task rootfs clone. The observer counts RTP bytes that enter +webrtcbin after the audio-queue src. It changes no media properties and +collects no PCM, payload, ICE credentials, or addresses. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import tempfile + + +# gstwebrtc_app.py from installed image 76967c5b, which already has the +# audio-queue diagnostic. +BASE_SHA256 = "d1813fc03015077d0c96e697123c53115220d870d44616431594bbc77fda9168" +MARKER = "# ElastOS bounded webrtc send diagnostic v1" +OBSERVER = r''' +# ElastOS bounded webrtc send diagnostic v1 +class _ElastosWebrtcSendDiagnostic: + def __init__(self, webrtcbin, gst, log): + import threading + self.webrtcbin, self.gst, self.log = webrtcbin, gst, log + self.lock = threading.Lock() + self.started = time.monotonic() + self.deadline = self.started + 60.0 + self.next_report = self.started + 0.2 + self.closed = False + self.rows = 0 + self.error = None + self.pads = {} + self.probes = [] + try: + mask = (gst.PadProbeType.BUFFER | gst.PadProbeType.BUFFER_LIST) + iterator = webrtcbin.iterate_sink_pads() + while True: + result, pad = iterator.next() + if result != gst.IteratorResult.OK: + break + name = pad.get_name() + self.pads[name] = dict(packets=0, bytes=0) + self.probes.append((pad, pad.add_probe(mask, self.observe, name))) + if not self.probes: + self.error = "no_sink_pads" + except Exception: + self.error = "attach_failed" + + def observe(self, _pad, info, name): + try: + with self.lock: + if self.closed or time.monotonic() >= self.deadline: + return self.gst.PadProbeReturn.OK + state = self.pads.setdefault(name, dict(packets=0, bytes=0)) + if info.type & self.gst.PadProbeType.BUFFER_LIST: + buffers = info.get_buffer_list() + for index in range(buffers.length()): + size = buffers.get(index).get_size() + state["packets"] += 1 + state["bytes"] += size + elif info.type & self.gst.PadProbeType.BUFFER: + size = info.get_buffer().get_size() + state["packets"] += 1 + state["bytes"] += size + except Exception: + with self.lock: + self.error = "probe_failed" + return self.gst.PadProbeReturn.OK + + def tick(self): + now = time.monotonic() + if self.closed: + return + if self.error or now >= self.deadline: + self.stop("error" if self.error else "timeout") + elif now >= self.next_report: + self.next_report = now + 0.2 + self.report("sample") + + def report(self, reason): + with self.lock: + if self.rows >= 301: + return + self.rows += 1 + row = dict(schema="elastos.browser.webrtc-send-diagnostic/v1", + reason=reason, row=self.rows, wall_time_ms=time.time() * 1000, + elapsed_ms=(time.monotonic() - self.started) * 1000, + measurement_limit_ms=60000, error=self.error, + pads={name: dict(state) for name, state in self.pads.items()}) + self.log.info("webrtc_send_diagnostic %s", json.dumps(row, separators=(",", ":"))) + + def stop(self, reason="stop"): + with self.lock: + if self.closed: + return + self.closed = True + for pad, probe in self.probes: + pad.remove_probe(probe) + self.probes.clear() + self.report(reason) +''' + +AUDIO_HOOK = ''' if os.environ.get("ELASTOS_BROWSER_WEBRTC_SEND_DIAGNOSTIC") == "1": + self._webrtc_send_diagnostic = _ElastosWebrtcSendDiagnostic(self.webrtcbin, Gst, logger) +''' +BUS_HOOK = ''' webrtc = getattr(self, "_webrtc_send_diagnostic", None) + if webrtc is not None: + webrtc.tick() if running else webrtc.stop("bus_stop") +''' +STOP_HOOK = ''' webrtc = getattr(self, "_webrtc_send_diagnostic", None) + if webrtc is not None: + webrtc.stop() +''' +HOOKS = ( + (" # [END build_audio_pipeline]\n", AUDIO_HOOK), + (" await asyncio.sleep(0.1)\n", BUS_HOOK), + (' logger.info("stopping pipeline")\n', STOP_HOOK), +) + + +def patch_source(source, expected_sha256=BASE_SHA256): + original = source + if MARKER in source: + if not source.endswith(OBSERVER): + raise ValueError("diagnostic payload differs from this patcher") + source = source[:-len(OBSERVER)] + for anchor, hook in HOOKS: + if source.count(hook + anchor) != 1: + raise ValueError("incomplete diagnostic hook") + source = source.replace(hook + anchor, anchor, 1) + if hashlib.sha256(source.encode()).hexdigest() != expected_sha256: + raise ValueError("Selkies input SHA-256 differs from the in-image source") + for anchor, hook in HOOKS: + if source.count(anchor) != 1: + raise ValueError("Selkies webrtc diagnostic insertion point changed") + source = source.replace(anchor, hook + anchor, 1) + result = source + OBSERVER + compile(result, "gstwebrtc_app.py", "exec") + if MARKER in original and original != result: + raise ValueError("diagnostic patch is inconsistent") + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("path", type=Path, help="gstwebrtc_app.py inside the task rootfs clone") + args = parser.parse_args() + before = args.path.read_bytes() + after = patch_source(before.decode()).encode() + if after != before: + temporary = None + try: + with tempfile.NamedTemporaryFile(dir=args.path.parent, delete=False) as handle: + temporary = Path(handle.name) + handle.write(after) + handle.flush() + os.fsync(handle.fileno()) + os.chmod(temporary, args.path.stat().st_mode & 0o777) + os.replace(temporary, args.path) + finally: + if temporary is not None and temporary.exists(): + temporary.unlink() + print(json.dumps(dict(schema="elastos.browser.webrtc-send-patch/v1", + input_sha256=hashlib.sha256(before).hexdigest(), + output_sha256=hashlib.sha256(after).hexdigest(), changed=after != before, + enabled_by_default=False, max_seconds=60, interval_ms=200))) + + +if __name__ == "__main__": + main() diff --git a/scripts/build/browser-webrtc-send-diagnostic.test.py b/scripts/build/browser-webrtc-send-diagnostic.test.py new file mode 100644 index 000000000..18a4a5d43 --- /dev/null +++ b/scripts/build/browser-webrtc-send-diagnostic.test.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""Source checks for the webrtcbin sink diagnostic. Real Gst belongs to the builder.""" + +import hashlib +import json +from pathlib import Path +import runpy +import unittest +from types import SimpleNamespace + + +HELPER = runpy.run_path(str(Path(__file__).with_name("browser-webrtc-send-diagnostic.py"))) +DUMPED = Path("/Users/anders/Code/elastos-runtime/.git/browser-analysis/u9-r8-gstwebrtc-app-76967c5b.py") + + +class Pad: + def __init__(self, name): + self.name = name + self.callbacks = {} + + def get_name(self): + return self.name + + def add_probe(self, mask, callback, name): + self.callbacks[1] = (mask, callback, name) + return 1 + + def remove_probe(self, probe): + del self.callbacks[probe] + + +GST = SimpleNamespace( + PadProbeType=SimpleNamespace(BUFFER=1, BUFFER_LIST=2), + PadProbeReturn=SimpleNamespace(OK=0), + IteratorResult=SimpleNamespace(OK=1, DONE=2), +) + + +class PadIterator: + def __init__(self, pads): + self.pads = list(pads) + + def next(self): + if not self.pads: + return GST.IteratorResult.DONE, None + return GST.IteratorResult.OK, self.pads.pop(0) + + +class Webrtc: + def __init__(self, pads): + self._pads = pads + + def iterate_sink_pads(self): + return PadIterator(self._pads) + + +class Buffer: + def __init__(self, size): + self._size = size + + def get_size(self): + return self._size + + +class TestPatch(unittest.TestCase): + def test_dumped_image_source_matches_pin(self): + text = DUMPED.read_text() + self.assertEqual(hashlib.sha256(text.encode()).hexdigest(), HELPER["BASE_SHA256"]) + + def test_patch_is_idempotent_on_dumped_source(self): + text = DUMPED.read_text() + once = HELPER["patch_source"](text) + twice = HELPER["patch_source"](once) + self.assertEqual(once, twice) + self.assertIn("webrtc_send_diagnostic", once) + self.assertIn("ELASTOS_BROWSER_WEBRTC_SEND_DIAGNOSTIC", once) + compile(once, "gstwebrtc_app.py", "exec") + + def test_observer_counts_sink_bytes(self): + pad = Pad("sink_1") + webrtc = Webrtc([pad]) + log = SimpleNamespace(rows=[], info=lambda tag, payload: log.rows.append((tag, json.loads(payload)))) + ns = {} + exec("import time, json\n" + HELPER["OBSERVER"], { + "time": __import__("time"), + "json": __import__("json"), + }, ns) + diagnostic = ns["_ElastosWebrtcSendDiagnostic"](webrtc, GST, log) + self.assertEqual(list(diagnostic.pads), ["sink_1"]) + info = SimpleNamespace(type=GST.PadProbeType.BUFFER, get_buffer=lambda: Buffer(120)) + self.assertEqual(diagnostic.observe(pad, info, "sink_1"), GST.PadProbeReturn.OK) + diagnostic.report("sample") + self.assertTrue(log.rows[-1][0].startswith("webrtc_send_diagnostic")) + self.assertEqual(log.rows[-1][1]["pads"]["sink_1"]["bytes"], 120) + self.assertEqual(log.rows[-1][1]["pads"]["sink_1"]["packets"], 1) + diagnostic.stop() + self.assertEqual(pad.callbacks, {}) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/build/stage-browser-vm-target.sh b/scripts/build/stage-browser-vm-target.sh index 60fabbd2e..959ec905a 100755 --- a/scripts/build/stage-browser-vm-target.sh +++ b/scripts/build/stage-browser-vm-target.sh @@ -1392,6 +1392,84 @@ def patch_selkies_input_writer(source): raise SystemExit("browser-vm-selkies-start: Selkies input writer patch target not found") return source.replace(input_marker, input_patch, 1).replace(import_marker, import_patch, 1).replace(close_marker, close_patch, 1) +def patch_selkies_display_renegotiate(signalling_source, gst_source, main_source): + signalling_cb = " self.on_session = lambda peer_id, meta: logger.warn('unhandled on_session callback')\n" + signalling_cb_patch = signalling_cb + " self.on_renegotiate = lambda: logger.warn('unhandled on_renegotiate callback')\n" + signalling_json = ''' elif data.get("ice", None): + logger.info("received ICE") + logger.debug("ICE:\\n%s" % data.get("ice")) + self.on_ice(data['ice'].get('sdpMLineIndex'), + data['ice'].get('candidate')) + else: +''' + signalling_json_patch = ''' elif data.get("ice", None): + logger.info("received ICE") + logger.debug("ICE:\\n%s" % data.get("ice")) + self.on_ice(data['ice'].get('sdpMLineIndex'), + data['ice'].get('candidate')) + elif data.get("elastos_display_renegotiate") is True: + logger.info("received display renegotiate") + self.on_renegotiate() + else: +''' + gst_marker = " def __on_negotiation_needed(self, webrtcbin):\n" + gst_offer_plain = """ def request_display_offer(self): + if self.webrtcbin: + self.__on_negotiation_needed(self.webrtcbin) + + def __on_negotiation_needed(self, webrtcbin): +""" + gst_offer_restart = """ def request_display_offer(self): + if not self.webrtcbin: + return + logger.info("creating display offer with ICE restart") + options = Gst.Structure("application/x-gst-webrtc-offer-options") + options.set_value("IceRestart", True) + promise = Gst.Promise.new_with_change_func( + self.__on_offer_created, self.webrtcbin, None) + self.webrtcbin.emit("create-offer", options, promise) + + def __on_negotiation_needed(self, webrtcbin): +""" + gst_patch = """ def request_display_offer(self): + logger.info("recreating display pipeline for a new viewer") + audio_only = bool(getattr(self, "_elastos_audio_only", False)) + self.stop_pipeline() + self.start_pipeline(audio_only=audio_only) + + def __on_negotiation_needed(self, webrtcbin): +""" + start_marker = ' logger.info("starting pipeline")\n' + start_patch = ' self._elastos_audio_only = audio_only\n logger.info("starting pipeline")\n' + main_marker = " app.on_sdp = signalling.send_sdp\n audio_app.on_sdp = audio_signalling.send_sdp\n" + main_patch = main_marker + " signalling.on_renegotiate = app.request_display_offer\n audio_signalling.on_renegotiate = audio_app.request_display_offer\n" + if signalling_cb_patch not in signalling_source: + if signalling_source.count(signalling_cb) != 1: + raise SystemExit("browser-vm-selkies-start: Selkies renegotiate callback target not found") + signalling_source = signalling_source.replace(signalling_cb, signalling_cb_patch, 1) + if signalling_json_patch not in signalling_source: + if signalling_source.count(signalling_json) != 1: + raise SystemExit("browser-vm-selkies-start: Selkies renegotiate JSON target not found") + signalling_source = signalling_source.replace(signalling_json, signalling_json_patch, 1) + if gst_patch not in gst_source: + if gst_offer_restart in gst_source: + gst_source = gst_source.replace(gst_offer_restart, gst_patch, 1) + elif gst_offer_plain in gst_source: + gst_source = gst_source.replace(gst_offer_plain, gst_patch, 1) + elif gst_source.count(gst_marker) == 1: + gst_source = gst_source.replace(gst_marker, gst_patch, 1) + else: + raise SystemExit("browser-vm-selkies-start: Selkies renegotiate offer target not found") + if start_patch not in gst_source: + if gst_source.count(start_marker) != 1: + raise SystemExit("browser-vm-selkies-start: Selkies pipeline audio flag target not found") + gst_source = gst_source.replace(start_marker, start_patch, 1) + if main_patch not in main_source: + if main_source.count(main_marker) != 1: + raise SystemExit("browser-vm-selkies-start: Selkies renegotiate wiring target not found") + main_source = main_source.replace(main_marker, main_patch, 1) + return signalling_source, gst_source, main_source + main_text = patch_selkies_input_writer(patch_selkies_signaling_retries(main_path.read_text())) transport_helper_marker = "\ndef parse_rtc_config(data):\n" transport_helper_patch = ''' @@ -1654,6 +1732,12 @@ if audio_offer_patch not in text: if audio_offer_marker not in text: raise SystemExit("browser-vm-selkies-start: Selkies split audio offer patch target not found") text = text.replace(audio_offer_marker, audio_offer_patch, 1) +signalling_path = path.with_name("webrtc_signalling.py") +if not signalling_path.is_file(): + raise SystemExit("browser-vm-selkies-start: webrtc_signalling.py not found") +signalling_text = signalling_path.read_text() +signalling_text, text, main_text = patch_selkies_display_renegotiate(signalling_text, text, main_text) +signalling_path.write_text(signalling_text) path.write_text(text) main_path.write_text(main_text) PY diff --git a/scripts/lib/browser-journey-harness.test.mjs b/scripts/lib/browser-journey-harness.test.mjs index 546631ca3..b58b30473 100644 --- a/scripts/lib/browser-journey-harness.test.mjs +++ b/scripts/lib/browser-journey-harness.test.mjs @@ -10,17 +10,26 @@ import { browserJourneyTargetConfig, readBrowserJourneyReceipt, browserJourneyProfileStorage, browserJourneyProfileBinding } from "./browser-journey-target.mjs"; const source = readFileSync(new URL("../home-passkey-virtual-auth-smoke.mjs", import.meta.url), "utf8"); -function harnessFunction(name, globals = {}) { +function extractDeclaration(name) { const start = source.search(new RegExp(`(?:async )?function ${name}\\(`)); assert.ok(start >= 0, name); const next = source.slice(start + 1).search(/\n(?:async )?function /); - const declaration = source.slice(start, start + 1 + next); + return source.slice(start, start + 1 + next); +} +const browserJourneySummaryUrl = vm.runInNewContext( + `(${extractDeclaration("browserJourneySummaryUrl")})`, + { URL, URLSearchParams }, +); +function harnessFunction(name, globals = {}) { + const declaration = extractDeclaration(name); return vm.runInNewContext(`(${declaration})`, { URL, URLSearchParams, Date, performance, CHECK_BROWSER_CONTROLLED_MEDIA: false, CHECK_BROWSER_CONTROLLED_INSPECTION: false, CHECK_BROWSER_CONTROLLED_OPERATOR: false, CHECK_BROWSER_CONTROLLED_JOURNEY: false, + CHECK_BROWSER_RELOAD_CENSUS: false, BROWSER_RELOAD_OBSERVE_MS: 0, BROWSER_REMOTE_EXIT_ID: "", browserQualification: null, qualificationCancellation: null, BROWSER_JOURNEY_TARGET: browserJourneyTargetConfig(), browserJourneyEngineChoice, browserJourneyEngineRoute, browserJourneyFixtureUrl, browserJourneyProfileStorage, browserJourneyProfileBinding, + browserJourneySummaryUrl, readBrowserJourneyReceipt: (config, run, options) => readBrowserJourneyReceipt(config, run, { ...options, fetchImpl: globals.fetch || fetch }), ...globals }); } @@ -67,7 +76,7 @@ test("controlled fixture isolates runs, records bounded events and rejects malfo } }); -test("page acquisition fails immediately on its terminal open settlement", async () => { +test("page acquisition keeps a published page before a later failed open settlement", async () => { let inspected = false; const frame = { evaluate: async () => { inspected = true; return ""; } }; const wait = harnessFunction("waitForEmbeddedBrowserPage", { @@ -76,11 +85,11 @@ test("page acquisition fails immediately on its terminal open settlement", async await assert.rejects(wait(frame, [{ frame, at: Date.now(), status: 200, body: { schema: "elastos.browser.open-status/v1", status: "failed", error: { code: "launch-failed" } } }]), error => error.details.settlement.status === "failed" && error.details.stage === "browser:page-acquisition"); - assert.equal(inspected, false); + assert.equal(inspected, true); const active = { evaluate: async () => "page-current" }; assert.equal(await wait(active, [{ frame, at: Date.now(), body: {} }]), "page-current"); assert.equal(await wait(active, [{ frame: active, open_id: "old-open", body: {} }], new Set(["old-open"])), "page-current"); - await assert.rejects(wait(active, [{ frame: active, open_id: "new-open", body: {} }], new Set(["old-open"]))); + assert.equal(await wait(active, [{ frame: active, open_id: "new-open", body: {} }], new Set(["old-open"])), "page-current"); }); test("explicit Browser inventory deadline aborts a hung response body; default API calls keep their options", async () => { @@ -175,6 +184,52 @@ test("launcher reuses the selected Browser through exact focus without toggling assert.deepEqual(actions, ["captured", "focus-exact"]); }); +test("controlled journey opens a fresh Home Browser and leaves hidden leftovers unused", async () => { + const actions = []; + const frame = { url: () => "http://localhost/apps/browser/?browser_instance=fresh" }; + const handle = { contentFrame: async () => frame, waitForElementState: async () => {} }; + const locators = { + "#identity-menu-show-desktop": { evaluate: async () => { actions.push("show-desktop"); } }, + "#taskbar-targets [data-target=\"browser\"]": { first: () => ({ count: async () => 1, click: async options => { + assert.equal(options.button, "right"); + actions.push("shelf-menu"); + } }) }, + "#desktop-context-menu [data-context-action=\"open-target-new-window\"]": { + count: async () => 1, click: async () => actions.push("new-window"), + }, + "#desktop-context-menu [data-context-action=\"open-target\"]": { count: async () => 0 }, + "section.window[data-target=\"browser\"]:not(.hidden).window-active iframe.window-frame": { + elementHandle: async () => handle, + }, + }; + const gui = { + waitForFunction: async (callback, prior) => { + if (prior) { + assert.deepEqual(prior, ["leftover"]); + actions.push("fresh-src"); + } + }, + evaluate: async () => ["leftover"], + locator: selector => { + if (selector === "#setup-sheet") return { isVisible: async () => false }; + const found = locators[selector]; + assert.ok(found, selector); + return found; + }, + }; + const identity = { token: "fresh-token" }; + const open = harnessFunction("openDesktopAppWindow", { + CHECK_BROWSER_CONTROLLED_JOURNEY: true, + HOME_URL: "http://localhost/apps/home/", waitForSignedHome: async () => {}, + waitForCapsuleFrame: async () => gui, assert: (condition, message) => assert.ok(condition, message), + captureBrowserWindowIdentity: async actual => { assert.equal(actual, frame); return identity; }, + focusCapturedBrowserWindow: async () => { actions.push("focus-exact"); }, + }); + const page = { url: () => "http://localhost/apps/home/", goto: async () => actions.push("reload-home") }; + assert.equal(await open(page, "browser", actual => { assert.equal(actual, frame); actions.push("captured"); }), frame); + assert.deepEqual(actions, ["show-desktop", "shelf-menu", "new-window", "fresh-src", "captured", "focus-exact"]); +}); + function closeHarnessSurface() { const listeners = new Map(); const messageListeners = new Set(); @@ -1015,9 +1070,10 @@ for (const observation of ["ready", "document-transition", "unexpected-viewer-er const outcome = await run; assert.equal(outcome.ok, true); assert.deepEqual(Array.from(outcome.state_observation.steps, row => row.step), - ["runtime_summary", "remote_page_status", "viewer_document_metrics"]); + ["viewer_document_metrics", "runtime_summary", "remote_page_status"]); assert.ok(outcome.state_observation.steps.every(row => row.end_ms >= row.start_ms)); - assert.equal(outcome.state_observation.steps[0].outcome, "complete"); + assert.equal(outcome.state_observation.steps.find(row => row.step === "runtime_summary").outcome, "complete"); + assert.equal(outcome.state_observation.steps[0].outcome, observation === "document-transition" ? "failed" : "complete"); assert.deepEqual(actions, ["wait-commit", "reload-frame", "input"]); assert.match(requests[1].url, /\/pages\/runtime-owner\/status$/); } @@ -1123,6 +1179,87 @@ test("observer cancellation disposes a listener handle returned after its setup assert.ok([...listeners.values()].every(value => value.size === 0)); }); +test("viewer reload wrapper fetches fresh Runtime ownership while the reloaded document has no page yet", async () => { + let summaryCalls = 0, viewerPageId = "page-one"; + const firstSessions = { recoverable_page: { page_id: "page-one" } }; + const freshSessions = { recoverable_page: { page_id: "page-one" }, generation: "fresh" }; + const frame = { + url: () => "http://localhost:61510/apps/browser/?browser_instance=instance-one#home_token=token-one", + evaluate: async fn => { + if (fn.toString().includes("location.reload()")) return; + return { viewer: { page_id: viewerPageId, browser_instance: "instance-one", actual_url: "", document_id: 100 }, video: null }; + }, + waitForNavigation: async () => {}, + locator: () => ({ pressSequentially: async () => {} }), + }; + const signal = new AbortController().signal; + const wrapper = harnessFunction("runControlledBrowserViewerReload", { + readBrowserViewerReloadDocument, markStage: () => {}, assert: (value, message) => assert.ok(value, message), + fetch: async url => { + if (String(url).includes("/pages/")) return { ok: true, json: async () => ({ actual_url: "http://localhost:61511/nav" }) }; + assert.match(String(url), /remote_services=0/); + summaryCalls++; + return { ok: true, json: async () => ({ sessions: summaryCalls === 1 ? firstSessions : freshSessions }) }; + }, + observeControlledBrowserRequests: async () => () => {}, + diagnoseBrowserViewerReload: async callbacks => { + const first = await callbacks.readState({ signal, deadlineMs: 5000 }); + assert.equal(first.sessions, firstSessions); + assert.equal(first.viewer.page_id, "page-one"); + const afterFirst = summaryCalls; + viewerPageId = ""; + const second = await callbacks.readState({ signal, deadlineMs: 5000 }); + assert.equal(second.sessions, freshSessions); + assert.equal(second.viewer.page_id, ""); + assert.ok(summaryCalls > afterFirst); + return { ok: true }; + }, + }); + assert.equal((await wrapper({}, frame, "token-one", async () => {}, "http://localhost:61511/nav?run=fixture")).ok, true); +}); + +test("reload samples reuse last status for binding and do not wait on a hung Engine probe", async () => { + let releaseStatus; + const hung = new Promise(resolve => { releaseStatus = resolve; }); + let statusCalls = 0; + const sessions = { recoverable_page: { page_id: "page-one" } }; + const frame = { + url: () => "http://localhost:61510/apps/browser/?browser_instance=instance-one#home_token=token-one", + evaluate: async fn => { + if (fn.toString().includes("location.reload()")) return; + return { viewer: { page_id: "page-one", document_id: 100 }, video: { decoded_frames: 4 } }; + }, + waitForNavigation: async () => {}, + locator: () => ({ pressSequentially: async () => {} }), + }; + const signal = new AbortController().signal; + const wrapper = harnessFunction("runControlledBrowserViewerReload", { + readBrowserViewerReloadDocument, markStage: () => {}, assert: (value, message) => assert.ok(value, message), + fetch: async url => { + if (String(url).includes("/pages/")) { + statusCalls++; + if (statusCalls === 1) return { ok: true, json: async () => ({ page_id: "page-one" }) }; + return { ok: true, json: () => hung }; + } + return { ok: true, json: async () => ({ sessions }) }; + }, + observeControlledBrowserRequests: async () => () => {}, + diagnoseBrowserViewerReload: async callbacks => { + const first = await callbacks.readState({ signal, deadlineMs: 5000 }); + assert.equal(first.page_status.page_id, "page-one"); + const started = Date.now(); + const second = await callbacks.readState({ signal, deadlineMs: 5000 }); + assert.ok(Date.now() - started < 200); + assert.equal(second.page_status.page_id, "page-one"); + assert.equal(second.video.decoded_frames, 4); + releaseStatus({ page_id: "page-one" }); + return { ok: true }; + }, + }); + assert.equal((await wrapper({}, frame, "token-one", async () => {}, "http://localhost:61511/nav?run=fixture")).ok, true); + assert.equal(statusCalls, 2); +}); + for (const mode of ["body-abort", "overflow"]) test(`reload substep evidence is bounded and retains ${mode}`, async () => { const controller = new AbortController(); let release, inFlight, count = 0; @@ -1152,10 +1289,10 @@ for (const mode of ["body-abort", "overflow"]) test(`reload substep evidence is } else { let evidence; await assert.rejects(run, error => { evidence = error.details.viewer_reload; return evidence.failure === "state_deadline"; }); - assert.equal(evidence.state_observation.steps.length, 1); - const row = evidence.state_observation.steps[0]; - assert.equal(row.step, "runtime_summary"); assert.equal(row.outcome, "aborted"); - assert.ok(row.start_ms <= row.headers_ms && row.headers_ms <= row.end_ms); + const summary = evidence.state_observation.steps.find(row => row.step === "runtime_summary"); + assert.equal(summary.outcome, "aborted"); + assert.ok(summary.start_ms <= summary.headers_ms && summary.headers_ms <= summary.end_ms); + assert.ok(evidence.state_observation.steps.some(row => row.step === "viewer_document_metrics")); const frozen = JSON.stringify(evidence); release({ sessions: { recoverable_page: { page_id: "private-page" } } }); await inFlight; assert.equal(JSON.stringify(evidence), frozen); assert.equal(count, 1); diff --git a/scripts/lib/browser-journey-turn-interruption.mjs b/scripts/lib/browser-journey-turn-interruption.mjs index 3f7b2cfa5..95464f3bb 100644 --- a/scripts/lib/browser-journey-turn-interruption.mjs +++ b/scripts/lib/browser-journey-turn-interruption.mjs @@ -99,12 +99,27 @@ const system = { watchdog: startWatchdog, }; +function adapterControlSocket(doc, dataDir) { + requireProof(doc && typeof doc === "object" && Array.isArray(doc.adapters) && doc.adapters.length > 0 && + doc.adapters.length <= 16, "turn_adapter_invalid"); + const matches = doc.adapters.filter(adapter => adapter?.kind === "chromium_microvm" && + typeof adapter?.supervisor?.control_socket_path === "string" && + adapter.supervisor.env?.ELASTOS_BROWSER_VM_DATA_DIR === dataDir && + adapter.supervisor.env?.ELASTOS_BROWSER_VM_CONTROL_SOCKET === adapter.supervisor.control_socket_path); + requireProof(matches.length === 1 && isAbsolute(matches[0].supervisor.control_socket_path), + "turn_adapter_ambiguous_or_absent"); + return matches[0].supervisor.control_socket_path; +} + async function bind(options, io, remaining, stage = () => {}) { stage("input"); requireProof(io.platform === "darwin", "turn_interruption_requires_macos"); const { testHome, pageId, runtimeOrigin } = options; + const controlSocketPath = typeof options.controlSocketPath === "string" && options.controlSocketPath + ? options.controlSocketPath : undefined; requireProof(typeof testHome === "string" && isAbsolute(testHome) && - typeof pageId === "string" && /^page:vz-[0-9a-f]{64}$/.test(pageId), "turn_explicit_fixture_required"); + typeof pageId === "string" && /^page:vz-[0-9a-f]{64}$/.test(pageId) && + (controlSocketPath === undefined || isAbsolute(controlSocketPath)), "turn_explicit_fixture_required"); stage("home_path"); const home = await io.realpath(testHome); stage("personal_home_path"); @@ -112,17 +127,28 @@ async function bind(options, io, remaining, stage = () => {}) { const dataDir = join(home, "Library/Application Support/elastos"); stage("data_dir"); requireProof(await io.realpath(dataDir) === dataDir, "turn_data_dir_redirected"); - stage("restart_receipt"); - const receipt = JSON.parse(await io.ownedFile(join(dataDir, "receipts/mac-source-home-restart.json"))); stage("runtime_origin"); - const origin = new URL(runtimeOrigin), homeUrl = new URL(receipt.home_url); + const origin = new URL(runtimeOrigin); requireProof(origin.origin === runtimeOrigin && origin.protocol === "http:" && ["localhost", "127.0.0.1", "[::1]"].includes(origin.hostname) && - homeUrl.origin === origin.origin && homeUrl.pathname === "/apps/home/" && - !homeUrl.username && !homeUrl.password && !homeUrl.search && !homeUrl.hash && - receipt.schema === "elastos.mac-source-home-restart/v1" && receipt.ok === true && receipt.dry_run === false && - receipt.test_home === home && receipt.data_dir === dataDir, "turn_restart_receipt_mismatch"); - const controlSocket = join(home, "run/browser.sock"); + !origin.username && !origin.password, "turn_runtime_origin_invalid"); + let controlSocket; + if (controlSocketPath) { + stage("adapter_config"); + const resolved = adapterControlSocket( + JSON.parse(await io.ownedFile(join(dataDir, "config/browser-engine-adapter.json"))), dataDir); + requireProof(resolved === controlSocketPath, "turn_control_socket_mismatch"); + controlSocket = resolved; + } else { + stage("restart_receipt"); + const receipt = JSON.parse(await io.ownedFile(join(dataDir, "receipts/mac-source-home-restart.json"))); + const homeUrl = new URL(receipt.home_url); + requireProof(homeUrl.origin === origin.origin && homeUrl.pathname === "/apps/home/" && + !homeUrl.username && !homeUrl.password && !homeUrl.search && !homeUrl.hash && + receipt.schema === "elastos.mac-source-home-restart/v1" && receipt.ok === true && receipt.dry_run === false && + receipt.test_home === home && receipt.data_dir === dataDir, "turn_restart_receipt_mismatch"); + controlSocket = join(home, "run/browser.sock"); + } stage("control_socket"); await io.socket(controlSocket); stage("control_status"); diff --git a/scripts/lib/browser-journey-turn-interruption.test.mjs b/scripts/lib/browser-journey-turn-interruption.test.mjs index 4e3d205ad..3fdca4129 100644 --- a/scripts/lib/browser-journey-turn-interruption.test.mjs +++ b/scripts/lib/browser-journey-turn-interruption.test.mjs @@ -87,6 +87,88 @@ function fixture() { return { state, io }; } +const adapterSocket = "/tmp/elastos-browser-fixture-vm-control.sock"; +const adapterJournal = `${adapterSocket}.launch-reconciliations.json`; +const adapterDoc = { + adapters: [{ + kind: "chromium_microvm", + supervisor: { + control_socket_path: adapterSocket, + env: { ELASTOS_BROWSER_VM_DATA_DIR: data, ELASTOS_BROWSER_VM_CONTROL_SOCKET: adapterSocket }, + }, + }], +}; + +function adapterFixture() { + const { state, io } = fixture(); + state.adapter = adapterDoc; + state.status.control_service.control_socket_path = adapterSocket; + state.record.control_service.control_socket_path = adapterSocket; + state.record.cleanup_binding.control_service.control_socket_path = adapterSocket; + state.record.cleanup_binding.shutdown_socket_path = adapterSocket; + const originalOwned = io.ownedFile; + const originalPids = io.socketPids; + io.ownedFile = async (path, max) => { + if (path === `${data}/config/browser-engine-adapter.json`) { + state.reads.push(path); + return JSON.stringify(state.adapter); + } + if (path === adapterJournal) { + state.reads.push(path); + assert.equal(max, 4 * 1024 * 1024); + return JSON.stringify(state.journal); + } + return originalOwned(path, max); + }; + io.socketPids = async path => path === adapterSocket ? state.controlPids : originalPids(path); + return { state, io }; +} + +test("adapter identity binds the live control socket without a matching restart receipt", async () => { + const { state, io } = adapterFixture(); + state.receipt.home_url = "http://localhost:61510/home/"; + state.receipt.ok = false; + const driver = await __test.createDriver({ ...options, controlSocketPath: adapterSocket }, io); + assert.equal(driver.evidence.bound, true); + assert.ok(state.reads.includes(`${data}/config/browser-engine-adapter.json`)); + assert.ok(state.reads.includes(adapterJournal)); + assert.ok(!state.reads.includes(`${data}/receipts/mac-source-home-restart.json`)); + await driver.cut({ timeoutMs: 1000 }); + await driver.restore({ timeoutMs: 1000 }); + assert.equal(driver.evidence.restored, true); + const evidence = JSON.stringify(driver.evidence); + assert.ok(!evidence.includes(adapterSocket)); + assert.ok(!evidence.includes(home)); +}); + +test("adapter identity rejects a control socket that the adapter does not own", async () => { + const { io } = adapterFixture(); + await assert.rejects(__test.createDriver({ ...options, controlSocketPath: "/tmp/foreign.sock" }, io), error => { + assert.equal(error.message, "turn_control_socket_mismatch"); + assert.ok(!JSON.stringify(error).includes(adapterSocket)); + return true; + }); +}); + +test("adapter identity filesystem failure exposes only the fixed stage", async () => { + const { io } = adapterFixture(); + const original = io.ownedFile; + io.ownedFile = async (path, max) => { + if (path.endsWith("browser-engine-adapter.json")) { + throw Object.assign(new Error("ENOENT private adapter NEVER-PRINT-THIS-SECRET"), { code: "ENOENT" }); + } + return original(path, max); + }; + await assert.rejects(__test.createDriver({ ...options, controlSocketPath: adapterSocket }, io), error => { + assert.equal(error.message, "turn_binding_adapter_config_failed"); + assert.deepEqual(error.evidence.failure, { + stage: "adapter_config", code: "turn_binding_adapter_config_failed", errno: "ENOENT", + }); + assert.ok(!JSON.stringify(error).includes("NEVER-PRINT")); + return true; + }); +}); + test("creation only binds; cut revalidates and starts watchdog; restore is idempotent and redacted", async () => { const { state, io } = fixture(); const driver = await __test.createDriver(options, io); diff --git a/scripts/lib/browser-journey-viewer-reload.mjs b/scripts/lib/browser-journey-viewer-reload.mjs index 7bc7566ab..3e84188ca 100644 --- a/scripts/lib/browser-journey-viewer-reload.mjs +++ b/scripts/lib/browser-journey-viewer-reload.mjs @@ -2,9 +2,40 @@ import { createHash } from "node:crypto"; import { browserJourneyBinding, browserJourneyVideo } from "./browser-journey-recovery.mjs"; const WINDOW_MS = 5000, BASELINE_MS = 10000, POLL_MS = 250, SUFFIX = "-reload"; +function signalingCensus(status) { + const signaling = status?.webrtc_signaling; + const picked = {}; + if (signaling && typeof signaling === "object") { + for (const key of [ + "browser_answers_received", + "browser_candidates_received", + "selkies_offers_received", + "selkies_candidates_received", + "pending_selkies_candidates", + ]) { + if (count(signaling[key])) picked[key] = signaling[key]; + } + } + if (typeof status?.ice_connection_state === "string") { + picked.ice_connection_state = status.ice_connection_state.slice(0, 32); + } + if (typeof status?.webrtc_connection_state === "string") { + picked.webrtc_connection_state = status.webrtc_connection_state.slice(0, 32); + } + return Object.keys(picked).length ? { engine_signaling: picked } : {}; +} const hash = value => `sha256:${createHash("sha256").update(String(value)).digest("hex").slice(0, 16)}`; const text = value => typeof value === "string" && value.length > 0 && value.length <= 512; const count = value => Number.isSafeInteger(value) && value >= 0; +function viewerFailureFields(viewer) { + return { + ...(typeof viewer?.status_text === "string" ? { viewer_status: viewer.status_text.slice(0, 240) } : {}), + ...(text(viewer?.failure_stage) ? { viewer_failure_stage: viewer.failure_stage.slice(0, 64) } : {}), + ...(text(viewer?.failure_message) ? { viewer_failure_message: viewer.failure_message.slice(0, 240) } : {}), + ...(count(viewer?.failure_status) && viewer.failure_status >= 400 && viewer.failure_status <= 599 + ? { viewer_failure_status: viewer.failure_status } : {}), + }; +} const documentId = value => Number.isFinite(value) && value > 0; class ReloadFailure extends Error {} function requireEvidence(ok, code) { if (!ok) throw new ReloadFailure(code); } @@ -33,7 +64,7 @@ function receipt(raw) { * Observer setup owns removal if it fails before returning its stop function. */ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, readState, - reloadViewer, extendInput, observeRequests, + reloadViewer, extendInput, observeRequests, observeReadState, observeMs = 0, clock = { now: () => performance.now(), setTimeout, clearTimeout }, }) { const started = clock.now(); @@ -42,6 +73,7 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re started_monotonic_ms: started }; let phase = "baseline", stop, failure, forbidden = false, original, initialViewer; let newDocument, newVideo, bytesRequired, initialReceipt, input, receiptSequence; + let reloadStarted; const interruptions = new Set(); const elapsed = () => Math.round(clock.now() - started); function binding(raw, source) { @@ -95,7 +127,7 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re (!bytesRequired || after.video_bytes_received > before.video_bytes_received); const matches = value => requireEvidence(value.every((item, i) => item === original[i]), "binding_changed"); async function pause(deadline) { - await within(deadline, phase === "baseline" ? "baseline_deadline" : "reload_deadline", ({ signal, timeoutMs }) => + await within(deadline, phase === "baseline" ? "baseline_deadline" : phase === "observe" ? "observe_deadline" : "reload_deadline", ({ signal, timeoutMs }) => new Promise(resolve => { const onAbort = () => { clock.clearTimeout(timer); resolve(); }; const timer = clock.setTimeout(() => { signal.removeEventListener("abort", onAbort); resolve(); }, @@ -130,7 +162,7 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re } evidence.samples.push({ at_ms: elapsed(), phase, binding_matches: true, viewer_ready: Boolean(media), viewer_has_page: Boolean(viewer?.page_id), - ...(typeof viewer?.status_text === "string" ? { viewer_status: viewer.status_text.slice(0, 240) } : {}), + ...viewerFailureFields(viewer), video_present: raw?.video?.present === true, ...(count(raw?.video?.ready_state) ? { video_ready_state: raw.video.ready_state } : {}), ...(count(raw?.video?.decoded_frames) ? { video_decoded_frames: raw.video.decoded_frames } : {}), @@ -179,7 +211,8 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re while (!advanced(current, initialVideo)); phase = "reload"; - const reloadStarted = clock.now(), deadline = reloadStarted + WINDOW_MS; + reloadStarted = clock.now(); + const deadline = reloadStarted + WINDOW_MS; evidence.reload_started_ms = elapsed(); await within(deadline, "reload_deadline", reloadViewer); while (true) { @@ -211,14 +244,55 @@ export async function diagnoseBrowserViewerReload({ expectedUrl, readReceipt, re } } catch (error) { failure = error instanceof ReloadFailure ? error.message : "reload_diagnostic_failed"; - } finally { - if (typeof stop === "function") { + } + if (observeMs > WINDOW_MS && reloadStarted) { + phase = "observe"; + evidence.observe_ms = observeMs; + const observeDeadline = reloadStarted + observeMs; + const readObserve = typeof observeReadState === "function" ? observeReadState : readState; + while (clock.now() < observeDeadline) { try { - await within(clock.now() + WINDOW_MS, "observer_stop_deadline", budget => stop(budget), true); - evidence.observer.stopped = true; - } catch { evidence.observer.failure = "observer_stop_failed"; failure ||= "observer_stop_failed"; } + const raw = await within(Math.min(clock.now() + 1000, observeDeadline), "observe_state_deadline", readObserve); + const viewer = raw?.viewer; + let media = null; + if (raw?.video) { + try { media = video(raw.video, false); } catch { media = null; } + } + evidence.samples.push({ + at_ms: elapsed(), + phase, + binding_matches: "unmeasured", + viewer_ready: Boolean(media), + viewer_has_page: Boolean(viewer?.page_id), + ...viewerFailureFields(viewer), + video_present: raw?.video?.present === true, + ...(count(raw?.video?.ready_state) ? { video_ready_state: raw.video.ready_state } : {}), + ...(count(raw?.video?.decoded_frames) ? { video_decoded_frames: raw.video.decoded_frames } : {}), + video_bytes_reported: count(raw?.video?.video_bytes_received), + ...(viewer ? { document_hash: hash(viewer.document_id) } : {}), + ...signalingCensus(raw?.page_status), + ...(media || {}), + }); + if (evidence.samples.length > 96) evidence.samples.shift(); + } catch { + evidence.dropped_observe_samples = (evidence.dropped_observe_samples || 0) + 1; + } + if (clock.now() >= observeDeadline) break; + try { await pause(observeDeadline); } catch { break; } + } + evidence.observe_last_ms = elapsed(); + const late = evidence.samples.find((row) => + row.phase === "observe" && row.visible === true && count(row.decoded_frames) && row.decoded_frames > 0); + if (late && Number.isFinite(evidence.reload_started_ms)) { + evidence.first_late_decoded_frame_ms = late.at_ms - evidence.reload_started_ms; } } + if (typeof stop === "function") { + try { + await within(clock.now() + WINDOW_MS, "observer_stop_deadline", budget => stop(budget), true); + evidence.observer.stopped = true; + } catch { evidence.observer.failure = "observer_stop_failed"; failure ||= "observer_stop_failed"; } + } if (forbidden) failure = "page_open_or_close_observed"; evidence.ok = !failure; if (failure) { @@ -248,6 +322,9 @@ export async function readBrowserViewerReloadDocument() { exit_id: document.querySelector("#browser-exit")?.value, document_id: performance.timeOrigin, status_text: (document.querySelector("#browser-status .browser-status-message")?.textContent || "").slice(0, 240), + failure_stage: (document.querySelector("#browser-status")?.dataset?.browserFailureStage || "").slice(0, 64), + failure_message: (document.querySelector("#browser-status")?.dataset?.browserFailureMessage || "").slice(0, 240), + failure_status: Number(document.querySelector("#browser-status")?.dataset?.browserFailureStatus || 0) || 0, }, video: element ? { present: true, hidden: element.hidden, paused: element.paused, ready_state: element.readyState, video_width: element.videoWidth, video_height: element.videoHeight, @@ -258,6 +335,316 @@ export async function readBrowserViewerReloadDocument() { } +const CENSUS_WATCH_MS = 5000; +const CENSUS_POLL_MS = 250; + +function recordCensusRequest(evidence, elapsed, event) { + if (event?.source_matches !== true) return; + const navigation = event.kind === "navigation" && event.phase === "commit"; + if (!navigation && (event.kind !== "signaling" || + !["request", "headers", "response", "failed"].includes(event.phase))) return; + if (evidence.requests.length === 64) { + evidence.dropped_requests++; + return; + } + evidence.requests.push({ + at_ms: elapsed(), + kind: event.kind, + event: event.phase, + ...browserViewerSignalMetadata(event), + ...(count(event.document_generation) ? { document_generation: event.document_generation } : {}), + ...(count(event.status) && event.status <= 599 ? { status: event.status } : {}), + }); +} + +function recordCensusDiagnostic(evidence, elapsed, raw) { + if (!raw || raw.schema !== "elastos.browser.media-diagnostic/v1") return; + if (evidence.diagnostics.length === 64) { + evidence.dropped_diagnostics++; + return; + } + evidence.diagnostics.push({ + at_ms: elapsed(), + event: raw.event, + ...(typeof raw.attach_from_url === "boolean" ? { attach_from_url: raw.attach_from_url } : {}), + ...(typeof raw.from_summary === "boolean" ? { from_summary: raw.from_summary } : {}), + ...(typeof raw.request_id === "string" && /^[a-f0-9]{32}$/.test(raw.request_id) + ? { request_id: raw.request_id } + : {}), + ...(typeof raw.candidate_type === "string" ? { candidate_type: raw.candidate_type } : {}), + ...(typeof raw.ice_connection_state === "string" ? { ice_connection_state: raw.ice_connection_state } : {}), + ...(typeof raw.ice_gathering_state === "string" ? { ice_gathering_state: raw.ice_gathering_state } : {}), + ...(count(raw.video_packets_received) ? { video_packets_received: raw.video_packets_received } : {}), + ...(count(raw.browser_candidate_count) ? { browser_candidate_count: raw.browser_candidate_count } : {}), + }); +} + +/** Attribute reload time. This is a measurement, not the five-second product gate. */ +export function classifyReloadCensus(evidence) { + const origin = Number.isFinite(evidence?.reload_started_ms) ? evidence.reload_started_ms : 0; + const rel = (ms) => Number.isFinite(ms) ? Math.round(ms - origin) : null; + const rows = evidence?.requests || []; + const diagnostics = evidence?.diagnostics || []; + const samples = evidence?.samples || []; + const request = (type, phase) => rows.find((row) => + row.kind === "signaling" && row.signal_type === type && row.event === phase); + const navigation = rows.find((row) => row.kind === "navigation" && row.event === "commit"); + const firstFrame = samples.find((row) => + (row.phase === "reload" || row.phase === "observe") && row.visible === true && count(row.decoded_frames) && row.decoded_frames > 0); + const generations = [...new Set(rows.filter((row) => count(row.document_generation)).map((row) => row.document_generation))]; + const attachRequests = rows.filter((row) => + row.kind === "signaling" && row.signal_type === "display_attach" && row.event === "request"); + const attachResponses = rows.filter((row) => + row.kind === "signaling" && row.signal_type === "display_attach" && row.event === "response"); + const attach = attachResponses[0]; + const reloadSamples = samples.filter((row) => row.phase === "reload" || row.phase === "observe"); + const documentHashes = [...new Set(reloadSamples.map((row) => row.document_hash).filter((value) => text(value)))]; + const firstReloadHash = documentHashes[0]; + const secondDocument = reloadSamples.find((row) => + text(row.document_hash) && firstReloadHash && row.document_hash !== firstReloadHash); + const viewerFailure = reloadSamples.find((row) => + text(row.viewer_failure_message) || + text(row.viewer_failure_stage) || + (typeof row.viewer_status === "string" && + /could not complete|session expired|failed to start|temporarily unavailable/i.test(row.viewer_status))); + return { + schema: "elastos.browser.journey-reload-census-classification/v1", + reload_started_ms: origin, + navigation_commit_ms: rel(navigation?.at_ms), + display_attach_request_ms: rel(attachRequests[0]?.at_ms), + display_attach_response_ms: rel(attachResponses[0]?.at_ms), + display_attach_request_count: attachRequests.length, + last_display_attach_request_ms: rel(attachRequests[attachRequests.length - 1]?.at_ms), + last_display_attach_response_ms: rel(attachResponses[attachResponses.length - 1]?.at_ms), + dying_page_attach_ms: rel(diagnostics.find((row) => row.event === "dying_page_attach")?.at_ms), + dying_page_attach_request_id: + diagnostics.find((row) => row.event === "dying_page_attach" && typeof row.request_id === "string") + ?.request_id ?? null, + restore_boot_ms: rel(diagnostics.find((row) => row.event === "restore_boot")?.at_ms), + restore_boot_attach_from_url: + diagnostics.find((row) => row.event === "restore_boot" && typeof row.attach_from_url === "boolean") + ?.attach_from_url ?? null, + restore_boot_request_id: + diagnostics.find((row) => row.event === "restore_boot" && typeof row.request_id === "string") + ?.request_id ?? null, + restore_boot_ready_ms: rel(diagnostics.find((row) => row.event === "restore_boot_ready")?.at_ms), + restore_boot_from_summary: + diagnostics.find((row) => row.event === "restore_boot_ready" && typeof row.from_summary === "boolean") + ?.from_summary ?? null, + first_answer_request_ms: rel(request("answer", "request")?.at_ms), + first_answer_response_ms: rel(request("answer", "response")?.at_ms), + first_local_candidate_created_ms: rel(diagnostics.find((row) => row.event === "viewer_browser_candidate")?.at_ms), + first_candidate_request_ms: rel(request("candidate", "request")?.at_ms), + first_candidate_response_ms: rel(request("candidate", "response")?.at_ms), + ice_checking_ms: rel(diagnostics.find((row) => row.ice_connection_state === "checking")?.at_ms), + ice_connected_ms: rel(diagnostics.find((row) => row.ice_connection_state === "connected")?.at_ms), + first_decoded_frame_ms: rel(firstFrame?.at_ms), + first_decoded_frame_after_deadline: Number.isFinite(rel(firstFrame?.at_ms)) && rel(firstFrame?.at_ms) > WINDOW_MS, + document_generations: generations, + observer_navigation_events: rows.filter((row) => row.kind === "navigation" && row.event === "commit").length, + document_hashes: documentHashes, + same_document: documentHashes.length <= 1, + second_document_ms: secondDocument ? rel(secondDocument.at_ms) : null, + viewer_failure_ms: viewerFailure ? rel(viewerFailure.at_ms) : null, + ...(text(viewerFailure?.viewer_failure_stage) ? { viewer_failure_stage: viewerFailure.viewer_failure_stage } : {}), + ...(text(viewerFailure?.viewer_failure_message) ? { viewer_failure_message: viewerFailure.viewer_failure_message } : {}), + ...(count(viewerFailure?.viewer_failure_status) ? { viewer_failure_status: viewerFailure.viewer_failure_status } : {}), + ...(typeof viewerFailure?.viewer_status === "string" ? { viewer_failure_status_text: viewerFailure.viewer_status.slice(0, 240) } : {}), + attach_initial_offer_ice: attach?.initial_offer_ice || null, + attach_audio_offer_ice: attach?.audio_offer_ice || null, + last_sample_ms: rel(samples.at(-1)?.at_ms), + last_sample: samples.at(-1) ? { + video_present: samples.at(-1).video_present === true, + video_decoded_frames: count(samples.at(-1).video_decoded_frames) + ? samples.at(-1).video_decoded_frames : "none", + ...(samples.at(-1).engine_signaling ? { engine_signaling: samples.at(-1).engine_signaling } : {}), + } : null, + }; +} + +/** + * Reload a working viewer and record attach, answer, candidate, ICE, and frame times. + * Reuses the journey observer. Does not type, scroll, or apply the five-second gate. + */ +export async function censusBrowserViewerReload({ + readState, reloadViewer, observeRequests, observeDiagnostics, + clock = { now: () => performance.now(), setTimeout, clearTimeout }, + watchMs = CENSUS_WATCH_MS, +}) { + const started = clock.now(); + const evidence = { + schema: "elastos.browser.journey-reload-census/v1", + ok: false, + requests: [], + samples: [], + diagnostics: [], + dropped_requests: 0, + dropped_diagnostics: 0, + observer: { stopped: false }, + started_monotonic_ms: started, + }; + let phase = "baseline", stopRequests, stopDiagnostics, failure; + const elapsed = () => Math.round(clock.now() - started); + async function within(deadlineMs, code, action) { + const timeoutMs = deadlineMs - clock.now(); + if (timeoutMs <= 0) throw new ReloadFailure(code); + const controller = new AbortController(); + let timer; + try { + const limit = new Promise((_, reject) => { + timer = clock.setTimeout(() => { + controller.abort(); + reject(new ReloadFailure(code)); + }, timeoutMs); + }); + const result = await Promise.race([ + limit, + Promise.resolve().then(() => action({ signal: controller.signal, deadlineMs, timeoutMs })), + ]); + if (clock.now() > deadlineMs) throw new ReloadFailure(code); + return result; + } catch (error) { + throw error instanceof ReloadFailure ? error : new ReloadFailure(code.replace(/_deadline$/, "_failed")); + } finally { + clock.clearTimeout(timer); + } + } + async function pause(deadline) { + await within(deadline, phase === "baseline" ? "baseline_deadline" : "census_deadline", ({ signal, timeoutMs }) => + new Promise((resolve) => { + const onAbort = () => { clock.clearTimeout(timer); resolve(); }; + const timer = clock.setTimeout(() => { + signal.removeEventListener("abort", onAbort); + resolve(); + }, Math.min(CENSUS_POLL_MS, timeoutMs)); + signal.addEventListener("abort", onAbort, { once: true }); + })); + } + async function sample(deadline) { + const raw = await within(deadline, "state_deadline", readState); + const viewer = raw?.viewer; + let media = null; + if (raw?.video) { + try { media = video(raw.video); } catch { media = null; } + } + evidence.samples.push({ + at_ms: elapsed(), + phase, + viewer_ready: Boolean(media?.visible), + viewer_has_page: Boolean(viewer?.page_id), + ...viewerFailureFields(viewer), + video_present: raw?.video?.present === true, + visible: media?.visible === true, + ...(count(raw?.video?.ready_state) ? { video_ready_state: raw.video.ready_state } : {}), + ...(count(raw?.video?.decoded_frames) ? { video_decoded_frames: raw.video.decoded_frames } : {}), + ...(count(media?.decoded_frames) ? { decoded_frames: media.decoded_frames } : {}), + ...(viewer?.document_id ? { document_hash: hash(viewer.document_id) } : {}), + ...signalingCensus(raw?.page_status), + }); + const sampleLimit = watchMs > CENSUS_WATCH_MS ? 96 : 32; + if (evidence.samples.length > sampleLimit) evidence.samples.shift(); + return media; + } + try { + const baselineDeadline = clock.now() + BASELINE_MS; + stopRequests = await within(baselineDeadline, "observer_deadline", + (budget) => observeRequests((event) => recordCensusRequest(evidence, elapsed, event), budget)); + if (typeof observeDiagnostics === "function") { + stopDiagnostics = await within(baselineDeadline, "observer_deadline", + (budget) => observeDiagnostics((raw) => recordCensusDiagnostic(evidence, elapsed, raw), budget)); + } + const raw = await within(baselineDeadline, "state_deadline", readState); + requireEvidence(raw?.viewer && documentId(raw.viewer.document_id), "baseline_viewer_missing"); + const initialVideo = video(raw.video); + requireEvidence(initialVideo.visible && initialVideo.decoded_frames > 0, "baseline_video_not_visible"); + let current; + do { + await pause(baselineDeadline); + current = await sample(baselineDeadline); + } while (!(current?.visible && current.decoded_frames > initialVideo.decoded_frames)); + + phase = "reload"; + const reloadStarted = clock.now(); + const deadline = reloadStarted + watchMs; + evidence.reload_started_ms = elapsed(); + await within(deadline, "reload_deadline", reloadViewer); + while (clock.now() < deadline) { + try { + await sample(deadline); + } catch (error) { + if (error instanceof ReloadFailure && error.message === "state_deadline") { + evidence.dropped_observe_samples = (evidence.dropped_observe_samples || 0) + 1; + } else { + throw error; + } + } + if (clock.now() >= deadline) break; + try { + await pause(deadline); + } catch (error) { + if (error instanceof ReloadFailure && error.message === "census_deadline") break; + throw error; + } + } + evidence.watch_ms = clock.now() - reloadStarted; + } catch (error) { + failure = error instanceof ReloadFailure ? error.message : "reload_census_failed"; + } finally { + if (typeof stopRequests === "function") { + try { + await within(clock.now() + WINDOW_MS, "observer_stop_deadline", (budget) => stopRequests(budget)); + evidence.observer.stopped = true; + } catch { + evidence.observer.failure = "observer_stop_failed"; + failure ||= "observer_stop_failed"; + } + } + if (typeof stopDiagnostics === "function") { + try { + await within(clock.now() + WINDOW_MS, "observer_stop_deadline", (budget) => stopDiagnostics(budget)); + } catch { + evidence.observer.diagnostics_failure = "observer_stop_failed"; + } + } + } + evidence.classification = classifyReloadCensus(evidence); + evidence.first_frame_in_window = Number.isFinite(evidence.classification.first_decoded_frame_ms); + evidence.ok = !failure; + if (failure) { + evidence.failure = failure; + evidence.failure_phase = phase; + const error = new Error(`Browser viewer reload census failed: ${failure}`); + error.evidence = evidence; + throw error; + } + return evidence; +} + + +function iceCensus(candidates) { + const list = Array.isArray(candidates) ? candidates : []; + return { + candidate_count: list.length, + relay_candidate_count: list.filter((candidate) => + /\btyp\s+relay\b/i.test(String(candidate?.candidate || "")), + ).length, + }; +} + +function passIceCensus(value) { + return value && + Number.isSafeInteger(value.candidate_count) && + value.candidate_count >= 0 && + Number.isSafeInteger(value.relay_candidate_count) && + value.relay_candidate_count >= 0 && + value.relay_candidate_count <= value.candidate_count + ? { + candidate_count: value.candidate_count, + relay_candidate_count: value.relay_candidate_count, + } + : null; +} + /** Keep signaling stage evidence bounded; SDP, candidates and authority remain private. */ export function browserViewerSignalMetadata(payload) { const types = ["display_attach", "offer", "answer", "candidate", "end_of_candidates"]; @@ -265,9 +652,25 @@ export function browserViewerSignalMetadata(payload) { "display_attach_unsupported", "display_attach_failed", "display_attach_uncertain"]; const type = payload?.signal_type ?? payload?.type; const code = payload?.error_code ?? payload?.code; + const attach = payload?.schema === "elastos.browser.display-attach-result/v1" || payload?.attached === true; + const initialOfferIce = passIceCensus(payload?.initial_offer_ice) || + (attach && Array.isArray(payload?.initial_offer?.candidates) + ? iceCensus(payload.initial_offer.candidates) + : null); + const audioOfferIce = passIceCensus(payload?.audio_offer_ice) || + (attach && Array.isArray(payload?.audio_offer?.candidates) + ? iceCensus(payload.audio_offer.candidates) + : null); + const ackIce = passIceCensus(payload?.ack_ice) || + (payload?.schema === "elastos.browser.webrtc-signal-ack/v1" && Array.isArray(payload?.candidates) + ? iceCensus(payload.candidates) + : null); return { ...(types.includes(type) ? { signal_type: type } : {}), ...(codes.includes(code) ? { error_code: code } : {}), - ...(payload?.schema === "elastos.browser.display-attach-result/v1" || payload?.attached === true ? { attached: true } : {}), + ...(attach ? { attached: true } : {}), + ...(initialOfferIce ? { initial_offer_ice: initialOfferIce } : {}), + ...(audioOfferIce ? { audio_offer_ice: audioOfferIce } : {}), + ...(ackIce ? { ack_ice: ackIce } : {}), }; } diff --git a/scripts/lib/browser-journey-viewer-reload.test.mjs b/scripts/lib/browser-journey-viewer-reload.test.mjs index 12aee332e..250dbd2f3 100644 --- a/scripts/lib/browser-journey-viewer-reload.test.mjs +++ b/scripts/lib/browser-journey-viewer-reload.test.mjs @@ -3,7 +3,7 @@ import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; import test from "node:test"; import vm from "node:vm"; -import { diagnoseBrowserViewerReload } from "./browser-journey-viewer-reload.mjs"; +import { censusBrowserViewerReload, classifyReloadCensus, diagnoseBrowserViewerReload } from "./browser-journey-viewer-reload.mjs"; const hash = value => `sha256:${createHash("sha256").update(value).digest("hex").slice(0, 16)}`; const secret = "private-token-turn://credential@private.invalid"; @@ -155,6 +155,18 @@ async function fails(options, expected, edit) { return { ...f, evidence }; } +test("late observe samples report unmeasured bindings instead of claiming a match", async () => { + const f = fixture(); + f.args.observeMs = 5500; + const proof = await f.run(); + assert.equal(proof.ok, true); + const observed = proof.samples.filter(sample => sample.phase === "observe"); + assert.ok(observed.length > 0); + assert.ok(observed.every(sample => sample.binding_matches === "unmeasured")); + assert.ok(proof.samples.filter(sample => sample.phase !== "observe") + .every(sample => sample.binding_matches === true)); +}); + test("a new viewer retains fresh Runtime binding, resets counters, decodes and appends exact input within five seconds", async () => { const f = fixture(); const proof = await f.run(); @@ -388,6 +400,114 @@ test("failed Runtime predicate is retained before any pending viewer check", asy assert.ok(evidence.binding_failure.at_ms >= evidence.reload_started_ms); }); +test("reload census records candidate creation, send, ICE, and first frame without the product gate", async () => { + const timer = virtualClock(); + let reloaded = false, frames = 40, record, recordDiagnostic; + const evidence = await timer.run(censusBrowserViewerReload({ + clock: timer.clock, + async observeRequests(callback) { + record = callback; + return async () => {}; + }, + async observeDiagnostics(callback) { + recordDiagnostic = callback; + return async () => {}; + }, + async readState() { + if (!reloaded) frames += 1; + return { + viewer: { + page_id: "page-current", engine_id: "selected-engine", exit_id: "selected-exit", + browser_instance: "instance-current", actual_url: fixtureUrl, document_id: reloaded ? 2 : 1, + }, + video: { + present: true, hidden: false, paused: false, ready_state: 4, + video_width: 640, video_height: 360, client_width: 640, client_height: 360, + decoded_frames: frames, video_bytes_received: 1000, + }, + }; + }, + async reloadViewer() { + reloaded = true; + frames = 0; + record({ kind: "navigation", phase: "commit", source_matches: true }); + record({ kind: "signaling", phase: "request", source_matches: true, signal_type: "display_attach" }); + record({ kind: "signaling", phase: "response", source_matches: true, signal_type: "display_attach", status: 200 }); + record({ kind: "signaling", phase: "request", source_matches: true, signal_type: "answer" }); + record({ kind: "signaling", phase: "response", source_matches: true, signal_type: "answer", status: 200 }); + recordDiagnostic({ schema: "elastos.browser.media-diagnostic/v1", event: "viewer_browser_candidate", + candidate_type: "relay" }); + record({ kind: "signaling", phase: "request", source_matches: true, signal_type: "candidate" }); + record({ kind: "signaling", phase: "response", source_matches: true, signal_type: "candidate", status: 200 }); + recordDiagnostic({ schema: "elastos.browser.media-diagnostic/v1", event: "viewer_ice_state", + ice_connection_state: "checking" }); + recordDiagnostic({ schema: "elastos.browser.media-diagnostic/v1", event: "viewer_ice_state", + ice_connection_state: "connected" }); + frames = 8; + }, + })); + const classified = classifyReloadCensus(evidence); + assert.equal(evidence.ok, true); + assert.equal(evidence.first_frame_in_window, true); + assert.ok(classified.display_attach_request_ms >= 0); + assert.ok(classified.first_answer_request_ms >= classified.display_attach_request_ms); + assert.ok(classified.first_local_candidate_created_ms >= classified.first_answer_request_ms); + assert.ok(classified.first_candidate_request_ms >= classified.first_local_candidate_created_ms); + assert.ok(classified.ice_connected_ms >= classified.ice_checking_ms); + assert.ok(classified.first_decoded_frame_ms >= classified.ice_connected_ms); +}); + +test("reload census keeps recording past five seconds and classifies a late first frame", async () => { + const timer = virtualClock(); + let reloaded = false, frames = 40, reloadAt = 0, record; + const evidence = await timer.run(censusBrowserViewerReload({ + clock: timer.clock, + watchMs: 20_000, + async observeRequests(callback) { + record = callback; + return async () => {}; + }, + async readState() { + const elapsed = reloaded ? timer.clock.now() - reloadAt : 0; + if (!reloaded) frames += 1; + else frames = elapsed >= 8_000 ? 4 : 0; + return { + viewer: { + page_id: "page-current", engine_id: "selected-engine", exit_id: "selected-exit", + browser_instance: "instance-current", actual_url: fixtureUrl, document_id: reloaded ? 2 : 1, + }, + page_status: { + webrtc_signaling: { + selkies_offers_received: reloaded ? 2 : 1, + browser_answers_received: reloaded && elapsed >= 1_000 ? 1 : 0, + }, + }, + video: { + present: true, hidden: false, paused: false, ready_state: reloaded && frames === 0 ? 0 : 4, + video_width: 640, video_height: 360, client_width: 640, client_height: 360, + decoded_frames: frames, video_bytes_received: 1000, + }, + }; + }, + async reloadViewer() { + reloaded = true; + reloadAt = timer.clock.now(); + frames = 0; + record({ kind: "navigation", phase: "commit", source_matches: true, document_generation: 1 }); + record({ kind: "signaling", phase: "response", source_matches: true, signal_type: "display_attach", + status: 200, initial_offer_ice: { candidate_count: 1, relay_candidate_count: 1 } }); + }, + })); + assert.equal(evidence.ok, true); + assert.equal(evidence.classification.first_decoded_frame_after_deadline, true); + assert.ok(evidence.classification.first_decoded_frame_ms >= 8_000); + assert.ok(evidence.classification.last_sample_ms >= 15_000); + assert.deepEqual(evidence.classification.attach_initial_offer_ice, + { candidate_count: 1, relay_candidate_count: 1 }); + assert.equal(evidence.classification.document_generations[0], 1); + assert.ok(evidence.samples.some((row) => row.engine_signaling?.selkies_offers_received === 2)); +}); + test("reload retains summary and attachment header timing without accepting bodies or changing its gate", async () => { const f = fixture(), reload = f.args.reloadViewer; f.args.reloadViewer = async budget => { @@ -401,3 +521,83 @@ test("reload retains summary and attachment header timing without accepting bodi assert.ok(evidence.requests.some(row => row.kind === "signaling" && row.event === "headers" && row.status === 200)); assert.ok(evidence.reload_ms <= 5000); assert.ok(!JSON.stringify(evidence).includes(secret)); }); + +test("reload census treats extra navigation events as observers when the document hash stays the same", () => { + const classified = classifyReloadCensus({ + reload_started_ms: 100, + requests: [ + { at_ms: 200, kind: "navigation", event: "commit", document_generation: 1 }, + { at_ms: 800, kind: "navigation", event: "commit", document_generation: 2 }, + ], + samples: [ + { at_ms: 150, phase: "reload", document_hash: "sha256:samehash000001", visible: false, decoded_frames: 0 }, + { at_ms: 900, phase: "reload", document_hash: "sha256:samehash000001", visible: true, decoded_frames: 4 }, + ], + diagnostics: [], + }); + assert.deepEqual(classified.document_generations, [1, 2]); + assert.equal(classified.observer_navigation_events, 2); + assert.deepEqual(classified.document_hashes, ["sha256:samehash000001"]); + assert.equal(classified.same_document, true); + assert.equal(classified.second_document_ms, null); +}); + +test("reload census records a second document only when the sampled document hash changes", () => { + const classified = classifyReloadCensus({ + reload_started_ms: 0, + requests: [{ at_ms: 10, kind: "navigation", event: "commit", document_generation: 1 }], + samples: [ + { at_ms: 20, phase: "reload", document_hash: "sha256:firstdocument01", visible: false }, + { at_ms: 80, phase: "reload", document_hash: "sha256:seconddocument1", visible: false }, + ], + diagnostics: [], + }); + assert.equal(classified.same_document, false); + assert.equal(classified.second_document_ms, 80); + assert.deepEqual(classified.document_hashes, ["sha256:firstdocument01", "sha256:seconddocument1"]); +}); + +test("reload census records dying-page and restore-boot request identity", () => { + const classified = classifyReloadCensus({ + reload_started_ms: 100, + requests: [], + samples: [], + diagnostics: [ + { at_ms: 101, event: "dying_page_attach", request_id: "a".repeat(32) }, + { at_ms: 180, event: "restore_boot", attach_from_url: true, request_id: "a".repeat(32) }, + { at_ms: 220, event: "restore_boot_ready", from_summary: true, request_id: "a".repeat(32) }, + ], + }); + assert.equal(classified.dying_page_attach_ms, 1); + assert.equal(classified.dying_page_attach_request_id, "a".repeat(32)); + assert.equal(classified.restore_boot_ms, 80); + assert.equal(classified.restore_boot_attach_from_url, true); + assert.equal(classified.restore_boot_request_id, "a".repeat(32)); + assert.equal(classified.restore_boot_ready_ms, 120); + assert.equal(classified.restore_boot_from_summary, true); +}); + +test("reload census captures viewer failure status and the original exception fields", () => { + const classified = classifyReloadCensus({ + reload_started_ms: 0, + requests: [ + { at_ms: 100, kind: "signaling", event: "response", signal_type: "display_attach", status: 200 }, + ], + samples: [ + { + at_ms: 366, + phase: "reload", + viewer_status: "Browser could not complete the request. Refresh Browser and try again.", + viewer_failure_stage: "peer_setup", + viewer_failure_message: "Browser display could not connect.", + viewer_failure_status: 400, + }, + ], + diagnostics: [], + }); + assert.equal(classified.viewer_failure_ms, 366); + assert.equal(classified.viewer_failure_stage, "peer_setup"); + assert.equal(classified.viewer_failure_message, "Browser display could not connect."); + assert.equal(classified.viewer_failure_status, 400); + assert.match(classified.viewer_failure_status_text, /could not complete/); +}); diff --git a/scripts/lib/browser-vm-webrtc-apply-gate.mjs b/scripts/lib/browser-vm-webrtc-apply-gate.mjs new file mode 100644 index 000000000..68de08f9b --- /dev/null +++ b/scripts/lib/browser-vm-webrtc-apply-gate.mjs @@ -0,0 +1,99 @@ +export function webrtcSignalKind(body) { + const type = body?.signal?.type; + if (type === "display_attach") return "display_attach"; + if (type === "answer") return "answer"; + if (type === "candidate" || type === "end_of_candidates") return "ice"; + return "other"; +} + +export function createGuestWebrtcApplyGate() { + const gates = new Map(); + + function gateKey(pageId, channel) { + return `${pageId}:${channel === "audio" ? "audio" : "video"}`; + } + + function gate(pageId, channel) { + const key = gateKey(pageId, channel); + let entry = gates.get(key); + if (!entry) { + entry = { answerApplied: false, expectedGeneration: "", pending: [] }; + gates.set(key, entry); + } + return entry; + } + + function rejectPending(entry, error) { + const pending = entry.pending.splice(0); + for (const item of pending) { + item.reject(error); + } + } + + function resetPage(pageId) { + for (const channel of ["video", "audio"]) { + const key = gateKey(pageId, channel); + const entry = gates.get(key); + if (entry) { + rejectPending( + entry, + new Error("Browser display attachment replaced pending ICE"), + ); + } + gates.delete(key); + } + } + + async function flushPending(entry, forward) { + const pending = entry.pending.splice(0); + await Promise.all( + pending.map((item) => + Promise.resolve() + .then(() => forward(item.body)) + .then(item.resolve, item.reject), + ), + ); + } + + async function apply(pageId, body, forward) { + const kind = webrtcSignalKind(body); + if (kind === "display_attach") { + resetPage(pageId); + const result = await forward(body); + const generation = result?.display_generation; + if (typeof generation === "string" && generation) { + gate(pageId, "video").expectedGeneration = generation; + gate(pageId, "audio").expectedGeneration = generation; + } + return result; + } + const entry = gate(pageId, body?.channel); + const generation = body?.signal?.display_generation; + if ( + entry.expectedGeneration && + typeof generation === "string" && + generation !== entry.expectedGeneration + ) { + return forward(body); + } + if (kind === "answer") { + try { + const result = await forward(body); + entry.answerApplied = true; + await flushPending(entry, forward); + return result; + } catch (error) { + rejectPending(entry, error); + throw error; + } + } + if (kind === "ice" && !entry.answerApplied) { + return new Promise((resolve, reject) => { + entry.pending.push({ body, resolve, reject }); + }); + } + return forward(body); + } + + return { apply, resetPage }; +} diff --git a/scripts/lib/browser-vm-webrtc-apply-gate.test.mjs b/scripts/lib/browser-vm-webrtc-apply-gate.test.mjs new file mode 100644 index 000000000..b3e5e0689 --- /dev/null +++ b/scripts/lib/browser-vm-webrtc-apply-gate.test.mjs @@ -0,0 +1,116 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + createGuestWebrtcApplyGate, + webrtcSignalKind, +} from "./browser-vm-webrtc-apply-gate.mjs"; + +test("classifies attach, answer, and ICE signals", () => { + assert.equal( + webrtcSignalKind({ signal: { type: "display_attach" } }), + "display_attach", + ); + assert.equal(webrtcSignalKind({ signal: { type: "answer" } }), "answer"); + assert.equal(webrtcSignalKind({ signal: { type: "candidate" } }), "ice"); + assert.equal( + webrtcSignalKind({ signal: { type: "end_of_candidates" } }), + "ice", + ); + assert.equal(webrtcSignalKind({ signal: { type: "offer" } }), "other"); +}); + +test("holds ICE until the same-channel answer is applied", async () => { + const gate = createGuestWebrtcApplyGate(); + const order = []; + const forward = async (body) => { + order.push(body.signal.type); + return { type: body.signal.type }; + }; + const ice = gate.apply( + "page:one", + { channel: "video", signal: { type: "candidate" } }, + forward, + ); + const answer = gate.apply( + "page:one", + { channel: "video", signal: { type: "answer" } }, + forward, + ); + assert.deepEqual(order, ["answer"]); + await answer; + await ice; + assert.deepEqual(order, ["answer", "candidate"]); +}); + +test("keeps audio ICE independent of the video answer", async () => { + const gate = createGuestWebrtcApplyGate(); + const order = []; + const forward = async (body) => { + order.push(`${body.channel}:${body.signal.type}`); + return { type: body.signal.type }; + }; + const audioIce = gate.apply( + "page:one", + { channel: "audio", signal: { type: "candidate" } }, + forward, + ); + await gate.apply( + "page:one", + { channel: "video", signal: { type: "answer" } }, + forward, + ); + assert.equal(order.join(","), "video:answer"); + await gate.apply( + "page:one", + { channel: "audio", signal: { type: "answer" } }, + forward, + ); + await audioIce; + assert.deepEqual(order, [ + "video:answer", + "audio:answer", + "audio:candidate", + ]); +}); + +test("forwards ICE from a previous display generation", async () => { + const gate = createGuestWebrtcApplyGate(); + const order = []; + const forward = async (body) => { + order.push(body.signal.type); + if (body.signal.type === "display_attach") { + return { display_generation: "display:new" }; + } + return { type: body.signal.type }; + }; + await gate.apply( + "page:one", + { signal: { type: "display_attach" } }, + forward, + ); + await gate.apply( + "page:one", + { + channel: "video", + signal: { type: "candidate", display_generation: "display:old" }, + }, + forward, + ); + assert.deepEqual(order, ["display_attach", "candidate"]); +}); + +test("drops held ICE when display_attach starts a new pair", async () => { + const gate = createGuestWebrtcApplyGate(); + const ice = gate.apply( + "page:one", + { channel: "video", signal: { type: "candidate" } }, + async () => ({ ok: true }), + ); + await gate.apply( + "page:one", + { signal: { type: "display_attach" } }, + async () => ({ attached: true }), + ); + await assert.rejects(ice, /replaced pending ICE/); +}); From 7690aabc6acb4f125b84b7ec0117e3e406898b91 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 20:18:34 -0500 Subject: [PATCH 339/535] docs: record D1 model evidence and keep Browser acceptance open TASKS and state now describe the Sash preview candidate while the paused Browser closeout remains a release obligation. --- TASKS.md | 158 +++++- scripts/home-entropy-check.mjs | 106 +++- scripts/home-passkey-virtual-auth-smoke.mjs | 572 +++++++++++++++++--- scripts/home-shell-bridge-smoke.mjs | 36 +- scripts/home-shell-regression-smoke.mjs | 18 + state.md | 109 ++-- 6 files changed, 863 insertions(+), 136 deletions(-) diff --git a/TASKS.md b/TASKS.md index 9f875cf96..dc36669a4 100644 --- a/TASKS.md +++ b/TASKS.md @@ -171,6 +171,133 @@ verification separate from product acceptance and required human review. ## Now +### Model delivery R14 — bounded Sash preview first + +The immediate D1 outcome is a normal Apple Silicon model demo: install the named +candidate, discover both signed model entries, Get/Use SmolLM2, send in Assistant, +see the reply, then reload/restart with saved conversation and payload reuse. +Reuse existing Qwen evidence within its recorded scope. The full R14 requirements +remain below; they follow this bounded preview instead of delaying its first test. + +1. Local owns actual Assistant Send/reply/reload/restart on the admitted model, + durable test-holder ownership and installed managed-Home catalogue sync. + First visible result or exact failure is due within ten minutes of resuming. + Preserve the human Qwen Home and existing admissions. +2. Cloud's catalogue-before-installation repair is integrated. The exact two-file + patch passed all 53 installer tests locally (four retained fixture skips). + Cloud now owns only the publisher shell, native-platform admission helper and + their focused tests for an explicit Apple Silicon canary preview. Default + release admission retains all three platforms and all provenance/trust gates. + Local freezes those packaging files until the reviewed patch returns. +3. Coordinator reviews the exact combined source and artifacts, prepares coherent + commits and a short test guide. Publication for review uses the existing + branch and preserves Browser ancestry. Show the candidate and verification + before named approval to push or expose the preview. Public Home is preserved. + +The broader R14 queue remains required after D1: independent full Mac/seed holder +reads and demonstrated transfer repairs; responsive preparation cancellation and +one total attempt budget; reconstructible cloud input and full Linux consumer +proof; seed-to-Mac Qwen inference; Qwen peer acquisition/pinning without activation. +Reuse unchanged receipts. Browser, J2 updates, three-platform W1 and all J1–J5, +M01–M06 and storage/capsule criteria retain their existing acceptance gates. + +The private handoff records the two-hour deadline and target ownership. A failed +demo checkpoint triggers a specific repair or a clear blocker, not a broad audit. + +### Preserved Browser closeout (paused during R14) + +Notion owns the approved scope and acceptance. This is the executable order; +older mission revisions below are history. Preserve B01-B16, the two accepted +Browser placements, applicable SA1-SA6/CA1, and all existing numerical and human +gates. Current source, target ownership and receipt identities belong in the +private development-loop checkpoint. Existing ordinary persistence, media, +recovery and scroll receipts retain their exact accepted scope. + +1. **Reconcile and consolidate.** The Mac leftover file for generation + `f9cd2b88` is gone after leftover-adapter install `10bf5c6c`. Seed leftover 1 + remains and is the next separate cleanup goal. The isolated Mac Runtime is + `8849b903` with the B13 Wallet slice and leftover adapter installed. + Offer `remote-engine-bc829519` stays approved and selectable as + `remote-engine-4cec4db7`. Shared image `76967c5b` stays unchanged. The + supervisor overlay is now an explicit `ELASTOS_BROWSER_WEBRTC_SEND_OVERLAY=1` + opt-in bound to input SHA-256 hashes. Installed diagnostic overlay files are + removed so ordinary launches stay on the admitted capsule and image. Keep + the host watcher retired. Keep helper leftover pages `vz-6a0bd4a3`, + `vz-2d1165cc`, `vz-a35a6fc4`, and `vz-245d4f18`. The five-second audio gate + stays open. After seed leftover clearance, run one hash-bound supervisor + overlay capture with a live selected grant checked immediately before launch. + Local review commit `6aa7c5ae` holds the ICE-pair probe. W1 canary + `release.json` is signed and still lacks a catalog and components endpoint. + The reviewed Cloud W1 sequence is local on `feat/remote-services` as + `2fef812c`, `aac9f263`, and `4009015b`. HEAD is `4009015b` / tree + `907bbf1f`, 169 ahead of last-fetched `origin/feat/0.7.1-integration`. + Publication and installer tests pass locally. Cloud receives no local Homes, + keys, VM images, or unpublished Browser work. Preserve unresolved profiles + and unrelated dirty work. Attempt chronology lives in the existing + `u9-r8-*` receipts. +2. **Finish Browser behavior.** After the current media stall owner is named or + repaired, finish remaining reload, P2 lifecycle, Browser profile + protection/checkpoint/transfer, authority, selected Engine/Exit, ordinary + files, and applicable daily-use/accessibility cases. B13 Wallet work is a + bounded owned slice: pin remaining source gaps with narrow regressions, then + repair consumer mediation and supported-account consent. The completed + signature status path now binds page origin and launch on the local + regression. Local `eth_requestAccounts` honors the selected default + connector when a managed account is also present, and it keeps origin + consent. Remote Engine launch now carries + `elastos.browser.wallet-consumer-mediation/v1` bound to the authenticated + consumer peer, page and generation. Production forwarding admits a request + built from that launch payload against the Engine peer plus page and + generation. Control-service keeps Wallet Bus on the consumer and refuses + Engine `home_token` fetch for that schema. The remaining gap is Carrier + transport of the admitted request. Keep Wallet Bus private and keep + approval on the consumer. Reuse the existing transaction journal. + Capture the actual ela.city sign-in, mint/buy and playback signing requests + before compatibility code. Installed local and remote connect/sign-in, + approve/reject/expiry, wrong principal/origin/page, and response-loss proof + stay required. Irzhy owns J5 metadata wiring and safe rejected-mint recovery + at published head `0edd56d5`; that work stays outside this Browser slice. + Use the shared state contract in + docs/STORAGE_AND_ACCESS.md and capsule composition in docs/CAPSULE_MODEL.md. + Browser implements its adapter and only the missing shared mechanisms it + needs. Builder/Home owns GBA and AI adapters after the contract/evidence + handoff; those remain required release work outside this Browser task. + Recheck the complete ordinary local and remote journeys after the affected + repairs pass. +3. **Prove normal delivery.** Use the common signed capsule/catalog, Content and + Runtime admission path. Include all Mac Engine helpers, dependencies and image; + preserve Browser as the entry capsule and separate package identity from + service offers. W1 supplies an isolated trusted signed candidate before tests. + This lane holds the reviewed Cloud W1 sequence in three local commits + `2fef812c`, `aac9f263`, and `4009015b`. Push waits for an explicit ask. + Prove acquisition from each promised availability source and safe reuse, + interruption, tamper and compatibility handling. Run fresh Apple-silicon + installation with empty Runtime data and no borrowed source-home files through + Home, Browser media/input, reload, close and reopen. Run fresh Linux Home using + an approved Mac Engine through Services without a local Engine image. Finish + B15/J2 update and repair with protected profile preservation. Build only changed + components; bind every installed result to exact artifacts/configuration. +4. **Qualify and hand over.** Freeze the compatible Browser UI, Runtime, Engine, + image and policy set before the original B11/B16 distributions and endurance + campaign. Keep manual UX, second-maintainer proof and browser-objective-audit + required. Changes reopen only dependent evidence and endurance runs. Prepare + the exact reviewed GitHub/seed candidate, readable results and remaining + non-Browser release dependencies. Local preparation can precede publication + approval; push, public deployment and final C5/C6/C7 release retain their + separate gates. Browser completion requires every applicable Required case, + not only the last successful experiment. + +Cursor is the single implementation and test-lane owner. The current user has +asked to resume and complete Browser. The Cloud W1 task is complete. This lane +holds the reviewed four-file intake in local commits through `4009015b`. Continue between evidence checkpoints; +30 minutes is a progress/replanning boundary, not an automatic stop. Each update +names accepted behavior or a narrowed cause, the next proof and any missing +input. A repeated identical failure needs a changed experiment before another +long run. Review coherent source and installed milestones; reuse unaffected +checks and receipts. When human input or a target is unavailable, state the exact +blocker and continue independent Browser work. Maintain the 10 percent disk floor +and preserve user Homes, keys, drafts, profiles, donor work and public services. + Mission revision R3, 2026-09-12 (reconciled 2026-09-13 UTC). The [approved five-journey plan](https://app.notion.com/p/wauio/ElastOS-0-7-1-release-plan-five-user-journeys-from-start-to-stop-3d6b682adcca81948f78d12abcd677b9) owns J1-J5 acceptance and D1-D6 decisions, including the remote inference @@ -184,17 +311,19 @@ September 11 artifacts. The [team report](docs/audits/2026-09-11-team-sync.md) and [contributor review](docs/audits/2026-09-11-contributor-review.md) retain their 5-11 September scope; current progress is recorded in state. -Use one execution owner for both installed test Homes. The existing Cursor -Model and Browser tasks are paused references until one receives the next bounded -slice. A reviewer checks the result at the evidence boundary. Recurring monitoring -stays paused. Prepare the clean candidate for draft review; publication and public -cutover retain their separate approvals. +Use one execution owner for both installed test Homes. Resume the existing +Browser owner for the next bounded slice; keep Model target use behind an explicit +handoff. A reviewer checks the result at the evidence boundary. The existing hourly +monitor reuses recent reviews and inspects new evidence. Prepare the candidate +locally for review; publication and public cutover retain separate approvals. -Next Browser slice: close the live 110 page through Home, then prove actual -viewer reload within five seconds, input/media and all 13 close effects on -Engine `remote-engine-ec54fd5eab67a5c5913513a47a797fc8` with seed Exit -`source-home-browser-exit`. Ordinary Automatic open fails at profile placement -while that grant is valid. Select the owning Mac Engine through Settings. +Historical Browser slice, 12 September 2026: close the then-live 110 page +through Home, then prove actual viewer reload within five seconds, +input/media and all 13 close effects on Engine +`remote-engine-ec54fd5eab67a5c5913513a47a797fc8` with seed Exit +`source-home-browser-exit`. That grant and page are no longer the current +U9 target. Ordinary Automatic open still fails at profile placement when a +valid grant exists. Select the owning Mac Engine through Settings. The independent Qwen slice is one immediate Busy replay and retry after capacity frees. Review the prepared helper only when that slice starts. Preserve accepted @@ -218,7 +347,7 @@ The C1-C7 sequence below retains the full release gate. | C3R / U8 | Approved seed user runs the Mac model | Partial installed acceptance / next execution owner | Accepted results are in state. Next: immediate Busy replay and successful retry, restart during a run without another dispatch, and denial of an unapproved second principal. AI1 adds exact package/offer/terms identity and Marketplace handoff before final M01-M06 acceptance. | | J4 / C4 / U9 | Browser works in the two agreed placements | Functional proof open / next execution owner | Ordinary A/B/A open 110 completed after Engine grant renew and Mac Engine selection. Close that live page, then prove actual reload within five seconds, audio/video/input and all 13 close effects. Then finish authority, recovery, state and daily-use checks; B11 performance and B16 qualification retain their gates. | | U10 | Qwen and Browser work together | Waiting for U8 and U9 / execution owner | One seed principal uses both on the same Mac; each keeps its own acceptance criteria. Include AI1 when its UX/identity changes enter the candidate. | -| J5 | Protect, list, buy, play and close controlled video | Planned / protected-content owner | PR60 original history is integrated in the published candidate; coordinate its open PR disposition with Irzhy. The focused video preparation repair adopted during C1 comes from the foundation inherited by PR62, rather than its four follow-up commits. Review PR59 authority cutover against final callers, then take PR62 work by readiness. J5 installed/human acceptance remains Required; extensions enter by readiness before freeze. | +| J5 | Protect, list, buy, play and close controlled video | Implementation open / protected-content owner | Irzhy published head `0edd56d5` on `feat/protected-content-0.7.1-followup` adds connector mint, one wallet effect, terminal approval failures and completed-mint replay. Metadata URI/royalty/Creator wiring and safe rejected-mint recovery after `EffectRaised` remain his J5 work. Browser now has a local completed-signature origin/launch bind regression, local `eth_requestAccounts` honors the selected connector when a managed account is also present, and remote Engine launch mediation plus production admit bind authenticated peer, page and generation. The remaining B13 Wallet gap is Carrier transport of that admitted request. J5 stays one Runtime, two principals, Brave and Base mainnet; funded runs still need the agreed spend limit. Installed/human acceptance remains Required. | | C5 | Assemble and freeze the reviewed candidate | Planned / coordinator | Reviewed required source; exact optional task list. Optional work cannot delay freeze. | | C6 | Accept the combined signed candidate | Planned / independent reviewer and human testers | All required journeys, shared Home/app regression, Browser qualification and first-hop update proof on matching artifacts. | | C7 | Publish and verify the accepted release | Planned / release owner | C6 and explicit approvals, accepted main tree/artifact identity, then public delivery checks. | @@ -244,8 +373,11 @@ Chat and repeated System Save. One combined record serves J1 and J3. The [journey register](docs/audits/ElastOS-Home-Journey-Audit.xlsx) retains the underlying findings and their proof requirements. -Existing two-Runtime, independent-operator, funded Base and wider Browser goals -keep their Later gates as set by D2/D4 and the approved plan. Preserve the +Existing protected-content two-Runtime, independent-operator and wider Browser +goals keep their Later gates as set by D2/D4 and the approved plan. The 15 September +J5 scope decision uses Base mainnet for bounded acceptance, with dedicated test +wallets/assets and an agreed spending limit before funded runs. Controlled failure +and replay tests remain local; Anvil repair is outside the critical path. Preserve the protected-content Runtime authority cutover, DocumentsSaveRequest/if_revision, bridge lifecycle, completed-mint adoption and cleanup obligations during integration. Required failures stay open until the relevant proof passes. diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index 0a1c3fa61..f35485529 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -732,6 +732,7 @@ const activeUiFiles = [ "capsules/wallet/browser/style.css", "capsules/browser/browser/index.html", "capsules/browser/browser/browser.js", + "capsules/browser/browser/browser-restore-boot.js", "capsules/browser/browser/browser-clipboard.js", "capsules/browser/browser/browser-history.js", "capsules/browser/browser/browser-input.js", @@ -888,6 +889,10 @@ assert( assistantModes.includes('from "./assistant.js"') && assistantModes.includes("studioOnly: true"), "Assistant boots the Sash shell and uses the typed controller for Studio", ); +assert( + assistantController.includes('from "./model-contract.js"'), + "Assistant Chat and Studio decode events through the typed contract module", +); assert( !/\blocalStorage\b|\bsessionStorage\b|\bindexedDB\b/.test(assistantScript), "Assistant must not add browser-owned persistence", @@ -2619,6 +2624,13 @@ assert( shellJs.includes('library: new Set(["archive-manager", "browser", "chat-room"])'), "Home must allow Library picker results to return only to Archive, Browser and Chat Room", ); +assert( + shellJs.includes('assistant: new Set(["marketplace"])') && + shellJs.includes('"home-agent": new Set(["marketplace"])') && + shellJs.includes("assistantModelsMarketplaceQuery") && + shellJs.includes("marketplaceAssistantHandoffQuery"), + "Open Models must hand off through Marketplace with a bounded Assistant CID query", +); assert( shellJs.includes('marketplace: "runtime-target"') && shellJs.includes('if (policy === "runtime-target")') && @@ -4774,6 +4786,42 @@ const publishRustRequired = rustConstItems( ); const homeProfile = new Set(components.profiles.home.components); const demoProfile = new Set(components.profiles.demo.components); +assert( + homeProfile.has("model-provider") && + homeProfile.has("llama-server") && + demoProfile.has("model-provider") && + demoProfile.has("llama-server"), + "Home and demo profiles must install model-provider and llama-server", +); +assert( + ![...homeProfile].some((component) => component.startsWith("model-qwen")), + "Home profile must not install huggingface GGUF components", +); +assert( + typeof components.model_catalog?.head_cid === "string" && + components.model_catalog.head_cid.startsWith("bafkrei") && + Array.isArray(components.model_catalog.publisher_dids) && + components.model_catalog.publisher_dids.length === 1 && + components.model_catalog.local_use?.max_cache_bytes >= 18516684377 && + components.model_catalog.local_use?.max_model_memory_bytes >= 8589934592, + "components.json must pin the signed permanent model catalog with Qwen local-use floors", +); +const modelCatalog = JSON.parse(read("model-catalog.json")); +assert( + modelCatalog.signer_did === components.model_catalog.publisher_dids[0] && + modelCatalog.payload?.schema === "elastos.model.catalog/v1" && + !Object.hasOwn(modelCatalog.payload, "expires_at") && + modelCatalog.payload.entries?.[0]?.cid === + "bafybeid5l7gfgsqy2wozia2q7mtyux2wrbnlfehzz4at3ic3cngvyku6hi", + "repo model-catalog.json must be the permanent Qwen publisher snapshot", +); +assert( + publishReleaseScript.includes("model_catalog") && + publishReleaseScript.includes("model-catalog.json") && + setupSourceHome.includes("install_signed_model_catalog") && + setupSourceHome.includes("install_local_model_engine"), + "publish and source-home setup must carry the signed catalog and llama-server", +); const protectedRuntimeProviders = new Set([ "protected-content-protect-provider", "media-provider", @@ -4880,10 +4928,11 @@ for (const component of [ "browser-engine-supervisor", "browser-native-proxy-engine", "browser-stream-bridge", - "browser-local-exit", - "object-provider", - "wallet-provider", -]) { + "browser-local-exit", + "object-provider", + "wallet-provider", + "model-provider", + ]) { assert(homeProfile.has(component), `Home profile must install ${component}`); assert( components.external[component], @@ -5034,6 +5083,19 @@ assert( !marketplaceUi.includes("isBuiltIn"), "Marketplace must trust the installed-active catalog instead of inferring product state from source or bundle labels", ); +assert( + marketplaceUi.includes('elastos.marketplace.navigate/v1') && + marketplaceUi.includes("function applyModelsLaunch") && + marketplaceUi.includes('target: "assistant"') && + marketplaceUi.includes("onReadyOpen: openReadyModelInAssistant") && + read("capsules/home-gui/browser/shell-windows.js").includes( + 'type: "elastos.marketplace.navigate/v1"', + ) && + read("capsules/marketplace/browser/model-management.js").includes( + '"Open in Assistant"', + ), + "Marketplace must accept Models navigation and open a ready CID in Assistant", +); assert( marketplaceUi.includes("const category = String(capsule.category || \"\").toLowerCase()") && marketplaceUi.includes("function acceptedContentLabels(") && @@ -5268,6 +5330,7 @@ const walletStyle = read("capsules/wallet/browser/style.css"); const browserManifest = read("capsules/browser/capsule.json"); const browser = read("capsules/browser/browser/index.html"); const browserMain = read("capsules/browser/browser/browser.js"); +const browserRestoreBoot = read("capsules/browser/browser/browser-restore-boot.js"); const browserJs = readAll([ "capsules/browser/browser/browser.js", "capsules/browser/browser/browser-clipboard.js", @@ -5289,13 +5352,18 @@ const browserUnloadReleaseBlock = sourceBlock( ); assert( browserJs.includes('window.addEventListener("beforeunload"') && - browserJs.includes('window.addEventListener("pagehide", releaseRuntimePageForUnload)') && + browserJs.includes('window.addEventListener("pagehide"') && + browserJs.includes("startRecoverableDisplayAttach();") && browserUnloadReleaseBlock.includes("stopPageStatusPolling();") && browserUnloadReleaseBlock.includes("stopPageHeartbeat();") && !browserUnloadReleaseBlock.includes("resizeObserver") && !browserUnloadReleaseBlock.includes("closeRuntimePage(") && !browserUnloadReleaseBlock.includes("publishRuntimePageForHost(null)") && !browserUnloadReleaseBlock.includes("closeRemoteDisplay()") && + !browserUnloadReleaseBlock.includes("prepareRecoverableDisplayAttach") && + !browserUnloadReleaseBlock.includes("startRecoverableDisplayAttach") && + !browserUnloadReleaseBlock.includes("display_attach") && + !browserUnloadReleaseBlock.includes("keepalive") && browserJs.includes("createRuntimePageCleanupController") && browserJs.includes("requireTerminalRuntimePageCloseOutcome") && !browserJs.includes("__elastosBrowserReleaseRuntimePage") && @@ -9872,6 +9940,8 @@ assert( ) && browserSelkiesControlService.includes("HELLO client") && browserSelkiesControlService.includes("SESSION server") && + browserSelkiesControlService.includes("elastos_display_renegotiate") && + browserSelkiesControlService.includes("DISPLAY_RENEGOTIATE_SIGNAL") && browserSelkiesControlServiceSmoke.includes("HELLO 1") && browserSelkiesControlServiceSmoke.includes("base64") && browserSelkiesControlService.includes('offerer: "engine"') && @@ -10310,6 +10380,13 @@ assert( "ice_servers: this.config.iceServers", ) && browserSelkiesControlServiceSmoke.includes("status lost Runtime media relay proof") && + browserSelkiesControlService.includes("waitForRenegotiatedOffer") && + browserSelkiesControlService.includes("settleRenegotiatedOffer") && + browserSelkiesControlService.includes("Selkies relay ICE for the new offer") && + browserSelkiesControlService.includes("relayIceCandidates") && + browserSelkiesControlService.includes("rememberIceCandidate") && + browserSelkiesControlService.includes("this.remoteCandidateHistory = [];") && + browserSelkiesControlService.includes("relayIceCandidates(") && browserSelkiesControlService.includes("function mediaKindsForSdp") && !browserSelkiesControlService.includes("isSelkiesAudioUnavailable") && !browserSelkiesControlService.includes("audio_offer_unavailable") && @@ -10730,9 +10807,28 @@ assert( shellWindows.includes('const launchQuery = targetId === "browser"') && shellWindows.includes("withBrowserInstanceQuery({ query: options.query }).query") && shellWindows.includes("query: restoredWindow.query") && + shellWindows.includes("if (restoredWindow.hidden)") && browserJs.includes("const stalePage = previousPage ? null : recoverableRuntimePage();") && browserJs.includes("await closeRuntimePage(stalePage, {") && browserJs.includes("elastos.browser.cleanup-handle/v1") && + browser.includes("browser-restore-boot.js?v=browser-20260915c") && + browserRestoreBoot.includes("function waitForReadyAttach") && + browserRestoreBoot.includes("function attachFromKnownUrl") && + browser.includes('rel="modulepreload"') && + browserRestoreBoot.includes("__elastosBrowserRestoreBoot") && + browserRestoreBoot.includes("display_generation") && + browserRestoreBoot.includes('generation.slice("display:".length)') && + browserJs.includes("__elastosBrowserRestoreBoot") && + browserJs.includes("persistRecoverableDisplayQuery") && + browserJs.includes("function startRecoverableDisplayAttach") && + browserJs.includes("keepalive: true") && + browserJs.includes('pending.state === "pending" || pending.state === "ready"') && + !browserJs.includes("prepareRecoverableDisplayAttach") && + browserJs.includes("const connecting = connectRemoteDisplay") && + browserJs.includes("postAnswerWithQueuedLocalIce") && + !read("capsules/browser/browser/browser-remote-display.js").includes("waitForLocalAnswerIce") && + browserJs.includes('url.searchParams.set("page_id"') && + !browserRestoreBoot.includes("sessionStorage") && !browserJs.includes("sessionStorage") && !browserJs.includes("__elastosBrowserReleaseRuntimePage"), "Home must persist Browser window launch query/browser_instance across restore while Browser recovers opaque Runtime cleanup ownership without frame-local session storage", diff --git a/scripts/home-passkey-virtual-auth-smoke.mjs b/scripts/home-passkey-virtual-auth-smoke.mjs index 38ae1746e..ac1b96e5e 100644 --- a/scripts/home-passkey-virtual-auth-smoke.mjs +++ b/scripts/home-passkey-virtual-auth-smoke.mjs @@ -19,7 +19,9 @@ import { browserJourneyTargetConfig, readBrowserJourneyHealth, readBrowserJourne browserJourneyProfileStorage, browserJourneyProfileBinding } from "./lib/browser-journey-target.mjs"; import { installBrowserJourneyAudioProbe, controlledTonePresent } from "./lib/browser-journey-audio.mjs"; import { diagnoseBrowserJourneyRecovery } from "./lib/browser-journey-recovery.mjs"; -import { diagnoseBrowserViewerReload, readBrowserViewerReloadDocument, browserViewerSignalMetadata } from "./lib/browser-journey-viewer-reload.mjs"; +import { readBrowserPageStatus, isPageStatusTimeout } from "./lib/browser-journey-page-status.mjs"; +import { recoveredInputClickRect } from "./lib/browser-journey-recovered-input.mjs"; +import { censusBrowserViewerReload, diagnoseBrowserViewerReload, readBrowserViewerReloadDocument, browserViewerSignalMetadata } from "./lib/browser-journey-viewer-reload.mjs"; import { runBrowserOperatorJourney } from "./lib/browser-journey-operator.mjs"; import { qualificationOptions, createQualificationHarness, createQualificationCancellation, qualificationInteraction } from "./lib/browser-qualification-observer.mjs"; @@ -101,6 +103,12 @@ const BROWSER_INPUT_MAX_MS = parseBoundedIntegerEnv( 1, 60_000, ); +const BROWSER_AUDIO_WINDOW_MS = parseBoundedIntegerEnv( + "HOME_VIRTUAL_AUTH_BROWSER_AUDIO_WINDOW_MS", + 2500, + 500, + 10_000, +); const BROWSER_INPUT_EXPECT_URL_RE = process.env.HOME_VIRTUAL_AUTH_BROWSER_INPUT_EXPECT_URL_RE || ""; const BROWSER_OPEN_VIEWPORT_WIDTH = parseBoundedIntegerEnv( "HOME_VIRTUAL_AUTH_BROWSER_OPEN_VIEWPORT_WIDTH", @@ -157,6 +165,8 @@ const CHECK_BROWSER_EMBEDDED_UI_INPUT = const CHECK_BROWSER_CONTROLLED_JOURNEY = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_JOURNEY === "1"; const CHECK_BROWSER_CONTROLLED_MEDIA = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_MEDIA === "1"; +const CHECK_BROWSER_AUDIO_ENGINE_DIAGNOSTICS = + process.env.HOME_VIRTUAL_AUTH_BROWSER_AUDIO_ENGINE_DIAGNOSTICS === "1"; const CHECK_BROWSER_CONTROLLED_INSPECTION = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_INSPECTION === "1"; const CHECK_BROWSER_CONTROLLED_OPERATOR = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_OPERATOR === "1"; const REUSE_SIGNED_HOME = process.env.HOME_VIRTUAL_AUTH_REUSE_SIGNED_HOME === "1"; @@ -165,8 +175,17 @@ const qualificationCancellation = createQualificationCancellation(BROWSER_QUALIF let browserQualification = null; const BROWSER_OPERATOR_COORDS_PATH = process.env.HOME_VIRTUAL_AUTH_BROWSER_OPERATOR_COORDS || ""; const CHECK_BROWSER_CONTROLLED_RECOVERY = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_RECOVERY === "1"; +const BROWSER_DEFER_SCROLL = process.env.HOME_VIRTUAL_AUTH_BROWSER_DEFER_SCROLL === "1"; const CHECK_BROWSER_VIEWER_RELOAD = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_VIEWER_RELOAD === "1"; +const CHECK_BROWSER_RELOAD_CENSUS = process.env.HOME_VIRTUAL_AUTH_BROWSER_RELOAD_CENSUS === "1"; +const BROWSER_RELOAD_CENSUS_OUT = process.env.U9_RELOAD_CENSUS_OUT || ""; +const BROWSER_RELOAD_OBSERVE_MS = Number.parseInt( + process.env.HOME_VIRTUAL_AUTH_BROWSER_RELOAD_OBSERVE_MS || "0", + 10, +); const BROWSER_CONTROLLED_TURN_TEST_HOME = process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_TURN_TEST_HOME || ""; +const BROWSER_CONTROLLED_TURN_CONTROL_SOCKET = + process.env.HOME_VIRTUAL_AUTH_BROWSER_CONTROLLED_TURN_CONTROL_SOCKET || ""; const REQUIRE_BROWSER_VZ_TRANSPORT = process.env.HOME_VIRTUAL_AUTH_BROWSER_REQUIRE_VZ_TRANSPORT === "1"; const BROWSER_JOURNEY_TARGET = CHECK_BROWSER_CONTROLLED_JOURNEY || process.env.HOME_VIRTUAL_AUTH_BROWSER_PROFILE_MODE || process.env.HOME_VIRTUAL_AUTH_BROWSER_PROFILE_MARKER ? browserJourneyTargetConfig(process.env) : null; @@ -2055,6 +2074,8 @@ async function holdBrowserUiForSetup(context, browserToken, route) { async function waitForEmbeddedBrowserPage(appFrame, failures, ignoredOpenIds = new Set()) { const deadline = Date.now() + BROWSER_UI_PAGE_ID_TIMEOUT_MS; while (Date.now() < deadline) { + const pageId = await appFrame.evaluate(() => window.__elastosBrowserCurrentPageId || ""); + if (pageId) return pageId; const failure = failures.find(entry => entry.frame === appFrame && !(entry.open_id && ignoredOpenIds.has(entry.open_id))); if (failure) { @@ -2063,8 +2084,6 @@ async function waitForEmbeddedBrowserPage(appFrame, failures, ignoredOpenIds = n response_format: failure.response_format, admission_reason: failure.admission_reason }, }); } - const pageId = await appFrame.evaluate(() => window.__elastosBrowserCurrentPageId || ""); - if (pageId) return pageId; await delay(200); } throw new Error("Embedded Browser UI did not publish the current page id before timeout"); @@ -2256,8 +2275,15 @@ async function closeControlledBrowserWindow(page, appFrame, windowIdentity, toke browser_instance: safeText(body?.browser_instance) }, receipt: null }; record.detail = detail; evidence.close_responses.push(detail); - const receipt = await response.json().catch(() => null); + const raw = await response.text().catch(() => ""); + let receipt = null; + try { + receipt = raw ? JSON.parse(raw) : null; + } catch { + detail.error_body = safeText(raw); + } record.receipt = receipt; + if (!receipt && raw && !detail.error_body) detail.error_body = safeText(raw); if (receipt) detail.receipt = { schema: safeText(receipt.schema), page_id: safeText(receipt.page_id), cleanup_id: safeText(receipt.cleanup_id), browser_instance: safeText(receipt.browser_instance), closed: receipt.closed === true, @@ -2267,6 +2293,9 @@ async function closeControlledBrowserWindow(page, appFrame, windowIdentity, toke terminal_effects: Object.fromEntries([...effects, ...transportEffects] .filter(key => key in (receipt.terminal_effects || {})).map(key => [key, receipt.terminal_effects[key] === true])), transport_proof_present: Boolean(receipt.transport_proof), + ...(receipt.profile_durability === "failed" || receipt.profile_durability === "unknown" + || receipt.profile_durability === "proved" + ? { profile_durability: receipt.profile_durability } : {}), }; } catch (error) { evidence.response_error = safeText(error.message); @@ -2522,45 +2551,209 @@ async function observeControlledBrowserRequests(page, appFrame, token, record, } } +async function extendControlledBrowserInput(page, appFrame, readReceipt, suffix, { timeoutMs }, evidence = {}) { + const deadline = performance.now() + Math.max(1, timeoutMs); + const remain = () => Math.max(1, deadline - performance.now()); + const receipt = await readReceipt(); + const events = receipt.events || []; + const last = [...events].reverse().find(event => event?.input_rect); + const load = events.find(event => event?.type === "load" && event.input_rect); + evidence.field_y = last?.input_rect?.y ?? null; + evidence.scroll_y = last?.scroll_y ?? null; + evidence.scrolled = 0; + evidence.clicked = false; + evidence.typed = false; + let rect = last?.input_rect || load?.input_rect; + const restRect = load?.input_rect; + // Journey wheel(0, 640) can move the field more than 640 px. Guest scroll + // then returns the field to its rest viewport y. Click the rest rectangle + // after one lift. A guessed lifted y misses the field; a settle poll can + // lift again from a mid-scroll y. + const clickRect = recoveredInputClickRect(rect, restRect); + if (clickRect.lift && remain() > 200) { + await appFrame.locator("#browser-remote-display").hover({ timeout: remain() }); + await page.mouse.wheel(0, -clickRect.lift); + evidence.scrolled = -clickRect.lift; + rect = clickRect.rect; + evidence.field_y = rect.y; + if (Number.isFinite(load?.scroll_y)) evidence.scroll_y = load.scroll_y; + } + if (!rect) { + throw new Error("Recovered input has no fixture field rectangle"); + } + const point = await remoteVideoClickPositionForPagePoint(appFrame, { + x: rect.x + rect.width / 2, + y: rect.y + rect.height / 2, + }); + if (!point || point.x < 0 || point.y < 0 || point.x > point.video_width || point.y > point.video_height) { + throw new Error("Recovered input field is outside the visible remote display"); + } + await appFrame.locator("#browser-remote-display").click({ + position: { x: point.x, y: point.y }, + timeout: remain(), + }); + evidence.clicked = true; + const pasted = await appFrame.evaluate((text) => { + const panel = document.querySelector("#browser-render-panel"); + const data = new DataTransfer(); + data.setData("text/plain", text); + const event = new ClipboardEvent("paste", { bubbles: true, cancelable: true }); + Object.defineProperty(event, "clipboardData", { value: data }); + return panel.dispatchEvent(event) || event.defaultPrevented; + }, suffix); + if (!pasted) { + throw new Error("Recovered input paste did not reach the Browser page"); + } + evidence.typed = true; + return evidence; +} + async function runControlledBrowserRecovery(page, appFrame, token, readReceipt, expectedUrl) { markStage("browser:controlled-recovery"); + const recoveredInput = { scrolled: 0, clicked: false, typed: false }; const instance = new URL(appFrame.url()).searchParams.get("browser_instance"); assert(instance, "Recovery probe requires the current Browser instance"); const runtimeOrigin = new URL(appFrame.url()).origin; - const summaryUrl = new URL(`/api/apps/browser/summary?browser_instance=${encodeURIComponent(instance)}`, runtimeOrigin); + const summaryUrl = browserJourneySummaryUrl(runtimeOrigin, instance); const probeId = randomUUID(); + const knownPageId = await appFrame.evaluate(() => window.__elastosBrowserCurrentPageId || ""); const mediaInterruption = BROWSER_CONTROLLED_TURN_TEST_HOME ? await (await import("./lib/browser-journey-turn-interruption.mjs")).createBrowserTurnInterruption({ testHome: BROWSER_CONTROLLED_TURN_TEST_HOME, runtimeOrigin, - pageId: await appFrame.evaluate(() => window.__elastosBrowserCurrentPageId), + pageId: knownPageId, + ...(BROWSER_CONTROLLED_TURN_CONTROL_SOCKET + ? { controlSocketPath: BROWSER_CONTROLLED_TURN_CONTROL_SOCKET } : {}), }).catch(error => { error.details = { recovery: { ok: false, failure: "media_driver_binding_failed", media_driver: error.evidence } }; throw error; }) : null; const { session: cdp, ancestorDepth } = await browserRecoveryCdpSession(page, appFrame); + let lastViewer = { + page_id: knownPageId, + browser_instance: instance, + actual_url: "", + engine_id: "", + exit_id: "", + }; + const bindingReads = []; + let lastPageStatus = null; try { const evidence = await diagnoseBrowserJourneyRecovery({ cdp, expectedUrl, mediaInterruption, readBinding: async ({ signal }) => { - // This observer uses the same scoped authority outside the cut viewer. - const response = await fetch(summaryUrl, { headers: { Origin: "null", "x-elastos-home-token": token }, signal }); - assert(response.ok, "Recovery Runtime summary failed"); - const body = await response.json(); - const viewer = await appFrame.evaluate(() => ({ + // Runtime ownership is fetched outside the interrupted viewer transport. + // Viewer evaluate can stall during reconnect; known page_id starts the + // Runtime reads immediately and fills chrome fields if evaluate aborts. + const evaluateViewer = (timeoutMs) => appFrame.locator("html").evaluate(() => ({ page_id: window.__elastosBrowserCurrentPageId || "", browser_instance: new URL(location.href).searchParams.get("browser_instance"), actual_url: document.querySelector("#browser-url")?.value || "", engine_id: document.querySelector("#browser-engine")?.value, exit_id: document.querySelector("#browser-exit")?.value, - })); - const statusResponse = await fetch(new URL(`/api/apps/browser/pages/${encodeURIComponent(viewer.page_id)}/status`, runtimeOrigin), - { headers: { Origin: "null", "x-elastos-home-token": token }, signal }); - assert(statusResponse.ok, "Recovery page status failed"); - return { sessions: body.sessions, page_status: await statusResponse.json(), ...viewer }; + }), undefined, { timeout: Math.max(1, Math.ceil(timeoutMs)) }); + const fetchRuntime = async (pageId, timing) => { + const headers = { Origin: "null", "x-elastos-home-token": token }; + const statusUrl = new URL(`/api/apps/browser/pages/${encodeURIComponent(pageId)}/status`, runtimeOrigin); + const started = performance.now(); + const response = await fetch(summaryUrl, { headers, signal }); + assert(response.ok, "Recovery Runtime summary failed"); + const sessions = (await response.json()).sessions; + if (timing) timing.summary_ms = Math.round(performance.now() - started); + // page_status probes the Engine. Timeout reuse keeps the last + // matching status and marks it stale. HTTP and JSON errors stay + // visible. A first miss without a cached status retries once on + // the remaining binding window. + const probeStatus = async (budgetMs) => { + const result = await readBrowserPageStatus({ + fetchImpl: fetch, + url: statusUrl, + headers, + signal, + budgetMs, + lastPageStatus, + }); + if (timing) { + timing.status_ms = Math.round(performance.now() - started); + timing.status_reused = !result.page_status_fresh; + } + if (result.page_status_fresh) lastPageStatus = result.page_status; + return result; + }; + let status; + try { + status = lastPageStatus ? await probeStatus(800) : await probeStatus(3000); + } catch (error) { + if (timing) timing.status_ms = Math.round(performance.now() - started); + if (signal?.aborted || lastPageStatus || !isPageStatusTimeout(error)) throw error; + status = await probeStatus(null); + } + if (!status.page_status_fresh && !signal?.aborted) { + status = await probeStatus(null); + } + return { + sessions, + page_status: status.page_status, + page_status_fresh: status.page_status_fresh, + }; + }; + const knownId = lastViewer.page_id; + if (knownId) { + // Runtime ownership is the required observation. A reconnecting + // viewer evaluate can hang until Playwright's default timeout and + // then lose the 5 s clock even after summary and status return. + const chromeMs = 400; + const started = performance.now(); + const read = { known_id: true, evaluate_ms: null, runtime_ms: null, summary_ms: null, status_ms: null }; + try { + const [viewerResult, runtime] = await Promise.all([ + evaluateViewer(chromeMs).then((value) => { + read.evaluate_ms = Math.round(performance.now() - started); + return value; + }).catch(() => { + read.evaluate_ms = Math.round(performance.now() - started); + return null; + }), + fetchRuntime(knownId, read).then((value) => { + read.runtime_ms = Math.round(performance.now() - started); + return value; + }), + ]); + bindingReads.push(read); + if (bindingReads.length > 8) bindingReads.shift(); + const viewer = { + page_id: viewerResult?.page_id || lastViewer.page_id || runtime.page_status?.page_id || "", + browser_instance: viewerResult?.browser_instance || lastViewer.browser_instance || instance, + actual_url: viewerResult?.actual_url || runtime.page_status?.actual_url || lastViewer.actual_url, + engine_id: viewerResult?.engine_id ?? lastViewer.engine_id ?? "", + exit_id: viewerResult?.exit_id ?? lastViewer.exit_id ?? "", + }; + if (viewer.page_id) lastViewer = viewer; + return { ...runtime, ...viewer }; + } catch (error) { + bindingReads.push(read); + if (bindingReads.length > 8) bindingReads.shift(); + throw error; + } + } + const aborted = () => { + const error = new Error("binding_aborted"); + error.name = "AbortError"; + return error; + }; + const untilAbort = new Promise((_, reject) => { + if (signal?.aborted) { + reject(aborted()); + return; + } + signal?.addEventListener("abort", () => reject(aborted()), { once: true }); + }); + const viewer = await Promise.race([evaluateViewer(5000), untilAbort]); + lastViewer = viewer; + return { ...await fetchRuntime(viewer.page_id, null), ...viewer }; }, readVideo: async () => { const metrics = await browserRemoteVideoMetrics(appFrame); @@ -2569,8 +2762,16 @@ async function runControlledBrowserRecovery(page, appFrame, token, readReceipt, return { ...metrics, ...(Number.isSafeInteger(bytes) ? { video_bytes_received: bytes } : {}) }; }, readReceipt, - extendInput: (suffix, { timeoutMs }) => appFrame.locator("#browser-keyboard-capture") - .pressSequentially(suffix, { timeout: Math.ceil(timeoutMs) }), + extendInput: async (suffix, budget) => { + try { + Object.assign(recoveredInput, await extendControlledBrowserInput( + page, appFrame, readReceipt, suffix, budget, recoveredInput, + )); + } catch (error) { + recoveredInput.error = String(error?.message || error).slice(0, 240); + throw error; + } + }, probeViewerRequest: ({ timeoutMs }) => appFrame.evaluate(async ({ probeId, timeoutMs }) => { const id = window.__elastosBrowserCurrentPageId; const token = new URLSearchParams(location.hash.slice(1)).get("home_token"); @@ -2582,26 +2783,111 @@ async function runControlledBrowserRecovery(page, appFrame, token, readReceipt, }, { probeId, timeoutMs }), observeRequests: (record, { signal }) => observeControlledBrowserRequests(page, appFrame, token, record, { probeId, signal }), }); - return { ...evidence, cdp_ancestor_depth: ancestorDepth, + return { ...evidence, cdp_ancestor_depth: ancestorDepth, recovered_input: recoveredInput, + binding_reads: bindingReads, ...(mediaInterruption ? { media_driver: mediaInterruption.evidence } : {}) }; } catch (error) { error.details = { ...error.details, recovery: { ...error.evidence, failure: error.evidence?.failure || "probe_setup_or_observation_failed", cdp_ancestor_depth: ancestorDepth, + recovered_input: recoveredInput, binding_reads: bindingReads, ...(mediaInterruption ? { media_driver: mediaInterruption.evidence } : {}) } }; throw error; } } +function browserJourneySummaryUrl(runtimeOrigin, instance) { + const url = new URL("/api/apps/browser/summary", runtimeOrigin); + url.searchParams.set("browser_instance", instance); + // Installed Mac 63be1c09 and seed 21388ce6 both accept this query. + // The older Mac-only browser_instance shape stays in its historical receipt. + url.searchParams.set("remote_services", "0"); + return url; +} + +async function runControlledBrowserReloadCensus(page, appFrame, token, expectedUrl) { + markStage("browser:reload-census"); + const instance = new URL(appFrame.url()).searchParams.get("browser_instance"); + const runtimeOrigin = new URL(appFrame.url()).origin; + assert(instance, "Reload census requires the current Browser instance"); + const headers = { Origin: "null", "x-elastos-home-token": token }; + let lastPageStatus = null; + let statusInFlight = null; + const readVisible = () => appFrame.evaluate(readBrowserViewerReloadDocument).catch((error) => { + if (!/Execution context was destroyed|Cannot find context with specified id/.test(String(error.message))) { + throw error; + } + return { viewer: null, video: null }; + }); + const peekPageStatus = (pageId, signal) => { + if (pageId && !statusInFlight) { + statusInFlight = fetch( + `${runtimeOrigin}/api/apps/browser/pages/${encodeURIComponent(pageId)}/status`, + { headers, signal }, + ).then((response) => (response.ok ? response.json() : null), () => null) + .then((status) => { + if (status) lastPageStatus = status; + return status; + }) + .finally(() => { statusInFlight = null; }); + } + return lastPageStatus; + }; + const evidence = await censusBrowserViewerReload({ + expectedUrl, + watchMs: Number.isFinite(BROWSER_RELOAD_OBSERVE_MS) && BROWSER_RELOAD_OBSERVE_MS > 5000 + ? BROWSER_RELOAD_OBSERVE_MS + : 5000, + readState: async ({ signal }) => { + const visible = await readVisible(); + return { + sessions: null, + page_status: peekPageStatus(visible.viewer?.page_id, signal), + ...visible, + }; + }, + reloadViewer: async ({ timeoutMs }) => { + await Promise.all([ + appFrame.waitForNavigation({ waitUntil: "commit", timeout: Math.ceil(timeoutMs) }), + appFrame.evaluate(() => location.reload()), + ]); + }, + observeRequests: (record, { signal }) => + observeControlledBrowserRequests(page, appFrame, token, record, { recordNavigation: true, signal }), + observeDiagnostics: (record) => { + const onConsole = (message) => { + const text = message.text(); + if (!text.includes("elastos.browser.media-diagnostic/v1")) return; + try { record(JSON.parse(text)); } catch { /* keep the census bounded to valid events */ } + }; + page.on("console", onConsole); + return () => { page.off("console", onConsole); }; + }, + }); + return evidence; +} + async function runControlledBrowserViewerReload(page, appFrame, token, readReceipt, expectedUrl) { markStage("browser:controlled-viewer-reload"); const instance = new URL(appFrame.url()).searchParams.get("browser_instance"); const runtimeOrigin = new URL(appFrame.url()).origin; assert(instance, "Viewer reload requires the current Browser instance"); const stateObservation = { clock: "performance.now", steps: [], dropped_steps: 0 }; - let sampleId = 0; + let sampleId = 0, lastFreshPageId = "", lastFreshStatus = null, statusRefresh = null; try { const evidence = await diagnoseBrowserViewerReload({ expectedUrl, readReceipt, + observeMs: Number.isFinite(BROWSER_RELOAD_OBSERVE_MS) && BROWSER_RELOAD_OBSERVE_MS > 5000 + ? BROWSER_RELOAD_OBSERVE_MS + : 0, + observeReadState: async ({ signal }) => { + const visible = await appFrame.evaluate(readBrowserViewerReloadDocument).catch((error) => { + if (!/Execution context was destroyed|Cannot find context with specified id/.test(String(error.message))) { + throw error; + } + return { viewer: null, video: null }; + }); + return { sessions: null, page_status: lastFreshStatus, ...visible }; + }, readState: async ({ signal, deadlineMs }) => { const sample = ++sampleId; const step = async (name, action) => { @@ -2623,31 +2909,50 @@ async function runControlledBrowserViewerReload(page, appFrame, token, readRecei finally { signal.removeEventListener("abort", aborted); } }; const headers = { Origin: "null", "x-elastos-home-token": token }; - const { sessions } = await step("runtime_summary", async headersReceived => { - const response = await fetch(new URL(`/api/apps/browser/summary?browser_instance=${encodeURIComponent(instance)}`, runtimeOrigin), + const readSummary = () => step("runtime_summary", async headersReceived => { + const response = await fetch(browserJourneySummaryUrl(runtimeOrigin, instance), { headers, signal }); headersReceived(); assert(response.ok, "Viewer reload Runtime summary failed"); return response.json(); }); + const readStatus = pageId => step("remote_page_status", async headersReceived => { + const status = await fetch(new URL(`/api/apps/browser/pages/${encodeURIComponent(pageId)}/status`, runtimeOrigin), { headers, signal }); + headersReceived(); + if (status.ok) return status.json(); + assert(status.status === 404, "Viewer reload Runtime page status failed"); + return null; + }); + const readViewer = async () => { + try { + // Read the document identity and media in one execution context. + return await step("viewer_document_metrics", () => appFrame.evaluate(readBrowserViewerReloadDocument)); + } catch (error) { + if (!/Execution context was destroyed|Cannot find context with specified id/.test(String(error.message))) throw error; + return { viewer: null, video: null }; + } + }; + // Fresh summary remains the live owner. Last status is binding + // continuity only, so a later sample can publish viewer metrics + // while an Engine probe is still in flight. + const [visible, summaryResult] = await Promise.all([readViewer(), readSummary()]); + const { sessions } = summaryResult; const pageId = sessions?.recoverable_page?.page_id; let page_status = null; - if (pageId) { - page_status = await step("remote_page_status", async headersReceived => { - const status = await fetch(new URL(`/api/apps/browser/pages/${encodeURIComponent(pageId)}/status`, runtimeOrigin), { headers, signal }); - headersReceived(); - if (status.ok) return status.json(); - assert(status.status === 404, "Viewer reload Runtime page status failed"); - return null; - }); - } - let visible = { viewer: null, video: null }; - try { - // Read the document identity and media in one execution context. - visible = await step("viewer_document_metrics", () => appFrame.evaluate(readBrowserViewerReloadDocument)); - } catch (error) { - if (!/Execution context was destroyed|Cannot find context with specified id/.test(String(error.message))) throw error; + if (pageId && pageId === lastFreshPageId && lastFreshStatus) { + page_status = lastFreshStatus; + if (!statusRefresh) { + statusRefresh = readStatus(pageId).then(status => { + if (status) lastFreshStatus = status; + }).finally(() => { statusRefresh = null; }); + } + } else if (pageId) { + page_status = await readStatus(pageId); + if (page_status) lastFreshStatus = page_status; + } else { + lastFreshStatus = null; } + lastFreshPageId = pageId || ""; return { sessions, page_status, ...visible }; }, reloadViewer: async ({ timeoutMs }) => { @@ -2656,8 +2961,9 @@ async function runControlledBrowserViewerReload(page, appFrame, token, readRecei appFrame.evaluate(() => location.reload()), ]); }, - extendInput: (suffix, { timeoutMs }) => appFrame.locator("#browser-keyboard-capture") - .pressSequentially(suffix, { timeout: Math.ceil(timeoutMs) }), + extendInput: (suffix, budget) => extendControlledBrowserInput( + page, appFrame, readReceipt, suffix, budget, + ), observeRequests: (record, { signal }) => observeControlledBrowserRequests(page, appFrame, token, record, { recordNavigation: true, signal }), }); return { ...evidence, state_observation: stateObservation }; @@ -2901,7 +3207,10 @@ async function runControlledBrowserJourney(page, appFrame, windowIdentity, token null, { timeout: BROWSER_UI_PAGE_ID_TIMEOUT_MS }); if (fixture.engineId) { await appFrame.locator("#browser-settings").click(); - await appFrame.locator("#browser-engine").selectOption(fixture.engineId); + const selectedEngine = await appFrame.locator("#browser-engine").inputValue(); + if (selectedEngine !== fixture.engineId) { + await appFrame.locator("#browser-engine").selectOption(fixture.engineId); + } assert(await appFrame.locator("#browser-engine").inputValue() === fixture.engineId, "Requested Browser Engine UI selection failed"); await appFrame.locator("#browser-settings-close").click(); @@ -2921,7 +3230,7 @@ async function runControlledBrowserJourney(page, appFrame, windowIdentity, token }); const ignoredOpenIds = new Set(settled.map(entry => entry.open_id)); result.prior_open_settlements = settled.map(({ open_id, body }) => ({ open_id, settlement: body })); - for (const name of ["main", "nav"]) { + for (const name of CHECK_BROWSER_RELOAD_CENSUS ? ["nav"] : ["main", "nav"]) { markStage(`browser:controlled-${name}`); const documentSequence = fixture.profile ? (await readReceipt()).events.at(-1)?.sequence || 0 : 0; const url = browserJourneyFixtureUrl(fixture, run, name, @@ -2973,6 +3282,19 @@ async function runControlledBrowserJourney(page, appFrame, windowIdentity, token await checkProfileDocument(name, status.page_id, documentSequence); } const current = result.pages.at(-1); + if (CHECK_BROWSER_RELOAD_CENSUS) { + try { + result.reload_census = await runControlledBrowserReloadCensus(page, appFrame, token, current.url); + } catch (error) { + if (error.evidence) result.reload_census = error.evidence; + throw error; + } finally { + if (BROWSER_RELOAD_CENSUS_OUT && result.reload_census) { + writeFileSync(BROWSER_RELOAD_CENSUS_OUT, `${JSON.stringify(result.reload_census, null, 2)}\n`); + } + } + return { page_id: current.page_id, display_mode: BROWSER_OPEN_DISPLAY_MODE, controlled_journey: result }; + } inputObserver = await observeControlledBrowserInput(page, appFrame, token, current.page_id); result.input_observation = inputObserver.evidence; const rect = current.load.input_rect; @@ -3027,22 +3349,32 @@ async function runControlledBrowserJourney(page, appFrame, windowIdentity, token event.audio_state === "running" && event.frequency_hz === 440), "controlled Engine tone starts"); result.audio = await appFrame.evaluate(() => window.__readBrowserJourneyAudio()); result.audio.fixture_event = toneReceipt.events.find(event => event.page === "nav" && event.type === "audio"); - if (!controlledTonePresent(result.audio)) { + const afterTone = await readReceipt(); + result.audio.fixture_event_after = [...afterTone.events].reverse().find(event => + event.page === "nav" && event.type === "audio") || null; + if (!controlledTonePresent(result.audio) || CHECK_BROWSER_AUDIO_ENGINE_DIAGNOSTICS) { result.audio.receiver_metrics = await browserRemoteDisplayMetrics(appFrame).catch(() => null); result.audio.engine_diagnostics = await checkBrowserPageDiagnostics(appFrame, token, current.page_id) .catch(error => ({ error: error.message, details: error.details })); } assert(controlledTonePresent(result.audio), "Controlled 440 Hz tone was not decoded by the product receiver", result.audio); } - markStage("browser:scroll"); - await appFrame.locator("#browser-remote-display").hover(); - await page.mouse.wheel(0, 640); - const receipt = await waitForJourneyEvidence(readReceipt, - value => value.events.some(event => event.page === "nav" && event.type === "scroll" && - event.value === text && event.scroll_y >= current.load.scroll_y + 100), "Engine page scroll movement"); - const afterInput = await waitForJourneyEvidence(() => browserRemoteVideoMetrics(appFrame), - value => value.decoded_frames > current.video.decoded.decoded_frames, "decoded frames after input"); - result.input = { text, receipt, video_after_input: afterInput }; + const typedReceipt = await readReceipt(); + // Recovery proves reconnection on the field the person already sees. + // Ordinary scroll stays a later complete-journey case. + if (BROWSER_DEFER_SCROLL) { + result.input = { text, receipt: typedReceipt, scroll_deferred: true }; + } else { + markStage("browser:scroll"); + await appFrame.locator("#browser-remote-display").hover(); + await page.mouse.wheel(0, 640); + const receipt = await waitForJourneyEvidence(readReceipt, + value => value.events.some(event => event.page === "nav" && event.type === "scroll" && + event.value === text && event.scroll_y >= current.load.scroll_y + 100), "Engine page scroll movement"); + const afterInput = await waitForJourneyEvidence(() => browserRemoteVideoMetrics(appFrame), + value => value.decoded_frames > current.video.decoded.decoded_frames, "decoded frames after input"); + result.input = { text, receipt, video_after_input: afterInput }; + } if (browserQualification) { result.qualification = await browserQualification.observe({ appFrame, pageId: current.page_id, readReceipt, readStatus: () => browserApi(appFrame, token, `/api/apps/browser/pages/${encodeURIComponent(current.page_id)}/status`), @@ -3215,6 +3547,12 @@ async function checkBrowserEmbeddedUiInput(page, baselineToken) { }, { capture: true }); }); markStage("browser:page-acquisition"); + await appFrame.locator("#browser-url").waitFor({ state: "visible", timeout: 15_000 }); + await appFrame.waitForFunction(() => document.querySelector("#browser-url")?.disabled === false, + null, { timeout: BROWSER_UI_PAGE_ID_TIMEOUT_MS }); + if (!(await appFrame.evaluate(() => window.__elastosBrowserCurrentPageId || ""))) { + await appFrame.locator("#browser-url").press("Enter"); + } pageId = await waitForEmbeddedBrowserPage(appFrame, openFailures).catch(async error => { error.details = { ...error.details, ui: await embeddedBrowserDebugState(windowLocator, appFrame) }; throw error; @@ -3860,31 +4198,67 @@ async function openDesktopAppWindow(page, target, onFrame = null) { await setupReminder.waitFor({ state: "hidden", timeout: 5_000 }); } if (target === "browser") { - // Match Home's foreground choice once. DOM order can differ from z-order, - // and a later restored window must not replace the chosen frame. - const selected = (await homeGuiFrame.evaluateHandle(() => { - const windows = [...document.querySelectorAll('section.window[data-target="browser"]')]; - const active = windows.filter(node => node.classList.contains("window-active")); - if (active.length > 1) throw new Error("Home has ambiguous active Browser windows"); - const visible = windows.filter(node => !node.classList.contains("hidden")); - return active[0] || (visible.length ? visible : windows) - .sort((a, b) => Number(b.style.zIndex || 0) - Number(a.style.zIndex || 0))[0] || null; - })).asElement(); let handle; - if (selected) { - handle = await selected.$("iframe.window-frame"); - } else { + if (CHECK_BROWSER_CONTROLLED_JOURNEY) { + // Hidden leftovers stay unused. Home New Window creates the journey surface. + await homeGuiFrame.locator("#identity-menu-show-desktop").evaluate(el => el.click()).catch(() => {}); + await homeGuiFrame.waitForFunction(() => document.querySelector("#launcher")?.hidden !== false, null, { timeout: 5_000 }).catch(() => {}); + const priorSrcs = await homeGuiFrame.evaluate(() => + [...document.querySelectorAll('section.window[data-target="browser"] iframe.window-frame')] + .map(frame => frame.getAttribute("src") || ""), + ); const shelf = homeGuiFrame.locator('#taskbar-targets [data-target="browser"]').first(); - if (await shelf.isVisible()) await shelf.click(); + if (await shelf.count()) await shelf.click({ button: "right" }); else { await homeGuiFrame.locator("#launcher-toggle").click(); - await homeGuiFrame.locator('#launcher-grid [data-target="browser"]').first().click(); + await homeGuiFrame.locator('#launcher-grid [data-target="browser"]').first().click({ button: "right" }); + } + await homeGuiFrame.waitForFunction(() => { + const menu = document.querySelector("#desktop-context-menu"); + return Boolean(menu && menu.hidden !== true && + menu.querySelector('[data-context-action="open-target-new-window"], [data-context-action="open-target"]')); + }, null, { timeout: 5_000 }).catch(() => { + throw new Error("Home did not offer Browser Open or New Window"); + }); + const newWindow = homeGuiFrame.locator('#desktop-context-menu [data-context-action="open-target-new-window"]'); + const ordinaryOpen = homeGuiFrame.locator('#desktop-context-menu [data-context-action="open-target"]'); + if (await newWindow.count()) await newWindow.click(); + else if (await ordinaryOpen.count()) await ordinaryOpen.click(); + else throw new Error("Home did not offer Browser Open or New Window"); + await homeGuiFrame.waitForFunction(prior => { + return [...document.querySelectorAll('section.window[data-target="browser"]:not(.hidden) iframe.window-frame')] + .some(frame => { + const src = frame.getAttribute("src") || ""; + return src.includes("/apps/browser/") && !prior.includes(src); + }); + }, priorSrcs, { timeout: 20_000 }); + handle = await homeGuiFrame.locator('section.window[data-target="browser"]:not(.hidden).window-active iframe.window-frame').elementHandle(); + } else { + // Match Home's foreground choice once. DOM order can differ from z-order, + // and a later restored window must not replace the chosen frame. + const selected = (await homeGuiFrame.evaluateHandle(() => { + const windows = [...document.querySelectorAll('section.window[data-target="browser"]')]; + const active = windows.filter(node => node.classList.contains("window-active")); + if (active.length > 1) throw new Error("Home has ambiguous active Browser windows"); + const visible = windows.filter(node => !node.classList.contains("hidden")); + return visible.sort((a, b) => Number(b.style.zIndex || 0) - Number(a.style.zIndex || 0))[0] || + active[0] || null; + })).asElement(); + if (selected) { + handle = await selected.$("iframe.window-frame"); + } else { + const shelf = homeGuiFrame.locator('#taskbar-targets [data-target="browser"]').first(); + if (await shelf.isVisible()) await shelf.click(); + else { + await homeGuiFrame.locator("#launcher-toggle").click(); + await homeGuiFrame.locator('#launcher-grid [data-target="browser"]').first().click(); + } + // There was no Browser at selection. Ambiguous concurrent creations fail + // the strict locator instead of assigning another window's authority. + const created = homeGuiFrame.locator('section.window[data-target="browser"]:not(.hidden) iframe.window-frame'); + await created.waitFor({ state: "attached", timeout: 20_000 }); + handle = await created.elementHandle(); } - // There was no Browser at selection. Ambiguous concurrent creations fail - // the strict locator instead of assigning another window's authority. - const created = homeGuiFrame.locator('section.window[data-target="browser"] iframe.window-frame'); - await created.waitFor({ state: "attached", timeout: 20_000 }); - handle = await created.elementHandle(); } const appFrame = handle ? await handle.contentFrame() : null; assert(appFrame, "Selected Home Browser window has no content frame"); @@ -4550,6 +4924,46 @@ async function checkBrowserLaunchGrant(page, homeToken) { assert(launched.body?.target === "browser", "Browser launch did not resolve the Browser capsule", launched); const route = String(launched.body?.route || ""); const browserToken = assertIsolatedLaunchRoute(route, "browser"); + const officialClosePageId = process.env.HOME_VIRTUAL_AUTH_OFFICIAL_CLOSE_PAGE || ""; + const officialCloseCleanupId = process.env.HOME_VIRTUAL_AUTH_OFFICIAL_CLOSE_CLEANUP || ""; + if (officialClosePageId) { + assert(officialCloseCleanupId, "Official close requires HOME_VIRTUAL_AUTH_OFFICIAL_CLOSE_CLEANUP"); + const officialCloseInstance = process.env.HOME_VIRTUAL_AUTH_OFFICIAL_CLOSE_INSTANCE || ""; + const appFrame = await openDesktopAppWindow(page, "browser"); + const frameToken = assertIsolatedLaunchRoute(appFrame.url(), "browser"); + const closeBody = { + schema: "elastos.browser.close-request/v2", + cleanup_id: officialCloseCleanupId, + }; + if (officialCloseInstance) { + closeBody.browser_instance = officialCloseInstance; + } + const closed = await browserApi( + appFrame, + frameToken, + `/api/apps/browser/pages/${encodeURIComponent(officialClosePageId)}/close`, + { + method: "POST", + body: closeBody, + }, + ); + launched.body.official_close = closed; + console.error(JSON.stringify(redactSensitive({ + stage: "browser:official-close", + page_id: officialClosePageId, + ok: closed.ok, + status: closed.status, + body: closed.body, + }))); + assert(closed.ok, "Official Runtime close failed", closed); + assert( + closed.body?.schema === "elastos.browser.close-result/v1" && + closed.body?.closed === true, + "Official Runtime close did not return a closed receipt", + closed, + ); + return launched.body; + } if (CHECK_BROWSER_UI_SETUP) { launched.body.browser_ui_setup = await holdBrowserUiForSetup(page.context(), browserToken, route); } @@ -5008,6 +5422,10 @@ async function revokeCurrentPasskey(page, proofBindingId, homeToken) { async function main() { assert(!BROWSER_CONTROLLED_TURN_TEST_HOME || CHECK_BROWSER_CONTROLLED_RECOVERY, "The task TURN interruption requires controlled Browser recovery"); + assert(!BROWSER_CONTROLLED_TURN_CONTROL_SOCKET || BROWSER_CONTROLLED_TURN_TEST_HOME, + "The adapter TURN control socket requires an explicit TURN test Home"); + assert(!(REQUIRE_BROWSER_VZ_TRANSPORT && CHECK_BROWSER_CONTROLLED_RECOVERY) || BROWSER_CONTROLLED_TURN_TEST_HOME, + "VZ recovery requires an explicit TURN test Home so the media path can be interrupted"); assert(!CHECK_BROWSER_CONTROLLED_MEDIA || CHECK_BROWSER_CONTROLLED_JOURNEY, "Controlled audio proof requires the controlled Browser journey"); assert(!BROWSER_QUALIFICATION_OPTIONS || (CHECK_BROWSER_CONTROLLED_JOURNEY && CHECK_BROWSER_CONTROLLED_MEDIA && @@ -5020,6 +5438,12 @@ async function main() { "Controlled operator proof requires the controlled inspection journey and explicit Runtime coordinates"); assert(!CHECK_BROWSER_VIEWER_RELOAD || CHECK_BROWSER_CONTROLLED_JOURNEY, "Browser viewer reload requires the controlled journey"); + assert(!CHECK_BROWSER_RELOAD_CENSUS || CHECK_BROWSER_CONTROLLED_JOURNEY, + "Reload census requires the controlled journey"); + assert(!(CHECK_BROWSER_RELOAD_CENSUS && CHECK_BROWSER_VIEWER_RELOAD), + "Reload census replaces the five-second reload gate"); + assert(!CHECK_BROWSER_RELOAD_CENSUS || CHECK_BROWSER_CONTROLLED_MEDIA, + "Reload census needs decoded video on a working page"); assert(!CHECK_BROWSER_CONTROLLED_RECOVERY || CHECK_BROWSER_CONTROLLED_JOURNEY, "Controlled Browser recovery requires the controlled journey"); assert(!CHECK_BROWSER_CONTROLLED_JOURNEY || (INCLUDE_BROWSER && CHECK_BROWSER_EMBEDDED_UI_INPUT && @@ -5047,10 +5471,12 @@ async function main() { headless: HEADLESS, executablePath: process.env.ELASTOS_BROWSER_EXECUTABLE || undefined, ignoreHTTPSErrors: true, - viewport: { width: 1280, height: 900 }, + viewport: HEADLESS ? { width: 1280, height: 900 } : null, }); const context = qualificationCancellation ? await qualificationCancellation.ownContext(contextPromise) : await contextPromise; - if (CHECK_BROWSER_CONTROLLED_MEDIA) await context.addInitScript(installBrowserJourneyAudioProbe); + if (CHECK_BROWSER_CONTROLLED_MEDIA) { + await context.addInitScript(installBrowserJourneyAudioProbe, { windowMs: BROWSER_AUDIO_WINDOW_MS }); + } let page = context.pages()[0] || await context.newPage(); let created = null; let passkey = null; diff --git a/scripts/home-shell-bridge-smoke.mjs b/scripts/home-shell-bridge-smoke.mjs index da717c99e..f2ed0ec52 100644 --- a/scripts/home-shell-bridge-smoke.mjs +++ b/scripts/home-shell-bridge-smoke.mjs @@ -246,6 +246,7 @@ const summary = { { target: "people", title: "People", attach_kind: "iframe", role: "app", target_kind: "app" }, { target: "assistant", title: "Assistant", attach_kind: "iframe", role: "app", target_kind: "app" }, { target: "home-agent", title: "Home Agent", attach_kind: "iframe", role: "app", target_kind: "app" }, + { target: "marketplace", title: "Apps", attach_kind: "iframe", role: "app", target_kind: "app" }, { target: "system", title: "System", attach_kind: "iframe", role: "app", target_kind: "app" }, { target: "wallet", title: "Wallet", attach_kind: "iframe", role: "app", target_kind: "app" }, ], @@ -515,7 +516,7 @@ globalThis.fetch = async (url, init = {}) => { title: "System", }); } - if (["assistant", "home-agent"].includes(body?.target)) { + if (["assistant", "home-agent", "marketplace"].includes(body?.target)) { return jsonResponse({ attach_kind: "iframe", target: body.target, route: `/apps/${body.target}/?home_origin=http%3A%2F%2Flocalhost%3A61180#home_token=${body.target}-token` }); } @@ -1182,7 +1183,7 @@ for (const actor of ["assistant", "home-agent"]) { assert(shellMessages.some(m => m.payload?.result?.target === actor), "composer launch registered", shellMessages); const source = { parent: shellFrameWindow, postMessage() {} }; const token = `${actor}-token`; - const intent = { type: "home:open-target", homeToken: token, target: "system", query: { settings: "models" } }; + const intent = { type: "home:open-target", homeToken: token, target: "marketplace", query: { category: "models" } }; const count = () => shellMessages.filter(m => m.payload?.command === "open-target").length; const before = count(); sendChildMessage("null", source, intent); @@ -1194,7 +1195,7 @@ for (const actor of ["assistant", "home-agent"]) { ["null", walletFrameWindow, { ...intent, homeToken: "wallet-token" }], ["null", source, { ...intent, target: "wallet" }], ["null", source, { ...intent, query: { settings: "accounts" } }], - ["null", source, { ...intent, query: { settings: "models", offer_id: "injected" } }], + ["null", source, { ...intent, query: { category: "models", offer_id: "injected" } }], ]) sendChildMessage(origin, sender, data); await new Promise(resolve => setTimeout(resolve, 0)); assert(count() === before, "Models handoff rejects wrong actor/source/token/origin/target/query", shellMessages); @@ -1202,7 +1203,34 @@ for (const actor of ["assistant", "home-agent"]) { for (let i = 0; i < 50 && count() === before; i++) await new Promise(resolve => setTimeout(resolve, 0)); assert(count() === before + 1, "registered composer opens Models exactly once", shellMessages); const opened = shellMessages.filter(m => m.payload?.command === "open-target").at(-1).payload; - assert(opened.target === "system" && JSON.stringify(opened.query) === JSON.stringify({ settings: "models" }), "exact Models destination", opened); + assert(opened.target === "marketplace" && JSON.stringify(opened.query) === JSON.stringify({ category: "models" }), "exact Models destination", opened); +} + +{ + const modelCid = `bafybei${"a".repeat(51)}a`; + sendChildMessage("null", shellFrameWindow, { type: "home:launch-target", requestId: "launch-marketplace", + target: "marketplace", query: {}, homeToken: "gui-token" }); + for (let i = 0; i < 50 && !shellMessages.some(m => m.payload?.requestId === "launch-marketplace"); i++) { + await new Promise(resolve => setTimeout(resolve, 0)); + } + assert(shellMessages.some(m => m.payload?.result?.target === "marketplace"), "marketplace launch registered", shellMessages); + const marketSource = { parent: shellFrameWindow, postMessage() {} }; + sendChildMessage("null", marketSource, { type: "home:app-ready", homeToken: "marketplace-token" }); + const handoff = { type: "home:open-target", homeToken: "marketplace-token", target: "assistant", query: { model_cid: modelCid } }; + const count = () => shellMessages.filter(m => m.payload?.command === "open-target").length; + const before = count(); + for (const data of [ + { ...handoff, query: { model_cid: modelCid, extra: "x" } }, + { ...handoff, query: { settings: "models" } }, + { ...handoff, target: "home-gui" }, + ]) sendChildMessage("null", marketSource, data); + await new Promise(resolve => setTimeout(resolve, 0)); + assert(count() === before, "Marketplace Assistant handoff rejects extra query keys", shellMessages); + sendChildMessage("null", marketSource, handoff); + for (let i = 0; i < 50 && count() === before; i++) await new Promise(resolve => setTimeout(resolve, 0)); + assert(count() === before + 1, "Marketplace opens Assistant with exact CID", shellMessages); + const opened = shellMessages.filter(m => m.payload?.command === "open-target").at(-1).payload; + assert(opened.target === "assistant" && JSON.stringify(opened.query) === JSON.stringify({ model_cid: modelCid }), "exact Assistant handoff", opened); } const connectorLaunchesBeforeDeniedRequests = requests.filter( diff --git a/scripts/home-shell-regression-smoke.mjs b/scripts/home-shell-regression-smoke.mjs index a02e9200e..d3b3320f8 100644 --- a/scripts/home-shell-regression-smoke.mjs +++ b/scripts/home-shell-regression-smoke.mjs @@ -2392,6 +2392,24 @@ const [safeBrowser] = shellWindows.normalizeRestorableSession(shellCore.shellSta }); assert(JSON.stringify(safeBrowser.query) === JSON.stringify({ browser_instance: "browser:restore-fixture", url: "https://example.invalid/restore" }), "Browser restore retained authority or unknown fields"); +shellCore.shellState.windows.clear(); +shellCore.shellState.activeWindowId = null; +const hiddenRestoreBefore = restoredBrowserLaunches.length; +shellCore.shellState.homeBrowserState.session = { + root_shell: "home-gui", + browser_context_id: shellCore.shellState.browserContextId, + windows: [ + { target: "browser", hidden: true, query: { url: "https://example.invalid/hidden-leftover" } }, + { target: "browser", hidden: false, active: true, query: { url: "https://example.invalid/visible" } }, + ], +}; +await shellWindows.restoreShellSession(); +assert(shellCore.shellState.windows.size === 1 && restoredBrowserLaunches.length === hiddenRestoreBefore + 1, + "hidden leftover Browser windows must not remount or launch an Engine"); +assert(restoredBrowserLaunches.at(-1).query.url === "https://example.invalid/visible", + "session restore launched a hidden leftover Browser"); +shellCore.shellState.windows.clear(); +shellCore.shellState.activeWindowId = null; const shellMenubar = await import(`../capsules/home-gui/browser/shell-menubar.js?v=${moduleVersion}`); const menuLaunches = []; shellMenubar.bindMenubar({ diff --git a/state.md b/state.md index 58f6c69e2..cf3477934 100644 --- a/state.md +++ b/state.md @@ -1,45 +1,50 @@ # State -## Current remote-services checkpoint, 2026-09-12 - -Reconciled 2026-09-13 UTC. The unpublished `feat/remote-services` candidate is -`4ab99128b5d9491ef0d1c3536954f519aeee0def`, tree -`2ae65c6737fe13986d0a7b50eaf346deae896f71`, with 161 commits beyond fetched -`origin/feat/0.7.1-integration` (`093b51ca`) and none missing from that base. -This identifies the tested source before the documentation reconciliation. -Product changes end at `8444ff51`; `4ab99128` changes the Browser test's baseline -budget to ten seconds while keeping actual viewer reload at five seconds. -The Browser donor `bf846cd8` is included. Its separate dirty updater work retains -its owner and has not entered this candidate. - -The isolated Mac and Linux Runtime binaries remain `50d835ed`; the Mac model -provider and seed Assistant use `8444ff51` components. Installation receipts bind -that mixed set explicitly. A fresh read confirms the Runtime/provider hashes and -available test Homes. Public Runtime retains its September 11 binary and responds -on the public Home route. These checks preserve earlier acceptance; they are not -a new full installed campaign. - -| Surface | Accepted evidence | Remaining proof | -| --- | --- | --- | -| Remote Qwen | Real reply and saved chat; exact retained replay without another dispatch; owner Revoke during a run and original-chat recovery with both identifiers; honest Outcome unknown; keyboard Assistant activation; saved historic failure display; fresh Busy classification | Immediate Busy replay and successful retry after capacity frees; restart during a run; unapproved second-principal denial; complete M01-M06 and AI1 identity/Marketplace acceptance | -| Browser | Bounded local run 92; remote run 100 navigation, media, input and 13 close effects; ordinary open 110 completed on selected Mac Engine `remote-engine-ec54fd5eab67a5c5913513a47a797fc8` with seed Exit and `https://ela.city/` | Actual remote reload and full U9 remain open. Automatic open still fails at profile placement. Attempts 101-103 stopped at automated Home focus; Anders did not reproduce that focus failure. Open 110 left a live page that still needs Home close. | -| Test desktop | Manual helper now uses the native viewport; Anders confirms desktop fit | The viewport correction proves fit only. It does not establish the cause of earlier automated pointer/focus failures. | - -Expired Busy replay preserves the run identity and terminal status after the -provider journal expires. The Runtime's status-only fallback is intentional; -missing expired error details are not a demonstrated defect. Earlier failed -receipts remain historical evidence. Stop and revoke report the backend's -`settlement_unknown` as Outcome unknown; confirmed cancellation is a separate claim. - -The source can enter bounded draft review after documentation and basic gates -pass. Public cutover needs U8, U9 and U10, the selected public configuration, and -approval of the exact candidate. Full J1-J5, Browser B01-B16, model M01-M06, -Marketplace AI1, cold model delivery, signed installation/update and contributor -acceptance remain open under the approved plan. Follow the current Now section in -[TASKS.md](TASKS.md); previous checkpoints below retain their dated scope. - - -## Current public preview and review checkpoint, 2026-09-11 +## Current model delivery checkpoint, 17 September 2026 UTC + +The unpublished working branch is `feat/remote-services` at +`4009015b945abca23a705e7cd3cb83a15afb649d`, committed tree +`907bbf1f7ec5611460a57dd88772e1160d213b4e`. It is 169 commits ahead and +0 behind the last fetched `origin/feat/0.7.1-integration` (`093b51ca`). +The working delta remains uncommitted. These Git identities identify the base; +installed results retain their own binary and patch bindings. + +The signed catalogue contains Qwen and SmolLM2-135M-Instruct Q8_0. Mac and seed +holders retain the small package. Complete Mac-holder delivery was hashed, and +an isolated Mac consumer admitted the package through Marketplace authority. +The installed weights match the published 144,811,072-byte SHA-256. A native +SmolLM2 run completed through the Assistant model contract. That check used a +test helper; the visible Assistant conversation stayed empty. Normal Send, +visible history, reload and full restart reuse remain open on this candidate. +Earlier accepted Qwen receipts keep their original scope. + +The reviewed Linux CPU and multi-model startup delta `e84dd728` is integrated. +The two-order startup, Mac symlink and catalogue identity tests passed. The +managed-Home catalogue propagation repair and its three regressions are present +in source; a fresh installed Home without the diagnostic manual copy remains +unverified. Preparation cancellation and a total deadline across holder attempts +remain open. Honest inference `settlement_unknown` remains a distinct contract +from responsive download cancellation. + +Eight seed-only 64 KiB reads took 18.516 seconds, including 18.513 seconds inside +the Content requests. Seed-local Kubo read the same amount in 0.04 seconds. +This rules out the driver as the main delay; it does not identify the slow +Runtime, Carrier or network stage. Full independent seed delivery remains open. + +The installer audit found catalogue validation after installed-file replacement. +Cloud owns the narrow installer/test repair. The existing Mac-only publication +path can prepare Sash's preview; complete three-platform W1 remains required for +the release. A source review push has its own source gates and exact-candidate +approval; platform release artifacts are not a prerequisite for draft review. + +Whitespace, Home entropy, Browser entropy and required formatting checks are the +current source review gates. Installed and human acceptance retain their separate +scope. Browser is paused during model delivery. Preserve its prior receipts, +source and J4 obligations, plus all J1–J5, M01–M06 and shared-state criteria. +Current execution is in [TASKS.md](TASKS.md); target paths, process ownership and +raw proof identities remain in the private development-loop checkpoint. + +## Published preview and review checkpoint, last verified 11 September 2026 `feat/0.7.1-integration` is the active working/review branch in [draft PR64](https://github.com/Elacity/elastos-runtime/pull/64), based on @@ -69,7 +74,7 @@ Main/dev and contributor PRs retain their prior identities. Full J1 to J5 and D1 to D6 acceptance remains open. The [team report](docs/audits/2026-09-11-team-sync.md) and [contributor review](docs/audits/2026-09-11-contributor-review.md) cover 5 to 11 September. Next implementation: exact Marketplace-to-Assistant handoff -and safe removal, then cold Content/Carrier delivery. Recurring monitoring stays paused. +and safe removal, then cold Content/Carrier delivery. Earlier dated records below retain their original artifacts and verdicts. This checkpoint owns the current public installation status. @@ -961,6 +966,28 @@ complete. Installed proof and the atomic authority cutover remain open. - Principal-root object protection exists for selected Home/runtime state; this does not include Browser VM Chromium profile disks yet. - Product-readiness claims remain gated on target-specific objective audit and matching manual UX evidence; source inclusion does not waive that gate. +- The 0.7.1 Browser U9 journeys remain open. Isolated Mac Home now runs + Runtime `b6d182b7` (PID 50635). Local U9 evidence stays bound to + Runtime `63be1c09`, which completed input, TURN SIGSTOP media cut + (stall 4344 ms, recovery 1810 ms), viewer reload at 1196 ms, official close + with all 13 terminal effects, and a second open (stall 4018 ms, recovery + 1489 ms, reload 1216 ms). The 22:08 local log remains classified as an + unproven interruption: CDP offline produced `cut_ms=5000` and `stall_ms=0`. + The earlier `bd0f4beb` 797 ms local first-frame sample and the remote census + 5340 ms first-frame sample on seed `21388ce6` stay bound to those artifacts. + Remote full input/reload/recovery/reopen on the current grant is not run: + the 00:51 seed dump still reports expired offer + `remote-engine-fdf0eaa2ec5d6ede` and `remote_runtime_binding_required`. The + vm-control settlement fixture including exit-then-reopen-before-cleanup + passed. Live page capacity requires the VM to own that exact page. A later + same-profile VM does not revive an old pending page. Consumer close settles + already-absent Engine ownership and releases viewer ingress. Source now + allocates a fresh display-attach request ID after a terminal failure and + commits preadmitted media only while ingress still owns the page. Those two + U9 lifecycle races have source regressions. The installed `63be1c09` pair + still runs the earlier source. Source also closes the model revoke/create + race and the AI0 Chat/Studio decode plus remote-reply identity boundary. + The four installed Qwen U8 Busy/retry/restart/denial checks stay open. ## Browser Provider Evidence From 1f9797b8d95ddd6c8c17bac7b1aaa6d81057eb59 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 20:59:32 -0500 Subject: [PATCH 340/535] fix(publisher): discover health and bootstrap from distinct source URLs Gateway control serves /api/health. Publisher well-known lives on an explicit public bind, or on the same URL for a legacy operator runtime. --- TASKS.md | 37 ++-- scripts/discover-source-bootstrap-test.py | 177 ++++++++++++++++++ scripts/discover-source-bootstrap.py | 128 +++++++++++++ scripts/publish-release.sh | 55 +----- .../publisher-bootstrap-integrity-smoke.sh | 14 +- state.md | 35 ++-- 6 files changed, 355 insertions(+), 91 deletions(-) create mode 100644 scripts/discover-source-bootstrap-test.py create mode 100644 scripts/discover-source-bootstrap.py diff --git a/TASKS.md b/TASKS.md index dc36669a4..1b1da1cf7 100644 --- a/TASKS.md +++ b/TASKS.md @@ -171,28 +171,21 @@ verification separate from product acceptance and required human review. ## Now -### Model delivery R14 — bounded Sash preview first - -The immediate D1 outcome is a normal Apple Silicon model demo: install the named -candidate, discover both signed model entries, Get/Use SmolLM2, send in Assistant, -see the reply, then reload/restart with saved conversation and payload reuse. -Reuse existing Qwen evidence within its recorded scope. The full R14 requirements -remain below; they follow this bounded preview instead of delaying its first test. - -1. Local owns actual Assistant Send/reply/reload/restart on the admitted model, - durable test-holder ownership and installed managed-Home catalogue sync. - First visible result or exact failure is due within ten minutes of resuming. - Preserve the human Qwen Home and existing admissions. -2. Cloud's catalogue-before-installation repair is integrated. The exact two-file - patch passed all 53 installer tests locally (four retained fixture skips). - Cloud now owns only the publisher shell, native-platform admission helper and - their focused tests for an explicit Apple Silicon canary preview. Default - release admission retains all three platforms and all provenance/trust gates. - Local freezes those packaging files until the reviewed patch returns. -3. Coordinator reviews the exact combined source and artifacts, prepares coherent - commits and a short test guide. Publication for review uses the existing - branch and preserves Browser ancestry. Show the candidate and verification - before named approval to push or expose the preview. Public Home is preserved. +### Model delivery R14 — 0.7.1 on public Home + +The 0.7.1 preview uses the existing public Home at +https://elastos.elacitylabs.com/home/. Ordinary install, Marketplace Get/Use, +Assistant Send/reply, reload and restart remain the demo path. Reuse existing +Qwen evidence within its recorded scope. The full R14 requirements remain below. + +1. Local owns the publisher discovery repair: health from one control coords + file, bootstrap from ELASTOS_SOURCE_PUBLISHER_URL or the legacy same URL. + Preserve human Home, admitted models and existing test Homes. +2. Prepare matching Linux native inputs from this source candidate and verify + the isolated seed before updating public Home. Default release admission + retains all three platforms and provenance gates. +3. Coordinator pushes the frozen `feat/remote-services` candidate to GitHub. + Public Home stays the preview surface. The broader R14 queue remains required after D1: independent full Mac/seed holder reads and demonstrated transfer repairs; responsive preparation cancellation and diff --git a/scripts/discover-source-bootstrap-test.py b/scripts/discover-source-bootstrap-test.py new file mode 100644 index 000000000..f2539d2de --- /dev/null +++ b/scripts/discover-source-bootstrap-test.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +"""Prove publisher discovery keeps gateway health when bootstrap lives on another URL.""" + +import importlib.util +import io +import json +import os +from pathlib import Path +import tempfile +import unittest +import urllib.error + +HELPER = Path(__file__).resolve().with_name("discover-source-bootstrap.py") +spec = importlib.util.spec_from_file_location("discover_source_bootstrap", HELPER) +bootstrap = importlib.util.module_from_spec(spec) +spec.loader.exec_module(bootstrap) + + +def http_404(url): + return urllib.error.HTTPError(url, 404, "Not found", hdrs=None, fp=io.BytesIO(b"Not found")) + + +class DiscoverSourceBootstrapTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix="discover-source-bootstrap-") + self.data = Path(self.temp.name) + self.control = "http://127.0.0.1:60474" + self.public = "http://127.0.0.1:61700" + self.gateway_coords = self.data / "gateway-runtime-coords.json" + self.operator_coords = self.data / "runtime-coords.json" + self.gateway_coords.write_text(json.dumps({ + "api_url": self.control, + "runtime_kind": "gateway", + "attach_secret": "control-secret", + })) + self.operator_coords.write_text(json.dumps({ + "api_url": self.public, + "runtime_kind": "operator", + "attach_secret": "operator-secret", + })) + self.addCleanup(self.temp.cleanup) + self.addCleanup(lambda: os.environ.pop("ELASTOS_SOURCE_PUBLISHER_URL", None)) + os.environ.pop("ELASTOS_SOURCE_PUBLISHER_URL", None) + + def opener(self, pages): + def fetch(url, timeout): + if url not in pages: + raise http_404(url) + payload = pages[url] + if isinstance(payload, Exception): + raise payload + return payload + return fetch + + def test_fresh_gateway_uses_gateway_coords_when_legacy_file_is_absent(self): + self.operator_coords.unlink() + missing = self.data / "runtime-coords.json" + pages = { + self.control + "/api/health": {"status": "ok", "version": "0.7.1-dev"}, + self.control + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "control-ticket", + "node_id": "control-node", + }, + } + self.assertEqual( + bootstrap.select_control_coords(str(missing), self.data), + self.gateway_coords, + ) + result = bootstrap.discover(str(missing), self.data, self.opener(pages)) + self.assertEqual(result, {"version": "0.7.1-dev"}) + + def test_explicit_publisher_url_overrides_leftover_operator_bootstrap(self): + advertised = "http://127.0.0.1:8090" + os.environ["ELASTOS_SOURCE_PUBLISHER_URL"] = advertised + pages = { + self.control + "/api/health": {"status": "ok", "version": "0.7.1-dev"}, + self.public + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "leftover-ticket", + "node_id": "leftover-node", + }, + advertised + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "advertised-ticket", + "node_id": "advertised-node", + }, + } + result = bootstrap.discover(self.gateway_coords, self.data, self.opener(pages)) + self.assertEqual(result, { + "version": "0.7.1-dev", + "ticket": "advertised-ticket", + "node_id": "advertised-node", + "role": "publisher", + }) + + def test_health_survives_when_publisher_bootstrap_is_absent(self): + pages = { + self.control + "/api/health": {"status": "ok", "version": "0.7.1-dev"}, + self.public + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "leftover-ticket", + "node_id": "leftover-node", + }, + } + result = bootstrap.discover(self.gateway_coords, self.data, self.opener(pages)) + self.assertEqual(result, {"version": "0.7.1-dev"}) + self.assertNotIn("ticket", result) + self.assertNotIn("node_id", result) + + def test_legacy_same_url_reads_health_and_bootstrap(self): + self.gateway_coords.unlink() + pages = { + self.public + "/api/health": {"status": "ok", "version": "0.7.0-dev"}, + self.public + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "publisher-ticket", + "node_id": "publisher-node", + }, + } + result = bootstrap.discover(self.operator_coords, self.data, self.opener(pages)) + self.assertEqual(result, { + "version": "0.7.0-dev", + "ticket": "publisher-ticket", + "node_id": "publisher-node", + "role": "publisher", + }) + + def test_publisher_pair_survives_when_health_is_absent(self): + self.gateway_coords.unlink() + pages = { + self.public + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "publisher-ticket", + "node_id": "publisher-node", + }, + } + result = bootstrap.discover(self.operator_coords, self.data, self.opener(pages)) + self.assertEqual(result, { + "ticket": "publisher-ticket", + "node_id": "publisher-node", + "role": "publisher", + }) + self.assertNotIn("version", result) + + def test_runtime_role_and_incomplete_pair_are_rejected(self): + self.gateway_coords.unlink() + pages = { + self.public + "/api/health": {"status": "ok", "version": "0.7.1-dev"}, + self.public + bootstrap.BOOTSTRAP_PATH: { + "schema": "elastos.carrier.bootstrap/v1", + "role": "runtime", + "ticket": "runtime-ticket", + "node_id": "runtime-node", + }, + } + result = bootstrap.discover(self.operator_coords, self.data, self.opener(pages)) + self.assertEqual(result, {"version": "0.7.1-dev"}) + + pages[self.public + bootstrap.BOOTSTRAP_PATH] = { + "schema": "elastos.carrier.bootstrap/v1", + "role": "publisher", + "ticket": "publisher-ticket", + "node_id": "", + } + result = bootstrap.discover(self.operator_coords, self.data, self.opener(pages)) + self.assertEqual(result, {"version": "0.7.1-dev"}) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/discover-source-bootstrap.py b/scripts/discover-source-bootstrap.py new file mode 100644 index 000000000..518397a97 --- /dev/null +++ b/scripts/discover-source-bootstrap.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +"""Resolve trusted-source health version and publisher Carrier bootstrap. + +Health comes from one selected control coords file. Bootstrap comes from +ELASTOS_SOURCE_PUBLISHER_URL, or from that same control URL on a legacy +operator layout. A fresh gateway has only gateway-runtime-coords.json. +""" + +import json +import os +import pathlib +import sys +import urllib.request + +sys.dont_write_bytecode = True + +BOOTSTRAP_PATH = "/.well-known/elastos/carrier-bootstrap.json?role=publisher" + + +def read_coords(path): + try: + return json.loads(pathlib.Path(path).read_text()) + except Exception: + return None + + +def read_api_url(path): + coords = read_coords(path) + if not isinstance(coords, dict): + return None + api = (coords.get("api_url") or "").rstrip("/") + return api or None + + +def select_control_coords(coords_path, data_dir): + explicit = pathlib.Path(coords_path) if coords_path else None + if explicit and explicit.is_file(): + return explicit + if not data_dir: + return None + data = pathlib.Path(data_dir) + gateway = data / "gateway-runtime-coords.json" + if gateway.is_file(): + return gateway + legacy = data / "runtime-coords.json" + if legacy.is_file(): + return legacy + return None + + +def is_gateway_control(path): + if path.name == "gateway-runtime-coords.json": + return True + coords = read_coords(path) + return isinstance(coords, dict) and coords.get("runtime_kind") == "gateway" + + +def fetch_json(url, timeout, opener=None): + if opener is not None: + return opener(url, timeout) + with urllib.request.urlopen(url, timeout=timeout) as resp: + return json.loads(resp.read().decode()) + + +def publisher_pair(data): + if data.get("schema") != "elastos.carrier.bootstrap/v1": + return None + if data.get("role") != "publisher": + return None + ticket = (data.get("ticket") or "").strip() + node_id = (data.get("node_id") or "").strip() + if not ticket or not node_id: + return None + return ticket, node_id + + +def discover(coords_path, data_dir, opener=None): + control = select_control_coords(coords_path, data_dir) + if control is None: + return {} + control_url = read_api_url(control) + explicit_publisher = (os.environ.get("ELASTOS_SOURCE_PUBLISHER_URL") or "").rstrip("/") + if explicit_publisher: + bootstrap_url = explicit_publisher + elif is_gateway_control(control): + bootstrap_url = None + else: + bootstrap_url = control_url + + version = "" + if control_url: + try: + health = fetch_json(control_url + "/api/health", 2, opener) + except Exception: + health = None + if isinstance(health, dict): + version = health.get("version") or "" + + ticket = "" + node_id = "" + if bootstrap_url: + try: + bootstrap = fetch_json(bootstrap_url + BOOTSTRAP_PATH, 5, opener) + except Exception: + bootstrap = None + pair = publisher_pair(bootstrap) if isinstance(bootstrap, dict) else None + if pair: + ticket, node_id = pair + + result = {} + if version: + result["version"] = version + if ticket and node_id: + result["ticket"] = ticket + result["node_id"] = node_id + result["role"] = "publisher" + return result + + +def main(): + print(json.dumps(discover( + os.environ.get("COORDS_PATH", ""), + os.environ.get("DATA_DIR", ""), + ))) + + +if __name__ == "__main__": + main() diff --git a/scripts/publish-release.sh b/scripts/publish-release.sh index ec266190b..dbf4f9b05 100755 --- a/scripts/publish-release.sh +++ b/scripts/publish-release.sh @@ -223,60 +223,13 @@ default_elastos_data_dir() { } discover_source_bootstrap_json() { - local data_dir + local data_dir helper data_dir="$(default_elastos_data_dir)" + helper="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/discover-source-bootstrap.py" DATA_DIR="${data_dir}" \ COORDS_PATH="${ELASTOS_RUNTIME_COORDS_FILE:-${data_dir}/runtime-coords.json}" \ - EXPECTED_VERSION="${ELASTOS_SOURCE_EXPECTED_VERSION:-${VERSION:-}}" \ - python3 - <<'PY' -import json -import os -import pathlib -import urllib.request - -coords_path = pathlib.Path(os.environ["COORDS_PATH"]) -if not coords_path.exists(): - print("{}") - raise SystemExit(0) - -try: - coords = json.loads(coords_path.read_text()) - api = coords["api_url"].rstrip("/") -except Exception: - print("{}") - raise SystemExit(0) - -version = "" -try: - with urllib.request.urlopen(api + "/api/health", timeout=2) as resp: - health = json.loads(resp.read().decode()) - version = health.get("version", "") or "" -except Exception: - version = "" - -try: - with urllib.request.urlopen(api + "/.well-known/elastos/carrier-bootstrap.json?role=publisher", timeout=5) as resp: - bootstrap = json.loads(resp.read().decode()) - if bootstrap.get("schema") != "elastos.carrier.bootstrap/v1": - print("{}") - raise SystemExit(0) - if bootstrap.get("role") != "publisher": - print("{}") - raise SystemExit(0) - ticket = (bootstrap.get("ticket") or "").strip() - node_id = (bootstrap.get("node_id") or "").strip() - if not ticket or not node_id: - print("{}") - raise SystemExit(0) - print(json.dumps({ - "ticket": ticket, - "node_id": node_id, - "role": "publisher", - "version": version, - })) -except Exception: - print("{}") -PY + PYTHONDONTWRITEBYTECODE=1 \ + python3 "$helper" } canonical_publisher_gateway() { diff --git a/scripts/publisher-bootstrap-integrity-smoke.sh b/scripts/publisher-bootstrap-integrity-smoke.sh index 800a5296f..b025f6591 100644 --- a/scripts/publisher-bootstrap-integrity-smoke.sh +++ b/scripts/publisher-bootstrap-integrity-smoke.sh @@ -9,6 +9,7 @@ from pathlib import Path install = Path("scripts/install.sh").read_text() publish = Path("scripts/publish-release.sh").read_text() +discover = Path("scripts/discover-source-bootstrap.py").read_text() required_install = [ 'data.get("role") != "publisher"', @@ -16,10 +17,15 @@ required_install = [ "trusted-source Carrier bootstrap overrides are atomic", ] required_publish = [ - "/.well-known/elastos/carrier-bootstrap.json?role=publisher", - 'bootstrap.get("role") != "publisher"', + "discover-source-bootstrap.py", "trusted-source Carrier bootstrap requires both ELASTOS_SOURCE_CONNECT_TICKET and ELASTOS_PUBLISHER_NODE_ID", ] +required_discover = [ + "/.well-known/elastos/carrier-bootstrap.json?role=publisher", + 'data.get("role") != "publisher"', + 'data.get("schema") != "elastos.carrier.bootstrap/v1"', + '"/api/health"', +] for needle in required_install: if needle not in install: @@ -29,6 +35,10 @@ for needle in required_publish: if needle not in publish: raise SystemExit(f"[publisher-bootstrap-integrity] publish-release.sh missing {needle!r}") +for needle in required_discover: + if needle not in discover: + raise SystemExit(f"[publisher-bootstrap-integrity] discover-source-bootstrap.py missing {needle!r}") + if "keys node-id" in publish: raise SystemExit("[publisher-bootstrap-integrity] publish-release.sh must not pair a ticket with keys node-id fallback") diff --git a/state.md b/state.md index cf3477934..032ec63db 100644 --- a/state.md +++ b/state.md @@ -2,21 +2,22 @@ ## Current model delivery checkpoint, 17 September 2026 UTC -The unpublished working branch is `feat/remote-services` at -`4009015b945abca23a705e7cd3cb83a15afb649d`, committed tree -`907bbf1f7ec5611460a57dd88772e1160d213b4e`. It is 169 commits ahead and -0 behind the last fetched `origin/feat/0.7.1-integration` (`093b51ca`). -The working delta remains uncommitted. These Git identities identify the base; -installed results retain their own binary and patch bindings. +The model-delivery source line is `feat/remote-services`. Last frozen prefix is +`7690aabc6acb4f125b84b7ec0117e3e406898b91`, tree +`e7d8d0e1ac2b343e3104f7eefd1dcfeb3fba6bbc`, 174 ahead and 0 behind the last +fetched `origin/feat/0.7.1-integration` (`093b51ca`). This source includes the +publisher discovery repair. Installed results retain their own binary and +patch bindings. The 0.7.1 preview target is the existing public Home at +https://elastos.elacitylabs.com/home/. GitHub review uses that same branch. Matching Linux native inputs and isolated +seed verification precede the public Home update. Preserve existing Homes. The signed catalogue contains Qwen and SmolLM2-135M-Instruct Q8_0. Mac and seed holders retain the small package. Complete Mac-holder delivery was hashed, and an isolated Mac consumer admitted the package through Marketplace authority. -The installed weights match the published 144,811,072-byte SHA-256. A native -SmolLM2 run completed through the Assistant model contract. That check used a -test helper; the visible Assistant conversation stayed empty. Normal Send, -visible history, reload and full restart reuse remain open on this candidate. -Earlier accepted Qwen receipts keep their original scope. +The installed weights match the published 144,811,072-byte SHA-256. Coordinator +verified ordinary Assistant Send on the isolated consumer Home. Reload and +restart preserved that history. Earlier accepted Qwen receipts keep their +original scope. The reviewed Linux CPU and multi-model startup delta `e84dd728` is integrated. The two-order startup, Mac symlink and catalogue identity tests passed. The @@ -31,11 +32,13 @@ the Content requests. Seed-local Kubo read the same amount in 0.04 seconds. This rules out the driver as the main delay; it does not identify the slow Runtime, Carrier or network stage. Full independent seed delivery remains open. -The installer audit found catalogue validation after installed-file replacement. -Cloud owns the narrow installer/test repair. The existing Mac-only publication -path can prepare Sash's preview; complete three-platform W1 remains required for -the release. A source review push has its own source gates and exact-candidate -approval; platform release artifacts are not a prerequisite for draft review. +The installer catalogue-before-replacement repair is integrated. Publisher +discovery reads health from one selected control coords file. Bootstrap comes +from ELASTOS_SOURCE_PUBLISHER_URL, or from that same control URL on a legacy +operator layout. Matching Linux native inputs are required for this candidate. +Complete three-platform W1 remains required for the release. A source review +push has its own source gates and exact-candidate approval; platform release +artifacts are not a prerequisite for draft review. Whitespace, Home entropy, Browser entropy and required formatting checks are the current source review gates. Installed and human acceptance retain their separate From e9d98742a4055aa7d75d6843de952f7cdadca122 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 21:16:52 -0500 Subject: [PATCH 341/535] chore: remove personal fixture labels and stale status prose --- .../assistant/browser/workspace-transition.js | 2 +- capsules/system/browser/style.css | 2 +- docs/BROWSER_ACCEPTANCE.md | 23 -------- docs/MODEL_PACKAGE_HANDOFF.md | 17 +++--- .../elastos-server/src/api/auth_gateway.rs | 4 +- .../src/api/gateway_model_service.rs | 2 +- .../gateway_tests/collaboration_presence.rs | 10 ++-- .../src/api/gateway_tests/home_system.rs | 56 +++++++++---------- .../src/api/gateway_tests/room.rs | 22 ++++---- elastos/crates/elastos-server/src/chat_cmd.rs | 12 ++-- .../src/collaboration_startup.rs | 2 +- elastos/crates/elastos-server/src/home_cmd.rs | 12 ++-- .../crates/elastos-server/src/identity_cmd.rs | 2 +- .../crates/elastos-server/src/room_service.rs | 20 +++---- .../assistant-convergence-browser-smoke.mjs | 8 +-- .../assistant-workspace-transition.test.mjs | 18 +++--- scripts/browser-qualification-runner.mjs | 3 +- .../browser-webrtc-send-diagnostic.test.py | 13 ++++- scripts/home-agent-shell-smoke.mjs | 8 +-- scripts/home-entropy-check.mjs | 2 +- 20 files changed, 108 insertions(+), 130 deletions(-) diff --git a/capsules/assistant/browser/workspace-transition.js b/capsules/assistant/browser/workspace-transition.js index f095873c8..62b161127 100644 --- a/capsules/assistant/browser/workspace-transition.js +++ b/capsules/assistant/browser/workspace-transition.js @@ -118,7 +118,7 @@ export function migrateLegacyWorkspaces(legacy = {}, canonicalDocument = {}) { result.projects = list(result.projects); result.legacyImports = object(result.legacyImports) ? result.legacyImports : {}; const existingIds = new Set([...result.sessions, ...result.projects].map(record => record.id)); - // The current Sash workspace supplies the active view; every other selection + // The current Home Agent workspace supplies the active view; every other selection // remains in its source snapshot instead of silently replacing that choice. for (const source of ["homeAgent", "assistant", "homeSessionAgent"]) { const raw = legacy[source]; diff --git a/capsules/system/browser/style.css b/capsules/system/browser/style.css index 56fa7f337..789961b79 100644 --- a/capsules/system/browser/style.css +++ b/capsules/system/browser/style.css @@ -1735,7 +1735,7 @@ code { } /* ---- Freeze additions kept over the original sheet ---- - Anders' post-split features (detail view / lock screen / sidebar icons) + Post-split features (detail view / lock screen / sidebar icons) that the original sheet predates. Pinned by home-entropy-check. ---- */ .system-locked { diff --git a/docs/BROWSER_ACCEPTANCE.md b/docs/BROWSER_ACCEPTANCE.md index b918d6afb..76b75f3ed 100644 --- a/docs/BROWSER_ACCEPTANCE.md +++ b/docs/BROWSER_ACCEPTANCE.md @@ -41,29 +41,6 @@ Portable packages can contain compatible platform payloads selected by Runtime. Remote service use invokes an instance on another Runtime. Both paths need proof; one successful remote session does not prove package portability. -## Evidence that sets the initial priorities - -The initial analysis found these repair targets. Its exact source revision is -recorded in the [Browser maturity workstream](../TASKS.md#browser-maturity-workstream). - -- Artifact setup can report success with required Engine artifacts missing. - Home can offer an Engine from configuration or binary presence before launch - readiness is established. -- Continuous wheel input uses a trailing delay that can withhold events until - the gesture stops. Horizontal input also needs a regression case. -- A temporary heartbeat or status request failure can close the display and - trigger page cleanup before reconnection has a chance to recover. -- Shared synchronous provider request handling can delay unrelated session - operations behind an Engine launch. -- Remote Engine setup still needs operator transport machinery, while the - user service-request path handles Exit. Engine needs the same Runtime service - model. - -These findings set the order below. They do not establish the cause of another -person's failure without that installation's diagnostic receipt. The inspected -Brave session failed before an active Browser page; it supplied startup evidence, -not media or usability acceptance evidence. - ## Execution order B01-B16 define release acceptance areas. Current execution slices, owners and diff --git a/docs/MODEL_PACKAGE_HANDOFF.md b/docs/MODEL_PACKAGE_HANDOFF.md index 5d6f232bd..94f9518c7 100644 --- a/docs/MODEL_PACKAGE_HANDOFF.md +++ b/docs/MODEL_PACKAGE_HANDOFF.md @@ -230,13 +230,10 @@ leaves `components.json` alone. Removing the `model_catalog` key from `components.json` and restarting the Runtime withdraws the catalog trust pin and leaves the Kubo pin alone. -## Limits - -The helper and this document are verified against a synthetic closure and a -test-only signing key on two offline Kubo 0.40.1 daemons -(`scripts/model-package-handoff-test.py`). The Runtime side is proved by unit -tests: permanent-snapshot verification and same-CID alias reuse run on Linux, -and the readiness and startup-composer test for a permanent re-pin is gated to -`darwin-arm64` and waits for a Mac run. The real Qwen package, the production -signing key, installation on a target Home and a second-Home reply proof are -operator work and are recorded as evidence when they happen. +## Verification scope + +`scripts/model-package-handoff-test.py` checks block-closure transfer, catalogue +trust and failure handling with isolated Kubo instances and test signing keys. +This operator utility does not replace installed Marketplace acquisition tests. +Current target verification and remaining work belong in [state.md](../state.md) +and [TASKS.md](../TASKS.md). diff --git a/elastos/crates/elastos-server/src/api/auth_gateway.rs b/elastos/crates/elastos-server/src/api/auth_gateway.rs index 150a81dbb..aafe9f1d1 100644 --- a/elastos/crates/elastos-server/src/api/auth_gateway.rs +++ b/elastos/crates/elastos-server/src/api/auth_gateway.rs @@ -8189,7 +8189,7 @@ mod tests { &admission, &ceremony, Some(&response), - Some("Anders"), + Some("Owner"), crate::auth::now_ts(), ) .unwrap() @@ -8201,7 +8201,7 @@ mod tests { ®istered.proof_binding_id, ) .unwrap(); - assert_eq!(principal.display_name, "Anders"); + assert_eq!(principal.display_name, "Owner"); let legacy_record = serde_json::to_value(&principal).unwrap(); assert!(legacy_record.get("initial_profile_display_name").is_none()); assert!( diff --git a/elastos/crates/elastos-server/src/api/gateway_model_service.rs b/elastos/crates/elastos-server/src/api/gateway_model_service.rs index ee4407dd6..21c0eede1 100644 --- a/elastos/crates/elastos-server/src/api/gateway_model_service.rs +++ b/elastos/crates/elastos-server/src/api/gateway_model_service.rs @@ -1571,7 +1571,7 @@ mod tests { "rate_limited" ); assert_eq!( - redact_provider_error("unknown offer 'x' at /Users/anders/models"), + redact_provider_error("unknown offer 'x' at /Users/owner/models"), "offer_unavailable" ); assert_eq!(redact_provider_error("run owner mismatch"), "denied"); diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/collaboration_presence.rs b/elastos/crates/elastos-server/src/api/gateway_tests/collaboration_presence.rs index 5d51fe329..b2e03c089 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/collaboration_presence.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/collaboration_presence.rs @@ -268,8 +268,8 @@ async fn configured_room_poll_renders_signed_profile_names_only() { .await .unwrap(), ); - let authority = passkey_authority_with_name(dir.path(), Some("Anders")); - install_signed_profile(dir.path(), &authority, "Anders Signed"); + let authority = passkey_authority_with_name(dir.path(), Some("Owner")); + install_signed_profile(dir.path(), &authority, "Owner Signed"); let now = crate::auth::now_ts(); let app = gateway_router(state); @@ -366,7 +366,7 @@ async fn configured_room_poll_renders_signed_profile_names_only() { .iter() .find(|object| object["body"].as_str() == Some("local hello")) .expect("local text projected"); - assert_eq!(local_object["sender"].as_str(), Some("Anders Signed")); + assert_eq!(local_object["sender"].as_str(), Some("Owner Signed")); assert_eq!(local_object["sender_profile_verified"], json!(true)); let remote_object = objects .iter() @@ -393,11 +393,11 @@ async fn configured_room_poll_renders_signed_profile_names_only() { assert_eq!(remote_participant["profile_verified"], json!(true)); let local_participant = participants .iter() - .find(|participant| participant["display_name"].as_str() == Some("Anders Signed")) + .find(|participant| participant["display_name"].as_str() == Some("Owner Signed")) .expect("local participant listed"); assert_eq!( local_participant["display_name"].as_str(), - Some("Anders Signed") + Some("Owner Signed") ); assert_eq!(local_participant["profile_verified"], json!(true)); } diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs b/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs index d3a246095..6c2735935 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/home_system.rs @@ -1137,7 +1137,7 @@ async fn test_home_summary_reports_identity_and_launch_targets() { let state = library_test_state(dir.path()).await; let app = gateway_router(state); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let library_token = app_token_for_authority(dir.path(), LIBRARY_CAPSULE_ID, &authority); let public = app .clone() @@ -1254,7 +1254,7 @@ async fn test_home_summary_reports_identity_and_launch_targets() { let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); assert_eq!(payload["authority"]["signed_in"], true); assert!(payload["identity"]["profile"].is_null()); - assert_eq!(payload["identity"]["profile_setup_display_name"], "anders"); + assert_eq!(payload["identity"]["profile_setup_display_name"], "owner"); // Decided under invariant 1: the local device DID reaches exactly one // browser surface, System (asserted below at /api/apps/system/summary). // The Home shell has no consumer for it, so the Home summary strips it. @@ -3579,7 +3579,7 @@ async fn test_home_summary_does_not_turn_conversation_members_into_people_contac let guest = tempfile::tempdir().unwrap(); let state = library_test_state(dir.path()).await; let app = gateway_router(state); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); std::fs::create_dir_all(dir.path().join("config")).unwrap(); std::fs::write( dir.path().join("config/browser-engine-adapter.json"), @@ -3677,7 +3677,7 @@ async fn test_root_only_recovery_export_preserves_later_profile_consent() { let dir = tempfile::tempdir().unwrap(); let _ = elastos_identity::load_or_create_did(dir.path()).unwrap(); let app = gateway_router(wallet_test_state(dir.path()).await); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let principal = crate::auth::load_principal_for_proof_binding(dir.path(), &authority.proof_binding_id) .unwrap(); @@ -3763,7 +3763,7 @@ async fn test_root_only_recovery_export_preserves_later_profile_consent() { .uri("/api/apps/people/profile") .header("x-elastos-home-token", authority.people_token.as_str()) .header(CONTENT_TYPE, "application/json") - .body(Body::from(r#"{"display_name":"Anders"}"#)) + .body(Body::from(r#"{"display_name":"Owner"}"#)) .unwrap(), ) .await @@ -3787,7 +3787,7 @@ async fn test_root_only_recovery_export_preserves_later_profile_consent() { .unwrap(); let payload: Value = serde_json::from_slice(&body).unwrap(); assert_eq!(payload["identity"]["profile_readiness"]["status"], "ready"); - assert_eq!(payload["identity"]["profile"]["display_name"], "Anders"); + assert_eq!(payload["identity"]["profile"]["display_name"], "Owner"); } fn assert_recovery_readiness_projection(payload: &Value, status: &str, path: &str) { @@ -3947,7 +3947,7 @@ async fn test_recovery_readiness_does_not_claim_profile_coverage() { let dir = tempfile::tempdir().unwrap(); let _ = elastos_identity::load_or_create_did(dir.path()).unwrap(); let app = gateway_router(wallet_test_state(dir.path()).await); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let before_summary = file_snapshot(dir.path()); for (path, token, origin) in [ @@ -3994,11 +3994,11 @@ async fn test_recovery_readiness_does_not_claim_profile_coverage() { "/api/apps/people/profile", authority.people_token.as_str(), "null", - json!({ "display_name": "Anders" }), + json!({ "display_name": "Owner" }), ) .await; assert_eq!(status, StatusCode::OK); - assert_eq!(payload["profile"]["display_name"], "Anders"); + assert_eq!(payload["profile"]["display_name"], "Owner"); assert_eq!(payload["profile_readiness"]["status"], "ready"); } @@ -4007,7 +4007,7 @@ async fn test_recovery_readiness_projects_unavailable_and_first_profile_fails_cl let dir = tempfile::tempdir().unwrap(); let _ = elastos_identity::load_or_create_did(dir.path()).unwrap(); let app = gateway_router(wallet_test_state(dir.path()).await); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let mut protection = crate::auth::store_test_principal_root_protection(dir.path(), &authority.principal_id); protection.protectors.clear(); @@ -4043,7 +4043,7 @@ async fn test_recovery_readiness_projects_unavailable_and_first_profile_fails_cl .uri("/api/apps/people/profile") .header("x-elastos-home-token", authority.people_token.as_str()) .header(CONTENT_TYPE, "application/json") - .body(Body::from(r#"{"display_name":"Anders"}"#)) + .body(Body::from(r#"{"display_name":"Owner"}"#)) .unwrap(), ) .await @@ -4070,7 +4070,7 @@ async fn test_recovery_readiness_projects_unavailable_and_first_profile_fails_cl async fn test_people_invite_create_route_is_absent_and_read_only() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); crate::auth::store_test_principal_root_protection(dir.path(), &authority.principal_id); let profile = app .clone() @@ -4080,7 +4080,7 @@ async fn test_people_invite_create_route_is_absent_and_read_only() { .uri("/api/apps/people/profile") .header("x-elastos-home-token", authority.people_token.as_str()) .header(CONTENT_TYPE, "application/json") - .body(Body::from(r#"{"display_name":"Anders"}"#)) + .body(Body::from(r#"{"display_name":"Owner"}"#)) .unwrap(), ) .await @@ -4108,7 +4108,7 @@ async fn test_people_invite_create_route_is_absent_and_read_only() { async fn test_people_contact_remove_requires_profile_contact_authority() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let response = app .oneshot( @@ -4958,7 +4958,7 @@ async fn test_system_summary_reports_identity_and_app_id() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let resp = app .clone() .oneshot( @@ -4988,7 +4988,7 @@ async fn test_system_summary_reports_identity_and_app_id() { .unwrap(); let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); assert!(payload["identity"]["profile"].is_null()); - assert_eq!(payload["identity"]["profile_setup_display_name"], "anders"); + assert_eq!(payload["identity"]["profile_setup_display_name"], "owner"); assert!(payload["identity"]["device_did"].is_string()); assert_eq!(payload["home"]["id"], "home"); assert_eq!(payload["home"]["route"], "/home/"); @@ -5040,7 +5040,7 @@ async fn test_system_summary_reports_trusted_source_update_policy() { .unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_name(dir.path(), Some("anders")); + let authority = passkey_authority_with_name(dir.path(), Some("owner")); let resp = app .oneshot( test_browser_request("localhost:61180", "null") @@ -5235,7 +5235,7 @@ async fn test_removed_system_identity_mutations_cannot_succeed_or_mutate_state() .method("POST") .uri(uri) .header(CONTENT_TYPE, "application/json") - .body(Body::from(r#"{"display_name":"anders"}"#)) + .body(Body::from(r#"{"display_name":"owner"}"#)) .unwrap(), ) .await @@ -5275,7 +5275,7 @@ async fn test_people_profile_update_rejects_proofless_launch_token() { .header("origin", "null") .header("x-elastos-home-token", people_app_token(dir.path())) .header(CONTENT_TYPE, "application/json") - .body(Body::from(r#"{"display_name":"anders"}"#)) + .body(Body::from(r#"{"display_name":"owner"}"#)) .unwrap(), ) .await @@ -5297,7 +5297,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip let bus = Arc::new(TokioMutex::new(FakePeerBus::default())); let _runtime = start_fake_runtime(dir.path(), bus, "principal-handle-peer").await; let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_name(dir.path(), Some("Anders")); + let authority = passkey_authority_with_name(dir.path(), Some("Owner")); crate::auth::store_test_principal_root_protection(dir.path(), &authority.principal_id); let summary = app @@ -5318,7 +5318,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip .await .unwrap(); let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); - assert_eq!(payload["identity"]["profile_setup_display_name"], "Anders"); + assert_eq!(payload["identity"]["profile_setup_display_name"], "Owner"); assert!(payload["identity"]["profile"].is_null()); assert_eq!( payload["identity"]["profile_readiness"], @@ -5338,7 +5338,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip .header("origin", "null") .header("x-elastos-home-token", authority.people_token.as_str()) .header(CONTENT_TYPE, "application/json") - .body(Body::from(r#"{"display_name":"Anders Admin"}"#)) + .body(Body::from(r#"{"display_name":"Owner Admin"}"#)) .unwrap(), ) .await @@ -5349,7 +5349,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip .unwrap(); let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); assert_eq!(payload["profile"]["schema"], "elastos.profile-summary/v1"); - assert_eq!(payload["profile"]["display_name"], "Anders Admin"); + assert_eq!(payload["profile"]["display_name"], "Owner Admin"); assert_eq!(payload["profile_readiness"]["status"], "ready"); assert!(payload["profile"]["handle"].is_null()); assert!(payload["profile"].get("profile_did").is_none()); @@ -5366,7 +5366,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip let principal = crate::auth::load_principal_for_proof_binding(dir.path(), &authority.proof_binding_id) .unwrap(); - assert_eq!(principal.display_name, "Anders"); + assert_eq!(principal.display_name, "Owner"); let summary = app .clone() @@ -5388,7 +5388,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); assert_eq!( payload["identity"]["profile"]["display_name"], - "Anders Admin" + "Owner Admin" ); assert_eq!(payload["identity"]["profile_readiness"]["status"], "ready"); assert!(payload["identity"]["profile_setup_display_name"].is_null()); @@ -5442,7 +5442,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip .await .unwrap(); let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); - assert_eq!(payload["display_name"], "Anders Admin"); + assert_eq!(payload["display_name"], "Owner Admin"); let restarted_app = gateway_router(test_state(dir.path())); let (status, payload) = home_test_get_json( @@ -5456,7 +5456,7 @@ async fn test_people_profile_update_creates_signed_profile_under_passkey_princip assert_eq!(payload["identity"]["profile_readiness"]["status"], "ready"); assert_eq!( payload["identity"]["profile"]["display_name"], - "Anders Admin" + "Owner Admin" ); } @@ -5466,7 +5466,7 @@ async fn test_people_profile_update_uses_people_launch_token() { let bus = Arc::new(TokioMutex::new(FakePeerBus::default())); let _runtime = start_fake_runtime(dir.path(), bus, "people-profile-peer").await; let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_name(dir.path(), Some("Anders")); + let authority = passkey_authority_with_name(dir.path(), Some("Owner")); crate::auth::store_test_principal_root_protection(dir.path(), &authority.principal_id); let update = app diff --git a/elastos/crates/elastos-server/src/api/gateway_tests/room.rs b/elastos/crates/elastos-server/src/api/gateway_tests/room.rs index b0701cb91..6a3bfa931 100644 --- a/elastos/crates/elastos-server/src/api/gateway_tests/room.rs +++ b/elastos/crates/elastos-server/src/api/gateway_tests/room.rs @@ -283,7 +283,7 @@ async fn test_chat_room_summary_is_available_without_shell_launch_token() { async fn test_chat_room_session_start_connects_open_room_local_runtime() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let launch = app .clone() @@ -323,14 +323,14 @@ async fn test_chat_room_session_start_connects_open_room_local_runtime() { assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); let payload: serde_json::Value = serde_json::from_slice(&body).unwrap(); assert_eq!(payload["status"], "connected"); - assert_eq!(payload["display_name"], "anders"); + assert_eq!(payload["display_name"], "owner"); } #[tokio::test] async fn test_chat_room_join_link_create_returns_elastos_join_object() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let profile = load_profile_for_authority(dir.path(), &authority); let launch = app @@ -417,7 +417,7 @@ async fn test_chat_room_join_link_create_returns_elastos_join_object() { async fn test_chat_room_join_link_create_rejects_caller_supplied_issuer_gateway() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let launch = app .clone() @@ -479,7 +479,7 @@ async fn test_chat_room_session_start_requires_active_local_member_for_seeded_ro ) .unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let launch = app .clone() @@ -527,7 +527,7 @@ async fn test_chat_room_session_start_requires_active_local_member_for_seeded_ro #[tokio::test] async fn test_chat_room_session_start_connects_active_local_member() { let dir = tempfile::tempdir().unwrap(); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let profile = load_profile_for_authority(dir.path(), &authority); crate::room_service::seed_room_owner( dir.path(), @@ -579,7 +579,7 @@ async fn test_chat_room_session_start_connects_active_local_member() { assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); assert!(cookie.starts_with("room-session=")); assert_eq!(payload["status"], "connected"); - assert_eq!(payload["display_name"], "anders"); + assert_eq!(payload["display_name"], "owner"); assert!(payload["poll"]["participants"] .as_array() .unwrap() @@ -627,7 +627,7 @@ async fn test_chat_room_configured_send_uses_signed_home_authority_and_scoped_po .into_iter() .filter(|object| object.get("collaboration_scope").is_none()) .count(); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let port = crate::collaboration_product::test_chat_product_port( dir.path(), "route-network", @@ -1129,7 +1129,7 @@ async fn configured_chat_rejects_every_legacy_control_and_guest_route_before_mut #[tokio::test] async fn unconfigured_chat_room_access_policy_uses_strict_post_guard_decoding() { let dir = tempfile::tempdir().unwrap(); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let profile = load_profile_for_authority(dir.path(), &authority); crate::room_service::seed_room_owner( dir.path(), @@ -1321,7 +1321,7 @@ async fn test_chat_room_shell_can_kick_guest_without_exposing_session_token() { async fn test_chat_room_cookie_auth_prefers_home_room_session_over_browser_session() { let dir = tempfile::tempdir().unwrap(); let app = gateway_router(test_state(dir.path())); - let authority = passkey_authority_with_profile(dir.path(), "anders"); + let authority = passkey_authority_with_profile(dir.path(), "owner"); let launch = app .clone() @@ -1412,7 +1412,7 @@ async fn test_chat_room_cookie_auth_prefers_home_room_session_over_browser_sessi let objects = payload["objects"].as_array().cloned().unwrap_or_default(); assert!(objects.iter().any(|object| { object["kind"].as_str() == Some("text") - && object["sender"].as_str() == Some("anders") + && object["sender"].as_str() == Some("owner") && object["body"].as_str() == Some("home identity wins") })); } diff --git a/elastos/crates/elastos-server/src/chat_cmd.rs b/elastos/crates/elastos-server/src/chat_cmd.rs index a7b17acbc..36614605b 100644 --- a/elastos/crates/elastos-server/src/chat_cmd.rs +++ b/elastos/crates/elastos-server/src/chat_cmd.rs @@ -2330,10 +2330,10 @@ mod tests { let output = Arc::new(Mutex::new(Vec::new())); let ui = ChatTerminalUi::buffer(Arc::clone(&output)); - report_chat_send_outcome(ChatSendOutcome::Delivered, "anders", "hello", Some(&ui)); + report_chat_send_outcome(ChatSendOutcome::Delivered, "owner", "hello", Some(&ui)); let rendered = String::from_utf8(output.lock().unwrap().clone()).unwrap(); - assert!(rendered.contains(" hello")); + assert!(rendered.contains(" hello")); } #[test] @@ -2341,10 +2341,10 @@ mod tests { let output = Arc::new(Mutex::new(Vec::new())); let ui = ChatTerminalUi::buffer(Arc::clone(&output)); - report_chat_send_outcome(ChatSendOutcome::LocalOnly, "anders", "hello", Some(&ui)); + report_chat_send_outcome(ChatSendOutcome::LocalOnly, "owner", "hello", Some(&ui)); let rendered = String::from_utf8(output.lock().unwrap().clone()).unwrap(); - assert!(rendered.contains(" hello")); + assert!(rendered.contains(" hello")); assert!(rendered.contains("message stayed local")); } @@ -2370,7 +2370,7 @@ mod tests { fn incoming_message_from_same_nick_but_different_did_is_rendered() { let msg = serde_json::json!({ "sender_id": "did:key:peer", - "sender_nick": "anders", + "sender_nick": "owner", "content": "hello", }); assert!(should_render_incoming_message( @@ -2385,7 +2385,7 @@ mod tests { let msg = serde_json::json!({ "sender_id": "did:key:self", "sender_session_id": "session-self", - "sender_nick": "anders", + "sender_nick": "owner", "content": "hello", }); assert!(!should_render_incoming_message( diff --git a/elastos/crates/elastos-server/src/collaboration_startup.rs b/elastos/crates/elastos-server/src/collaboration_startup.rs index 0142e7493..db9f66764 100644 --- a/elastos/crates/elastos-server/src/collaboration_startup.rs +++ b/elastos/crates/elastos-server/src/collaboration_startup.rs @@ -747,7 +747,7 @@ mod tests { #[test] fn bounded_presence_projection_diagnostic_flattens_multiline_errors() { - let raw = "presence projection failed\n/Users/anders/secret\nretry\tlater"; + let raw = "presence projection failed\n/Users/owner/secret\nretry\tlater"; let bounded = bounded_presence_projection_diagnostic(&raw); assert_eq!(bounded, "presence projection failed retry later"); assert!(!bounded.contains('\n')); diff --git a/elastos/crates/elastos-server/src/home_cmd.rs b/elastos/crates/elastos-server/src/home_cmd.rs index 548b2a965..cb9de6d53 100644 --- a/elastos/crates/elastos-server/src/home_cmd.rs +++ b/elastos/crates/elastos-server/src/home_cmd.rs @@ -4528,13 +4528,13 @@ mod tests { }, "identity": { "profile": { - "display_name": "Anders", - "handle": "person:anders" + "display_name": "Owner", + "handle": "person:owner" } }, "authority": { "signed_in": true, - "principal_id": "person:anders" + "principal_id": "person:owner" }, "active_shell": { "schema": "elastos.home.active-shell/v1", @@ -4545,7 +4545,7 @@ mod tests { "contact_count": 1, "contacts": [{ "contact_id": "contact-a", - "display_name": "Anders", + "display_name": "Owner", "relationship": "accepted", "route": "elastos://peer/contact-a", "can_message": true @@ -4607,11 +4607,11 @@ mod tests { .as_ref() .unwrap() .display_name, - "Anders" + "Owner" ); assert_eq!( projection.authority.as_ref().unwrap().principal_id, - "person:anders" + "person:owner" ); assert!(projection.authority.as_ref().unwrap().signed_in); assert_eq!( diff --git a/elastos/crates/elastos-server/src/identity_cmd.rs b/elastos/crates/elastos-server/src/identity_cmd.rs index 73b35d215..a221b2d94 100644 --- a/elastos/crates/elastos-server/src/identity_cmd.rs +++ b/elastos/crates/elastos-server/src/identity_cmd.rs @@ -147,7 +147,7 @@ mod tests { #[test] fn nickname_validation_accepts_simple_value() { - assert!(elastos_identity::validate_nickname("anders").is_ok()); + assert!(elastos_identity::validate_nickname("owner").is_ok()); } #[test] diff --git a/elastos/crates/elastos-server/src/room_service.rs b/elastos/crates/elastos-server/src/room_service.rs index 0fcca5987..13ef3b09a 100644 --- a/elastos/crates/elastos-server/src/room_service.rs +++ b/elastos/crates/elastos-server/src/room_service.rs @@ -6536,15 +6536,15 @@ mod tests { let first = start_local_runtime_session(tmp.path(), &did, "Local runtime", "ElastOS shell") .unwrap(); let second = - start_local_runtime_session(tmp.path(), &did, "anders", "ElastOS shell").unwrap(); + start_local_runtime_session(tmp.path(), &did, "owner", "ElastOS shell").unwrap(); assert_eq!(first.token, second.token); - assert_eq!(second.display_name, "anders"); + assert_eq!(second.display_name, "owner"); let summary = load_summary(tmp.path()).unwrap(); assert_eq!(summary.active_session_count, 1); assert_eq!(summary.active_participants.len(), 1); - assert_eq!(summary.active_participants[0].display_name, "anders"); + assert_eq!(summary.active_participants[0].display_name, "owner"); assert_eq!( summary.active_participants[0].member_did.as_deref(), Some(did.as_str()) @@ -6557,7 +6557,7 @@ mod tests { let (_, did) = elastos_identity::load_or_create_did(tmp.path()).unwrap(); let local = - start_local_runtime_session(tmp.path(), &did, "anders", "ElastOS shell").unwrap(); + start_local_runtime_session(tmp.path(), &did, "owner", "ElastOS shell").unwrap(); let _ = append_object(tmp.path(), &local.token, "hello from shell").unwrap(); let request = @@ -6575,7 +6575,7 @@ mod tests { assert!(poll .participants .iter() - .any(|participant| participant.display_name == "anders" + .any(|participant| participant.display_name == "owner" && !participant.is_current_session)); assert!(poll .objects @@ -6590,16 +6590,14 @@ mod tests { let (_, did) = elastos_identity::load_or_create_did(tmp.path()).unwrap(); let local = - start_local_runtime_session(tmp.path(), &did, "anders", "ElastOS shell").unwrap(); + start_local_runtime_session(tmp.path(), &did, "owner", "ElastOS shell").unwrap(); let sent = append_object(tmp.path(), &local.token, "hello from shell").unwrap(); assert!(sent.from_current_session); let poll = room_poll(tmp.path(), &local.token, 0).unwrap(); - assert!(poll - .participants - .iter() - .any(|participant| participant.display_name == "anders" - && participant.is_current_session)); + assert!(poll.participants.iter().any( + |participant| participant.display_name == "owner" && participant.is_current_session + )); assert!(poll .objects .iter() diff --git a/scripts/assistant-convergence-browser-smoke.mjs b/scripts/assistant-convergence-browser-smoke.mjs index 7e3e9b4e8..bad08b3cc 100644 --- a/scripts/assistant-convergence-browser-smoke.mjs +++ b/scripts/assistant-convergence-browser-smoke.mjs @@ -32,8 +32,8 @@ const legacy = { document: { v: 1, activeSessionId: "collision", sessionMode: "chat", liveOfferId: offerId, selectedModelCid: cid, - sessions: [{ id: "collision", title: "Sash imported conversation", messages: [ - { id: "message-0", role: "user", text: "Sash history survives" }, + sessions: [{ id: "collision", title: "Home Agent imported conversation", messages: [ + { id: "message-0", role: "user", text: "Home Agent history survives" }, { id: "grant-0", role: "grant", text: "Historical grant record survives", decision: "approved" }, { id: "tool-0", role: "tool", text: "Historical tool result survives" }, { id: "system-0", role: "system", text: "Historical system context survives" }, @@ -172,7 +172,7 @@ try { assertComplete(stored.document); assert.equal(await frame.locator("#agent-harness-sidebar").count(), 1); assert.equal(await frame.locator("#assistant-sidebar").count(), 0); - for (const title of ["Assistant complete history", "Sash imported conversation", "Home session imported conversation", + for (const title of ["Assistant complete history", "Home Agent imported conversation", "Home session imported conversation", "Assistant saved draft", "Home Agent saved draft", "Home saved draft"]) { assert.equal(await frame.locator(".agent-harness-session-btn").filter({ hasText: title }).count(), 1, title); } @@ -181,7 +181,7 @@ try { await frame.locator("#agent-harness-stream-column").getByText("END-63", {exact:false}).first().waitFor({state:"attached"}); const transcript = await frame.locator("#agent-harness-stream-column").textContent(); for (let index = 0; index < 64; index++) assert.ok(transcript.includes(`END-${index}`), `Missing rendered message ${index}`); - await choose(frame, "Sash imported conversation"); + await choose(frame, "Home Agent imported conversation"); for (const role of ["grant", "tool", "system"]) { await frame.locator(`#agent-harness-stream-column .agent-msg-${role}`).getByText(`Historical ${role}`, {exact:false}).waitFor({state:"attached"}); } diff --git a/scripts/assistant-workspace-transition.test.mjs b/scripts/assistant-workspace-transition.test.mjs index e2fd152a8..5cad52c33 100644 --- a/scripts/assistant-workspace-transition.test.mjs +++ b/scripts/assistant-workspace-transition.test.mjs @@ -14,7 +14,7 @@ function legacyFixture() { v: 1, activeSessionId: "same", liveOfferId: "chosen", selectedModelCid: cid, projects: [{ id: "project", title: "Project", extra: true }], composerDraft: { text: "Home Agent draft", parts: [{ id: "attachment", text, uri: "localhost://Users/user/notes", authority: "untrusted_content", extra: "retain" }] }, - sessions: [{ id: "same", projectId: "project", title: "Sash chat", archived: true, branchId: "branch", messages: [ + sessions: [{ id: "same", projectId: "project", title: "Home Agent chat", archived: true, branchId: "branch", messages: [ { id: "m", branchId: "branch", role: "user", text, parts: [{ uri: "localhost://notes", text, extra: true }] }, { id: "m2", parentId: "m", branchId: "branch", role: "grant", summary: "Pending", args: { exact: true }, requestId: "original-grant-request" }, ], lastTurn: { providerRunId: "original-run", createRequestId: "original-request", turnId: "turn", state: "settlement_unknown" } }], @@ -41,14 +41,14 @@ test("imports all stores, full history, unknown fields and each draft without mu assert.equal(assistant.messages[3].role, "tool"); assert.equal(assistant.messages[63].run_id, "run-63"); assert.equal(assistant.messages[63].futureMessage, 63); - const sash = doc.sessions.find(s => s.legacyOrigin.source === "homeAgent" && s.legacyOrigin.id === "same"); - assert.equal(sash.archived, true); - assert.equal(sash.messages[1].role, "grant"); - assert.equal(sash.messages[1].requestId, "original-grant-request"); - assert.deepEqual(sash.lastTurn, { ...legacy.homeAgent.document.sessions[0].lastTurn, actorCapsule: "home-agent" }); - assert.equal(sash.messages[1].parentId, sash.messages[0].id); - assert.equal(sash.messages[1].branchId, sash.branchId); - assert.equal(doc.projects.find(p => p.id === sash.projectId).extra, true); + const homeAgentSession = doc.sessions.find(s => s.legacyOrigin.source === "homeAgent" && s.legacyOrigin.id === "same"); + assert.equal(homeAgentSession.archived, true); + assert.equal(homeAgentSession.messages[1].role, "grant"); + assert.equal(homeAgentSession.messages[1].requestId, "original-grant-request"); + assert.deepEqual(homeAgentSession.lastTurn, { ...legacy.homeAgent.document.sessions[0].lastTurn, actorCapsule: "home-agent" }); + assert.equal(homeAgentSession.messages[1].parentId, homeAgentSession.messages[0].id); + assert.equal(homeAgentSession.messages[1].branchId, homeAgentSession.branchId); + assert.equal(doc.projects.find(p => p.id === homeAgentSession.projectId).extra, true); assert.equal(new Set(doc.sessions.map(s => s.id)).size, doc.sessions.length); assert.equal(new Set(doc.projects.map(p => p.id)).size, doc.projects.length); const drafts = doc.sessions.filter(s => s.composerDraft); diff --git a/scripts/browser-qualification-runner.mjs b/scripts/browser-qualification-runner.mjs index 4beb0bc49..400439f12 100644 --- a/scripts/browser-qualification-runner.mjs +++ b/scripts/browser-qualification-runner.mjs @@ -152,8 +152,7 @@ async function resources(roots, tracked, child = {}) { cpu_percent: rows.reduce((n, p) => n + p.cpu_percent, 0) }; } export function journeyEnvironment(plan, index) { - // Matches the coordinator's proven run-installed-journey.py. Explicit values - // keep unrelated inherited Home test modes out of this one journey. + // Explicit values keep inherited Home test modes out of this journey. const env = { ...process.env }; for (const key of Object.keys(env)) if (key.startsWith("HOME_VIRTUAL_AUTH_")) delete env[key]; const p = plan.runtime; diff --git a/scripts/build/browser-webrtc-send-diagnostic.test.py b/scripts/build/browser-webrtc-send-diagnostic.test.py index 18a4a5d43..1458ca873 100644 --- a/scripts/build/browser-webrtc-send-diagnostic.test.py +++ b/scripts/build/browser-webrtc-send-diagnostic.test.py @@ -3,6 +3,7 @@ import hashlib import json +import os from pathlib import Path import runpy import unittest @@ -10,7 +11,13 @@ HELPER = runpy.run_path(str(Path(__file__).with_name("browser-webrtc-send-diagnostic.py"))) -DUMPED = Path("/Users/anders/Code/elastos-runtime/.git/browser-analysis/u9-r8-gstwebrtc-app-76967c5b.py") + + +def dumped_source(): + path = os.environ.get("ELASTOS_BROWSER_GST_SOURCE") + if not path: + raise RuntimeError("Set ELASTOS_BROWSER_GST_SOURCE to the pinned gstwebrtc_app.py fixture") + return Path(path).read_text() class Pad: @@ -64,11 +71,11 @@ def get_size(self): class TestPatch(unittest.TestCase): def test_dumped_image_source_matches_pin(self): - text = DUMPED.read_text() + text = dumped_source() self.assertEqual(hashlib.sha256(text.encode()).hexdigest(), HELPER["BASE_SHA256"]) def test_patch_is_idempotent_on_dumped_source(self): - text = DUMPED.read_text() + text = dumped_source() once = HELPER["patch_source"](text) twice = HELPER["patch_source"](once) self.assertEqual(once, twice) diff --git a/scripts/home-agent-shell-smoke.mjs b/scripts/home-agent-shell-smoke.mjs index d2fcacec9..101b4e072 100644 --- a/scripts/home-agent-shell-smoke.mjs +++ b/scripts/home-agent-shell-smoke.mjs @@ -1,7 +1,7 @@ #!/usr/bin/env node -/* Canonical Assistant Sash capsule gate. +/* Canonical Assistant capsule gate. - Encodes what the capsule owes the Runtime and Home GUI, in Anders' terms: + Encodes what the capsule owes the Runtime and Home GUI, through these contracts: - inference runs only through the typed model contract (offers_list, runs_create with a typed input, runs_events by after_sequence, runs_cancel); no offer is named in source, no ping, no mock provider, no mock reply, @@ -169,7 +169,7 @@ assert.ok(!indexHtml.includes('data-sidebar-nav="studio"'), "Studio uses the sha for (const mode of ["chat", "build", "studio"]) { assert.ok(indexHtml.includes(`data-assistant-mode="${mode}"`), `${mode} is a canonical Assistant mode`); } -assert.equal((indexHtml.match(/id="agent-harness-sidebar"/g) || []).length, 1, "one Sash sidebar"); +assert.equal((indexHtml.match(/id="agent-harness-sidebar"/g) || []).length, 1, "one Assistant sidebar"); const modes = read("capsules/assistant/browser/assistant-modes.js"); assert.ok(modes.includes('from "./assistant.js"') && modes.includes("studioOnly: true"), "Studio uses the typed controller"); for (const theatre of [ @@ -482,4 +482,4 @@ assert.throws( assert.throws(() => contract.applyRunEventsPage({ events: "nope" }, 0), /malformed/); assert.throws(() => contract.applyRunEventsPage(null, 0), /malformed/); -console.log("canonical Assistant Sash shell smoke: ok"); +console.log("canonical Assistant shell smoke: ok"); diff --git a/scripts/home-entropy-check.mjs b/scripts/home-entropy-check.mjs index f35485529..2c677ba33 100755 --- a/scripts/home-entropy-check.mjs +++ b/scripts/home-entropy-check.mjs @@ -887,7 +887,7 @@ assert( assistantIndex.includes('') && assistantEntry.includes('from "./assistant-modes.js"') && assistantModes.includes('from "./assistant.js"') && assistantModes.includes("studioOnly: true"), - "Assistant boots the Sash shell and uses the typed controller for Studio", + "Assistant boots the canonical shell and uses the typed controller for Studio", ); assert( assistantController.includes('from "./model-contract.js"'), From 678f1de9b700b1dbcd8fd4765bd1a217a349cb1d Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 21:25:56 -0500 Subject: [PATCH 342/535] fix(runtime): use the native KVM ioctl request type --- elastos/crates/elastos-crosvm/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/elastos/crates/elastos-crosvm/src/lib.rs b/elastos/crates/elastos-crosvm/src/lib.rs index bba18018c..466076119 100644 --- a/elastos/crates/elastos-crosvm/src/lib.rs +++ b/elastos/crates/elastos-crosvm/src/lib.rs @@ -65,7 +65,7 @@ fn kvm_api_available(path: &std::path::Path) -> bool { // Linux UAPI: _IO(KVMIO, 0x00), where KVMIO is 0xAE. This read-only ioctl // takes no third argument and returns the API version; it creates no VM. // Version 12 is the stable API required by KVM userspace implementations. - unsafe { libc::ioctl(device.as_raw_fd(), 0xAE00 as libc::c_ulong) == 12 } + unsafe { libc::ioctl(device.as_raw_fd(), 0xAE00) == 12 } } #[cfg(test)] From 103419760fb3fff1caaafb1b20aed037866d75cf Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 21:46:25 -0500 Subject: [PATCH 343/535] fix(runtime): preserve atomic capsule admission on musl --- .../src/api/capsule_inventory/preparation/storage.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/storage.rs b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/storage.rs index b45ab49a8..41f73b28d 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/storage.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation/storage.rs @@ -420,7 +420,8 @@ impl Inventory { self.stage(false)?.check()?; #[cfg(target_os = "linux")] let result = unsafe { - libc::renameat2( + libc::syscall( + libc::SYS_renameat2, self.dir.as_raw_fd(), c"stage".as_ptr(), self.dir.as_raw_fd(), From 0724479702a33d043c6a9527f147aae783a4c370 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Wed, 16 Sep 2026 23:09:32 -0500 Subject: [PATCH 344/535] fix(ipfs): generate multipart boundaries across Unix targets --- capsules/ipfs-provider/src/directory_hash.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/capsules/ipfs-provider/src/directory_hash.rs b/capsules/ipfs-provider/src/directory_hash.rs index d854be583..b4ec099a6 100644 --- a/capsules/ipfs-provider/src/directory_hash.rs +++ b/capsules/ipfs-provider/src/directory_hash.rs @@ -385,10 +385,10 @@ struct Multipart<'a> { } fn new_boundary() -> io::Result { + // The kernel CSPRNG through the standard device: the same bytes on every + // Unix libc, where getentropy is missing from musl's declarations. let mut random = [0u8; 32]; - if unsafe { libc::getentropy(random.as_mut_ptr().cast(), random.len()) } != 0 { - return Err(io::Error::last_os_error()); - } + File::open("/dev/urandom")?.read_exact(&mut random)?; let mut boundary = String::with_capacity(64); const HEX: &[u8] = b"0123456789abcdef"; for byte in random { From fdf529add0fbaea73e3568fdd906478f36fe17f9 Mon Sep 17 00:00:00 2001 From: cloud Date: Thu, 17 Sep 2026 04:42:12 +0000 Subject: [PATCH 345/535] feat(model): compact model detail copy from one phase table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shared model presentation derives one phase from the Runtime projection and reads its copy from a table. The full view keeps every established line, so System renders unchanged. The compact detail that Marketplace embeds speaks about this device and the Get action: "Not on this device yet" with a Get button before the first acquisition, "Getting the model…" while the Runtime prepares files, "Removed from this device", "Stopped before finishing" and "Not finished in time" after the Runtime settles, and the existing readiness lines when the model is available or its service is not. The compact detail drops the cache and memory hint paragraphs, hides the empty status line, and places Refresh after the controls. Retry, Cancel preparation, Keep on this device, Open in Assistant and the readiness and failure distinctions are unchanged on both surfaces. The unreachable "Waiting to prepare…" fallback is gone: every non-active state has a phase. The failure smoke pins the phase mapping and both copy tables literally. The browser smoke names the acquisition control and the compact copy per surface and observes a saved Keep choice through the control itself. Verification (Chrome 148 through playwright 1.59.1): node scripts/model-management-browser-smoke.mjs PASS (Marketplace and System) node scripts/model-preparation-failure-smoke.mjs PASS node scripts/home-entropy-check.mjs PASS Co-authored-by: Anders --- capsules/_shared/model-management.js | 48 +++++++++++++++---- .../marketplace/browser/model-management.js | 48 +++++++++++++++---- capsules/system/browser/model-management.js | 48 +++++++++++++++---- scripts/model-management-browser-smoke.mjs | 32 ++++++++----- scripts/model-preparation-failure-smoke.mjs | 37 ++++++++++++-- 5 files changed, 167 insertions(+), 46 deletions(-) diff --git a/capsules/_shared/model-management.js b/capsules/_shared/model-management.js index a770cc212..89d4cafa4 100644 --- a/capsules/_shared/model-management.js +++ b/capsules/_shared/model-management.js @@ -13,6 +13,29 @@ preparation_unavailable: "Preparation failed.", }); const failureText = p => FAILURE_TEXT[p.failure_class] || "Preparation failed."; + // One phase per Runtime projection. The full view (System) and the compact + // Marketplace detail read their copy from the same phase. + function phaseOf(r, p, message) { + if (message) return "unavailable"; + if (r.dispatch_ready) return "ready"; + if (r.admitted) return "service_unavailable"; + if (active(p)) return p.cancel_requested ? "cancelling" : p.state === "capacity_pending" ? "capacity" : p.state === "uncertain" ? "settling" : p.state === "verifying" ? "verifying" : "preparing"; + if (!p) return "absent"; + return p.state; + } + const PHASE_TEXT = Object.freeze({ + unavailable: "Current status unavailable", ready: "Available on this device", + service_unavailable: "Available on this device. Model service unavailable.", + cancelling: "Cancelling preparation…", capacity: "Waiting for local capacity…", settling: "Waiting for preparation to settle…", + verifying: "Checking model files…", preparing: "Preparing local files…", absent: "Ready to prepare", + reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", expired: "Preparation expired.", + }); + // The compact detail sits under the model's own title, so it speaks about + // this device and the Get action instead of the preparation mechanism. + const COMPACT_PHASE_TEXT = Object.freeze({ + ...PHASE_TEXT, absent: "Not on this device yet", preparing: "Getting the model…", + reclaimed: "Removed from this device", cancelled: "Stopped before finishing", expired: "Not finished in time", + }); const text = (v, max = 256) => typeof v === "string" && v.length > 0 && v.length <= max && !/[\u0000-\u001f]/.test(v); function check(ok) { if (!ok) throw new Error("Invalid model response"); } function parseRuntime(r, cid) { @@ -229,9 +252,10 @@ }; root.replaceChildren(); const refresh = button("Refresh models", () => void load(), loading); - refresh.dataset.modelControl = "refresh"; root.append(refresh); + refresh.dataset.modelControl = "refresh"; + if (!compact) root.append(refresh); const status = element("p", loading ? "Loading models…" : message); - status.setAttribute("role", "status"); root.append(status); + status.setAttribute("role", "status"); status.hidden = compact && !status.textContent; root.append(status); if (!model) { if (!loading && !message && !requiredCid && choices.length > 1) { root.append(element("p", "Select a verified model. Keep and removal apply to the model you select.")); @@ -244,10 +268,12 @@ list.append(row); } root.append(list); + if (compact) root.append(refresh); restoreFocus(); return; } if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); + if (compact) root.append(refresh); restoreFocus(); return; } @@ -260,16 +286,15 @@ element("p", `Content ID: ${model.cid}`, "model-identity")); row.append(identity); } - const label = message ? "Current status unavailable" : r.dispatch_ready ? "Available on this device" : r.admitted ? "Available on this device. Model service unavailable." - : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : p.state === "verifying" ? "Checking model files…" : "Preparing local files…") - : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Ready to prepare"; + const phase = phaseOf(r, p, message); + const label = phase === "failed" ? failureText(p) : (compact ? COMPACT_PHASE_TEXT : PHASE_TEXT)[phase]; row.append(element("p", label)); if (active(p)) { const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${formatBytes(p.completed_bytes)} of ${formatBytes(p.total_bytes)} prepared`)); } const controls = element("div", "", "model-controls"); - const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); + const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : compact ? "Get" : "Use", () => void act("use"), active(p) || reconcileRequired); use.dataset.modelControl = "use"; if (!active(p) && !r.dispatch_ready) controls.append(use); if (r.dispatch_ready && typeof onReadyOpen === "function") { @@ -283,11 +308,14 @@ toggle.addEventListener("change", () => void act("retention", toggle.checked)); labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); row.append(controls); - row.append(element("p", r.kept - ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") - : "Prepared files can be removed during cache cleanup.", "model-hint")); - row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + if (!compact) { + row.append(element("p", r.kept + ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") + : "Prepared files can be removed during cache cleanup.", "model-hint")); + row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + } root.append(row); + if (compact) root.append(refresh); if (!requiredCid && choices.length > 1) { const back = button("Choose another model", () => { selectedCid = null; model = null; void load(); }); back.dataset.modelControl = "choose-another"; diff --git a/capsules/marketplace/browser/model-management.js b/capsules/marketplace/browser/model-management.js index a770cc212..89d4cafa4 100644 --- a/capsules/marketplace/browser/model-management.js +++ b/capsules/marketplace/browser/model-management.js @@ -13,6 +13,29 @@ preparation_unavailable: "Preparation failed.", }); const failureText = p => FAILURE_TEXT[p.failure_class] || "Preparation failed."; + // One phase per Runtime projection. The full view (System) and the compact + // Marketplace detail read their copy from the same phase. + function phaseOf(r, p, message) { + if (message) return "unavailable"; + if (r.dispatch_ready) return "ready"; + if (r.admitted) return "service_unavailable"; + if (active(p)) return p.cancel_requested ? "cancelling" : p.state === "capacity_pending" ? "capacity" : p.state === "uncertain" ? "settling" : p.state === "verifying" ? "verifying" : "preparing"; + if (!p) return "absent"; + return p.state; + } + const PHASE_TEXT = Object.freeze({ + unavailable: "Current status unavailable", ready: "Available on this device", + service_unavailable: "Available on this device. Model service unavailable.", + cancelling: "Cancelling preparation…", capacity: "Waiting for local capacity…", settling: "Waiting for preparation to settle…", + verifying: "Checking model files…", preparing: "Preparing local files…", absent: "Ready to prepare", + reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", expired: "Preparation expired.", + }); + // The compact detail sits under the model's own title, so it speaks about + // this device and the Get action instead of the preparation mechanism. + const COMPACT_PHASE_TEXT = Object.freeze({ + ...PHASE_TEXT, absent: "Not on this device yet", preparing: "Getting the model…", + reclaimed: "Removed from this device", cancelled: "Stopped before finishing", expired: "Not finished in time", + }); const text = (v, max = 256) => typeof v === "string" && v.length > 0 && v.length <= max && !/[\u0000-\u001f]/.test(v); function check(ok) { if (!ok) throw new Error("Invalid model response"); } function parseRuntime(r, cid) { @@ -229,9 +252,10 @@ }; root.replaceChildren(); const refresh = button("Refresh models", () => void load(), loading); - refresh.dataset.modelControl = "refresh"; root.append(refresh); + refresh.dataset.modelControl = "refresh"; + if (!compact) root.append(refresh); const status = element("p", loading ? "Loading models…" : message); - status.setAttribute("role", "status"); root.append(status); + status.setAttribute("role", "status"); status.hidden = compact && !status.textContent; root.append(status); if (!model) { if (!loading && !message && !requiredCid && choices.length > 1) { root.append(element("p", "Select a verified model. Keep and removal apply to the model you select.")); @@ -244,10 +268,12 @@ list.append(row); } root.append(list); + if (compact) root.append(refresh); restoreFocus(); return; } if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); + if (compact) root.append(refresh); restoreFocus(); return; } @@ -260,16 +286,15 @@ element("p", `Content ID: ${model.cid}`, "model-identity")); row.append(identity); } - const label = message ? "Current status unavailable" : r.dispatch_ready ? "Available on this device" : r.admitted ? "Available on this device. Model service unavailable." - : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : p.state === "verifying" ? "Checking model files…" : "Preparing local files…") - : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Ready to prepare"; + const phase = phaseOf(r, p, message); + const label = phase === "failed" ? failureText(p) : (compact ? COMPACT_PHASE_TEXT : PHASE_TEXT)[phase]; row.append(element("p", label)); if (active(p)) { const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${formatBytes(p.completed_bytes)} of ${formatBytes(p.total_bytes)} prepared`)); } const controls = element("div", "", "model-controls"); - const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); + const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : compact ? "Get" : "Use", () => void act("use"), active(p) || reconcileRequired); use.dataset.modelControl = "use"; if (!active(p) && !r.dispatch_ready) controls.append(use); if (r.dispatch_ready && typeof onReadyOpen === "function") { @@ -283,11 +308,14 @@ toggle.addEventListener("change", () => void act("retention", toggle.checked)); labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); row.append(controls); - row.append(element("p", r.kept - ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") - : "Prepared files can be removed during cache cleanup.", "model-hint")); - row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + if (!compact) { + row.append(element("p", r.kept + ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") + : "Prepared files can be removed during cache cleanup.", "model-hint")); + row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + } root.append(row); + if (compact) root.append(refresh); if (!requiredCid && choices.length > 1) { const back = button("Choose another model", () => { selectedCid = null; model = null; void load(); }); back.dataset.modelControl = "choose-another"; diff --git a/capsules/system/browser/model-management.js b/capsules/system/browser/model-management.js index a770cc212..89d4cafa4 100644 --- a/capsules/system/browser/model-management.js +++ b/capsules/system/browser/model-management.js @@ -13,6 +13,29 @@ preparation_unavailable: "Preparation failed.", }); const failureText = p => FAILURE_TEXT[p.failure_class] || "Preparation failed."; + // One phase per Runtime projection. The full view (System) and the compact + // Marketplace detail read their copy from the same phase. + function phaseOf(r, p, message) { + if (message) return "unavailable"; + if (r.dispatch_ready) return "ready"; + if (r.admitted) return "service_unavailable"; + if (active(p)) return p.cancel_requested ? "cancelling" : p.state === "capacity_pending" ? "capacity" : p.state === "uncertain" ? "settling" : p.state === "verifying" ? "verifying" : "preparing"; + if (!p) return "absent"; + return p.state; + } + const PHASE_TEXT = Object.freeze({ + unavailable: "Current status unavailable", ready: "Available on this device", + service_unavailable: "Available on this device. Model service unavailable.", + cancelling: "Cancelling preparation…", capacity: "Waiting for local capacity…", settling: "Waiting for preparation to settle…", + verifying: "Checking model files…", preparing: "Preparing local files…", absent: "Ready to prepare", + reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", expired: "Preparation expired.", + }); + // The compact detail sits under the model's own title, so it speaks about + // this device and the Get action instead of the preparation mechanism. + const COMPACT_PHASE_TEXT = Object.freeze({ + ...PHASE_TEXT, absent: "Not on this device yet", preparing: "Getting the model…", + reclaimed: "Removed from this device", cancelled: "Stopped before finishing", expired: "Not finished in time", + }); const text = (v, max = 256) => typeof v === "string" && v.length > 0 && v.length <= max && !/[\u0000-\u001f]/.test(v); function check(ok) { if (!ok) throw new Error("Invalid model response"); } function parseRuntime(r, cid) { @@ -229,9 +252,10 @@ }; root.replaceChildren(); const refresh = button("Refresh models", () => void load(), loading); - refresh.dataset.modelControl = "refresh"; root.append(refresh); + refresh.dataset.modelControl = "refresh"; + if (!compact) root.append(refresh); const status = element("p", loading ? "Loading models…" : message); - status.setAttribute("role", "status"); root.append(status); + status.setAttribute("role", "status"); status.hidden = compact && !status.textContent; root.append(status); if (!model) { if (!loading && !message && !requiredCid && choices.length > 1) { root.append(element("p", "Select a verified model. Keep and removal apply to the model you select.")); @@ -244,10 +268,12 @@ list.append(row); } root.append(list); + if (compact) root.append(refresh); restoreFocus(); return; } if (!loading && !message) root.append(element("p", "No verified model is available. Ask your administrator to configure a trusted catalog.")); + if (compact) root.append(refresh); restoreFocus(); return; } @@ -260,16 +286,15 @@ element("p", `Content ID: ${model.cid}`, "model-identity")); row.append(identity); } - const label = message ? "Current status unavailable" : r.dispatch_ready ? "Available on this device" : r.admitted ? "Available on this device. Model service unavailable." - : active(p) ? (p.cancel_requested ? "Cancelling preparation…" : p.state === "capacity_pending" ? "Waiting for local capacity…" : p.state === "uncertain" ? "Waiting for preparation to settle…" : p.state === "verifying" ? "Checking model files…" : "Preparing local files…") - : p ? ({ reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", failed: failureText(p), expired: "Preparation expired." }[p.state] || "Waiting to prepare…") : "Ready to prepare"; + const phase = phaseOf(r, p, message); + const label = phase === "failed" ? failureText(p) : (compact ? COMPACT_PHASE_TEXT : PHASE_TEXT)[phase]; row.append(element("p", label)); if (active(p)) { const progress = element("progress"); progress.max = p.total_bytes || 1; progress.value = p.completed_bytes; progress.setAttribute("aria-label", "Preparation progress"); row.append(progress, element("p", `${formatBytes(p.completed_bytes)} of ${formatBytes(p.total_bytes)} prepared`)); } const controls = element("div", "", "model-controls"); - const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : "Use", () => void act("use"), active(p) || reconcileRequired); + const use = button((p && p.state !== "reclaimed") || unresolvedUse ? "Retry" : compact ? "Get" : "Use", () => void act("use"), active(p) || reconcileRequired); use.dataset.modelControl = "use"; if (!active(p) && !r.dispatch_ready) controls.append(use); if (r.dispatch_ready && typeof onReadyOpen === "function") { @@ -283,11 +308,14 @@ toggle.addEventListener("change", () => void act("retention", toggle.checked)); labelNode.append(toggle, document.createTextNode("Keep on this device")); controls.append(labelNode); row.append(controls); - row.append(element("p", r.kept - ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") - : "Prepared files can be removed during cache cleanup.", "model-hint")); - row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + if (!compact) { + row.append(element("p", r.kept + ? (r.admitted ? "Kept on this device. Release this choice to allow cache cleanup." : "Keep choice saved for this preparation.") + : "Prepared files can be removed during cache cleanup.", "model-hint")); + row.append(element("p", "The model loads into memory when you use it in Assistant.", "model-hint")); + } root.append(row); + if (compact) root.append(refresh); if (!requiredCid && choices.length > 1) { const back = button("Choose another model", () => { selectedCid = null; model = null; void load(); }); back.dataset.modelControl = "choose-another"; diff --git a/scripts/model-management-browser-smoke.mjs b/scripts/model-management-browser-smoke.mjs index 27286214a..c0f694aa3 100644 --- a/scripts/model-management-browser-smoke.mjs +++ b/scripts/model-management-browser-smoke.mjs @@ -116,6 +116,11 @@ const browser = await chromium.launch({ executablePath: brave, headless: true }) let activePage; try { for (const app of ["marketplace", "system"]) { + // Marketplace renders the compact detail: Get instead of Use, device-centred phase copy, no hints. + const acquire = app === "marketplace" ? "Get" : "Use"; + const copy = app === "marketplace" + ? { absent: "Not on this device yet", reclaimed: "Removed from this device" } + : { absent: "Ready to prepare", reclaimed: "Model removed from local cache." }; phase = "unprepared"; kept = false; failure = false; calls.length = 0; selectedCid = cid; executable = true; trust = "verified"; const page = await browser.newPage({ viewport: { width: 1100, height: 800 } }); @@ -151,22 +156,23 @@ try { assert.equal(calls.filter(c => c.method === "content.use").length, 0, "pending status never starts another Use"); phase = "reclaimed"; await frame.getByRole("button", { name: "Refresh models" }).click(); - await frame.getByText("Model removed from local cache.", { exact: true }).waitFor(); - assert.equal(await frame.getByRole("button", { name: "Use", exact: true }).isEnabled(), true); + await frame.getByText(copy.reclaimed, { exact: true }).waitFor(); + assert.equal(await frame.getByRole("button", { name: acquire, exact: true }).isEnabled(), true); assert.equal(await frame.getByRole("checkbox", { name: "Keep on this device" }).isEnabled(), false); phase = "unprepared"; await frame.getByRole("button", { name: "Refresh models" }).click(); - await frame.getByText("Ready to prepare", { exact: true }).waitFor(); + await frame.getByText(copy.absent, { exact: true }).waitFor(); let releaseUse; const useReceived = new Promise(resolveUse => { delayUse = reply => { releaseUse = reply; resolveUse(); }; }); - await frame.getByRole("button", { name: "Use", exact: true }).evaluate(button => { button.click(); button.click(); }); + await frame.getByRole("button", { name: acquire, exact: true }).evaluate(button => { button.click(); button.click(); }); await useReceived; assert.equal(calls.filter(c => c.method === "content.use").length, 1, "double-click dispatches once"); delayUse = null; releaseUse(); await frame.getByRole("button", { name: "Cancel preparation" }).waitFor(); assert.equal(calls.filter(c => c.method === "content.use").length, 1); await frame.getByRole("checkbox", { name: "Keep on this device" }).check(); - await frame.getByText("Keep choice saved for this preparation.", { exact: true }).waitFor(); + if (app === "marketplace") await frame.locator('[data-model-control="keep"]:enabled:checked').waitFor(); + else await frame.getByText("Keep choice saved for this preparation.", { exact: true }).waitFor(); await frame.getByRole("button", { name: "Refresh models" }).click(); await frame.locator('[data-model-management] input:enabled:checked').waitFor(); assert.equal(calls.filter(c => c.method === "content.use").length, 1, "retention never starts another preparation"); @@ -246,20 +252,20 @@ try { await page.setViewportSize({ width: 1100, height: 800 }); phase = "unprepared"; kept = false; await frame.getByRole("button", { name: "Refresh models" }).click(); - await frame.getByRole("button", { name: "Use", exact: true }).waitFor(); + await frame.getByRole("button", { name: acquire, exact: true }).waitFor(); let releaseCatalog; const catalogReceived = new Promise(resolveCatalog => { delayCatalog = reply => { releaseCatalog = reply; resolveCatalog(); }; }); const beforeRefreshUses = calls.filter(c => c.method === "content.use").length; await frame.getByRole("button", { name: "Refresh models" }).click(); await catalogReceived; - assert.equal(await frame.getByRole("button", { name: "Use", exact: true }).isDisabled(), true); - await frame.getByRole("button", { name: "Use", exact: true }).evaluate(n => n.click()); + assert.equal(await frame.getByRole("button", { name: acquire, exact: true }).isDisabled(), true); + await frame.getByRole("button", { name: acquire, exact: true }).evaluate(n => n.click()); assert.equal(calls.filter(c => c.method === "content.use").length, beforeRefreshUses, "catalog refresh blocks old-model mutation"); delayCatalog = null; releaseCatalog(); await frame.locator('[data-model-control="use"]:enabled').waitFor(); loseUseResponse = true; const usesBeforeLoss = calls.filter(c => c.method === "content.use").length; - await frame.getByRole("button", { name: "Use", exact: true }).click(); + await frame.getByRole("button", { name: acquire, exact: true }).click(); await frame.getByText("Model action could not be confirmed. Refresh to check its status.", { exact: true }).waitFor(); assert.equal(await frame.getByRole("button", { name: "Retry", exact: true }).isDisabled(), true); loseUseResponse = false; @@ -270,16 +276,16 @@ try { assert.equal(new Set(lostRequests.map(c => c.request_id)).size, 1, "transport retries and reconciliation preserve the same Use identity"); phase = "unprepared"; kept = false; await frame.getByRole("button", { name: "Refresh models" }).click(); - await frame.getByRole("button", { name: "Use", exact: true }).waitFor(); + await frame.getByRole("button", { name: acquire, exact: true }).waitFor(); wrongIdentity = true; - await frame.getByRole("button", { name: "Use", exact: true }).click(); + await frame.getByRole("button", { name: acquire, exact: true }).click(); await frame.getByText("Model action could not be confirmed. Refresh to check its status.", { exact: true }).waitFor(); assert.equal(await frame.getByRole("button", { name: "Cancel preparation" }).count(), 0, "mismatched response cannot install an operation"); wrongIdentity = false; executable = false; await frame.getByRole("button", { name: "Refresh models" }).click(); await frame.getByText("Models are unavailable. Check your connection and trusted catalog, then retry.", { exact: true }).waitFor(); - assert.equal(await frame.getByRole("button", { name: "Use", exact: true }).count(), 0); + assert.equal(await frame.getByRole("button", { name: acquire, exact: true }).count(), 0); executable = true; trust = "unavailable"; await frame.getByRole("button", { name: "Refresh models" }).click(); await frame.getByText("Models are unavailable. Check your connection and trusted catalog, then retry.", { exact: true }).waitFor(); @@ -355,7 +361,7 @@ try { assert.equal(await frame.getByRole("button", { name: "Cancel preparation" }).count(), 0, "late old operation cannot replace new identity"); assert.match(await frame.locator(app === "marketplace" ? ".model-content-identity" : ".model-identity").innerText(), new RegExp(selectedCid)); if (app === "marketplace") { - await frame.getByRole("button", { name: "Use", exact: true }).waitFor(); + await frame.getByRole("button", { name: acquire, exact: true }).waitFor(); const detailCid = selectedCid; await frame.locator("[data-model-management]").evaluate(node => { window.fixtureOpenModelDetail = node; diff --git a/scripts/model-preparation-failure-smoke.mjs b/scripts/model-preparation-failure-smoke.mjs index 5c14ea749..2aa641688 100644 --- a/scripts/model-preparation-failure-smoke.mjs +++ b/scripts/model-preparation-failure-smoke.mjs @@ -9,8 +9,8 @@ for (const capsule of ["system", "marketplace"]) { } const context = { window: {} }; vm.runInNewContext(source.replace("window.ElastosModelManagement = {", - "window.testFailure = { parseRuntime, operationRuntime, failureText, catalogEntries, selectCatalogEntry }; window.ElastosModelManagement = {"), context); -const { parseRuntime, operationRuntime, failureText, catalogEntries, selectCatalogEntry } = context.window.testFailure; + "window.testFailure = { parseRuntime, operationRuntime, failureText, catalogEntries, selectCatalogEntry, phaseOf, PHASE_TEXT, COMPACT_PHASE_TEXT }; window.ElastosModelManagement = {"), context); +const { parseRuntime, operationRuntime, failureText, catalogEntries, selectCatalogEntry, phaseOf, PHASE_TEXT, COMPACT_PHASE_TEXT } = context.window.testFailure; const cid = `bafybei${"a".repeat(52)}`; const preparation = { operation_id: "fixture", cid, state: "failed", total_bytes: 1024, completed_bytes: 512, cancel_requested: false, admitted: false, activation_pending: false }; @@ -55,4 +55,35 @@ assert.equal(selectCatalogEntry(two, cid).cid, cid); assert.equal(selectCatalogEntry(two, otherCid).cid, otherCid); assert.equal(selectCatalogEntry(two, `bafybei${"d".repeat(52)}`), null); assert.equal(selectCatalogEntry(two.slice(0, 1), null).cid, cid); -console.log("PASS model preparation failure: safe classes, operation/catalog parity, unknown history, strict rejection, shared copies, explicit two-entry selection"); +// One phase per projection. The full view keeps its established copy; the +// compact Marketplace detail changes only the device-centred lines. +const projection = (state, extra = {}) => ({ admitted: false, dispatch_ready: false, offer_id: null, + preparation: state === null ? null : { ...preparation, state, cancel_requested: false, failure_class: undefined, ...extra } }); +const phases = { + absent: [projection(null), null], ready: [{ ...projection("admitted"), admitted: true, dispatch_ready: true, offer_id: `model:${"b".repeat(64)}` }, null], + service_unavailable: [{ ...projection("admitted"), admitted: true }, null], + cancelling: [projection("preparing", { cancel_requested: true }), null], capacity: [projection("capacity_pending"), null], + settling: [projection("uncertain"), null], verifying: [projection("verifying"), null], preparing: [projection("preparing"), null], + failed: [projection("failed", { failure_class: "verification_failed" }), null], reclaimed: [projection("reclaimed"), null], + cancelled: [projection("cancelled"), null], expired: [projection("expired"), null], + unavailable: [projection("preparing"), "Status is unavailable. Refresh to check before trying again."], +}; +for (const [expected, [r, message]] of Object.entries(phases)) { + assert.equal(phaseOf(r, r.preparation, message), expected); +} +for (const reservedLike of ["reserved", "admission_pending"]) { + assert.equal(phaseOf(projection(reservedLike), projection(reservedLike).preparation, ""), "preparing", `${reservedLike} is an active preparation`); +} +// Table values cross the vm realm; compare their entries, not their prototype. +const plain = table => Object.fromEntries(Object.entries(table)); +assert.deepEqual(plain(PHASE_TEXT), { + unavailable: "Current status unavailable", ready: "Available on this device", + service_unavailable: "Available on this device. Model service unavailable.", + cancelling: "Cancelling preparation…", capacity: "Waiting for local capacity…", settling: "Waiting for preparation to settle…", + verifying: "Checking model files…", preparing: "Preparing local files…", absent: "Ready to prepare", + reclaimed: "Model removed from local cache.", cancelled: "Preparation cancelled.", expired: "Preparation expired.", +}); +assert.deepEqual(plain(COMPACT_PHASE_TEXT), { ...plain(PHASE_TEXT), absent: "Not on this device yet", preparing: "Getting the model…", + reclaimed: "Removed from this device", cancelled: "Stopped before finishing", expired: "Not finished in time" }); +assert.deepEqual(Object.keys(PHASE_TEXT).sort(), Object.keys(phases).filter(name => name !== "failed").sort(), "every phase except failed reads its copy from the table"); +console.log("PASS model preparation failure: safe classes, operation/catalog parity, unknown history, strict rejection, shared copies, explicit two-entry selection, phase copy tables"); From e8d35b6e711a339d7c24c085d69fabe645b7b4ee Mon Sep 17 00:00:00 2001 From: cloud Date: Thu, 17 Sep 2026 04:42:40 +0000 Subject: [PATCH 346/535] feat(marketplace): lead the model detail with identity and one forward action The model detail now opens with what a person needs to decide: the model's title, "Verified publisher" with the publisher DID in short form (full DID as the tooltip), the download size and version, then the controls, then exact identifiers under Technical details (full publisher DID, Content ID, bytes). The About paragraph, the Status checklist and the Works-with list no longer render for models; they repeated the header and the controls in other words. The one forward action, Get or Open in Assistant, is styled as the primary button, and the row's top rule and padding no longer separate it from the header. The card subtitle reads "Language model for Assistant." The Open in Assistant handoff, the acquisition methods and the Runtime's readiness and authority facts are unchanged. The browser smoke asserts the new header, the absence of checklist sections and hints, refresh placement, the exact identifiers in Technical details, Get before the first acquisition, and the literal handoff payload: one home:open-target message for Assistant carrying this model's content ID and live offer ID after Open in Assistant. Verification (Chrome 148 through playwright 1.59.1): node scripts/model-management-browser-smoke.mjs PASS node scripts/marketplace-product-behavior-smoke.mjs OK node scripts/marketplace-product-layout-smoke.mjs OK node scripts/home-entropy-check.mjs PASS git diff --check clean Co-authored-by: Anders --- capsules/marketplace/browser/marketplace.css | 23 +++++++++++++++++++ capsules/marketplace/browser/marketplace.js | 24 ++++++++++++++------ scripts/model-management-browser-smoke.mjs | 23 +++++++++++++++++++ 3 files changed, 63 insertions(+), 7 deletions(-) diff --git a/capsules/marketplace/browser/marketplace.css b/capsules/marketplace/browser/marketplace.css index a0a4ba72d..2c4bad821 100644 --- a/capsules/marketplace/browser/marketplace.css +++ b/capsules/marketplace/browser/marketplace.css @@ -686,6 +686,29 @@ button { background: var(--bg-hover); } +/* The model detail's one forward action reads as the primary button. */ +[data-model-management] .modal-btn[data-model-control="use"], +[data-model-management] .modal-btn[data-model-control="open-assistant"] { + border-color: transparent; + color: #fff; + background: var(--accent); +} + +[data-model-management] .modal-btn[data-model-control="use"]:hover, +[data-model-management] .modal-btn[data-model-control="open-assistant"]:hover { + background: var(--accent-hover); +} + +[data-model-management] .modal-btn[data-model-control="use"]:disabled, +[data-model-management] .modal-btn[data-model-control="open-assistant"]:disabled { + opacity: 0.55; +} + +[data-model-management] .model-row { + border-top: 0; + padding-top: 4px; +} + .store-loading { margin-top: 12px; } diff --git a/capsules/marketplace/browser/marketplace.js b/capsules/marketplace/browser/marketplace.js index 7f9b383ad..7d1ea1843 100644 --- a/capsules/marketplace/browser/marketplace.js +++ b/capsules/marketplace/browser/marketplace.js @@ -401,7 +401,7 @@ name: publicTitle(capsule), developer: String(model ? capsule.publisher_did : capsule.author || "Unknown publisher"), category: appCategory(capsule, role), - description: model ? `${publicTitle(capsule)} model for Assistant.` : publicDescription(capsule), + description: model ? "Language model for Assistant." : publicDescription(capsule), version: String(capsule.version || ""), installed, launchable, @@ -819,13 +819,24 @@ return roleLabel(app.role); } + function shortDid(did) { + return did.length <= 32 ? did : `${did.slice(0, 16)}…${did.slice(-4)}`; + } + function detailPublisher(app) { + if (app.modelCid) return `Verified publisher · ${shortDid(app.developer)}`; if (isFirstPartyPublisher(app.developer)) { return "ElastOS"; } return app.developer; } + function modelSizeLabel(bytes) { + const units = ["B", "KB", "MB", "GB", "TB"]; + const rank = bytes ? Math.min(4, Math.floor(Math.log(bytes) / Math.log(1000))) : 0; + return `${(bytes / 1000 ** rank).toLocaleString(undefined, { maximumFractionDigits: rank ? 2 : 0 })} ${units[rank]}`; + } + function renderAppRow(app) { return ` `; } -function renderInlineConfirmation(actionLabel) { +function renderInlineConfirmation(actionLabel, offer, enabling) { + const terms = hostedShareTerms(offer); + const termsBlock = enabling && terms + ? `` + : ""; return `
@@ -951,14 +962,14 @@ ${appIconHtml(app, "modal-icon-size")}
  • ${icons.check}${escapeHtml(signatureLabel(app.signatureState))}
  • - ${app.modelCid ? `
  • Content ID: ${escapeHtml(app.modelCid)}
  • ${app.contentBytes.toLocaleString()} bytes
  • ` : ""} + ${app.modelCid ? `
  • Publisher: ${escapeHtml(app.developer)}
  • Content ID: ${escapeHtml(app.modelCid)}
  • ${app.contentBytes.toLocaleString()} bytes
  • ` : ""}
  • ${icons.check}${escapeHtml(packageLabel(app))}
diff --git a/scripts/model-management-browser-smoke.mjs b/scripts/model-management-browser-smoke.mjs index c0f694aa3..897f7cef1 100644 --- a/scripts/model-management-browser-smoke.mjs +++ b/scripts/model-management-browser-smoke.mjs @@ -149,6 +149,19 @@ try { } await (app === "marketplace" ? frame.locator(".modal-title") : frame.getByRole("heading", { name: "Fixture model", exact: true })).waitFor(); assert.match(await frame.locator("body").textContent(), /did:key:zFixturePublisher/); + if (app === "marketplace") { + // The model detail leads with identity and verification, then the controls, then exact identifiers. + assert.equal(await frame.locator(".modal-developer").innerText(), "Verified publisher · did:key:zFixturePublisher"); + assert.match(await frame.locator(".modal-version").innerText(), /^1\.02 KB download$/); + assert.equal(await frame.locator(".modal-body .modal-section-title").filter({ hasText: /^(About|Status|Works with|Available actions)$/ }).count(), 0, + "the model detail has no checklist sections"); + assert.equal(await frame.locator("[data-model-management] .model-hint").count(), 0, "compact detail carries no cache or memory hints"); + assert.equal(await frame.locator("[data-model-management] > :last-child").getAttribute("data-model-control"), "refresh", + "refresh follows the model controls"); + assert.match(await frame.locator(".technical-details .model-publisher-identity").evaluate(node => node.textContent), /^Publisher: did:key:zFixturePublisher$/); + assert.equal(await frame.getByText("Not on this device yet", { exact: true }).count(), 1); + assert.equal(await frame.getByRole("button", { name: "Get", exact: true }).isEnabled(), true); + } phase = "capacity_pending"; await frame.getByRole("button", { name: "Refresh models" }).click(); await frame.getByText("Waiting for local capacity…", { exact: true }).waitFor(); @@ -188,6 +201,16 @@ try { activationPending = false; dispatchPending = false; await frame.locator("[data-model-management] p").filter({ hasText: /^Available on this device$/ }).waitFor(); assert.equal(calls.filter(c => c.method === "content.use").length, activationUses, "activation readiness uses status without another Use"); + if (app === "marketplace") { + // The ready detail hands exactly this model and its live offer to Assistant through Home. + await page.evaluate(() => { window.homeMessages = []; window.addEventListener("message", event => window.homeMessages.push(event.data)); }); + assert.equal(await frame.getByRole("button", { name: acquire, exact: true }).count(), 0, "a ready model offers no second acquisition"); + await frame.getByRole("button", { name: "Open in Assistant", exact: true }).click(); + await page.waitForFunction(() => window.homeMessages.some(message => message?.type === "home:open-target")); + const handoff = (await page.evaluate(() => window.homeMessages)).filter(message => message?.type === "home:open-target"); + assert.deepEqual(handoff, [{ type: "home:open-target", target: "assistant", homeToken: "fixture-token", + query: { model_cid: cid, offer_id: `model:${"b".repeat(64)}` } }]); + } const readyStatuses = calls.filter(c => c.method === "content.status").length; await page.waitForTimeout(1750); assert.equal(calls.filter(c => c.method === "content.status").length, readyStatuses, "ready state stops polling"); From 98ca3d6343292563f56339cb049e0593601790e7 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Thu, 17 Sep 2026 13:31:35 -0500 Subject: [PATCH 347/535] fix(home): handshake Assistant ready after a capsule document reload Home keeps dataset.route across an iframe document reload. The new document must handshake ready again so Home can raise the room. Verification: node scripts/home-agent-shell-smoke.mjs PASS node scripts/home-entropy-check.mjs PASS git diff --check clean --- capsules/home-gui/browser/shell-assistant-face.js | 12 ++++++++++-- scripts/home-agent-shell-smoke.mjs | 6 ++++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/capsules/home-gui/browser/shell-assistant-face.js b/capsules/home-gui/browser/shell-assistant-face.js index 9fa5ce49b..f627811b2 100644 --- a/capsules/home-gui/browser/shell-assistant-face.js +++ b/capsules/home-gui/browser/shell-assistant-face.js @@ -218,10 +218,17 @@ function postToFrame(message) { target.postMessage(message, "*"); } -function markFrameReady() { - if (frameReady) { +function onAssistantFrameDocumentLoad() { + if (!frame?.dataset.route) { return; } + /* Home keeps dataset.route across a capsule document reload. The new + document must handshake ready again, or the room stays empty. */ + frameReady = false; + frame.classList.remove("is-ready"); +} + +function markFrameReady() { frameReady = true; frame?.classList.add("is-ready"); deps.pushUiPreferencesToFrameWindow(frame?.contentWindow); @@ -573,6 +580,7 @@ export function bindAssistantFace(dependencies) { deps = dependencies; spaceEl = document.querySelector("#assistant-space"); frame = document.querySelector("#assistant-space-frame"); + frame?.addEventListener("load", onAssistantFrameDocumentLoad); registerShellPopover(FACE_ID, () => hideAssistantFace({ instant: true })); bindAgentSpace({ available: () => Boolean(toggleEl()) && !toggleEl().hidden, diff --git a/scripts/home-agent-shell-smoke.mjs b/scripts/home-agent-shell-smoke.mjs index 101b4e072..ac5bdbd72 100644 --- a/scripts/home-agent-shell-smoke.mjs +++ b/scripts/home-agent-shell-smoke.mjs @@ -125,6 +125,12 @@ assert.ok( /* ---- ownership split with Home GUI ---------------------------------------- */ assert.ok(homeFace.includes('const TARGET_ID = "assistant"')); +assert.ok( + homeFace.includes("function onAssistantFrameDocumentLoad(") && + homeFace.includes('frame?.addEventListener("load", onAssistantFrameDocumentLoad)') && + homeFace.includes("frameReady = false"), + "a replaced Assistant document must handshake ready again so Home can raise the room", +); assert.ok( homeWindows.includes('const HOME_AGENT_TARGET_ID = "assistant"') && (homeWindows.match(/targetId === HOME_AGENT_TARGET_ID/g) || []).length === 3 && From bced8c1298908011d7cffc0d1070499336cad639 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Thu, 17 Sep 2026 13:31:39 -0500 Subject: [PATCH 348/535] fix(ipfs): match directory add flags and wait briefly for a local miss Kubo add of a directory now uses the same CID flags as directory_hash so a complete local object keeps one CID. A bounded local miss waits 100ms instead of 1ms so a Ready backend can still return a 64 KiB hit. Verification: cargo test --manifest-path capsules/ipfs-provider/Cargo.toml bounded PASS (6) cargo fmt --manifest-path capsules/ipfs-provider/Cargo.toml -- --check git diff --check clean --- capsules/ipfs-provider/src/main.rs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/capsules/ipfs-provider/src/main.rs b/capsules/ipfs-provider/src/main.rs index 52271dd50..fc77d70ca 100644 --- a/capsules/ipfs-provider/src/main.rs +++ b/capsules/ipfs-provider/src/main.rs @@ -25,6 +25,7 @@ const LOCKFILE_POLL_TIMEOUT: Duration = Duration::from_secs(30); const HTTP_TIMEOUT: Duration = Duration::from_secs(30); const LARGE_HTTP_TIMEOUT: Duration = Duration::from_secs(300); const BOUNDED_READ_TIMEOUT: Duration = Duration::from_secs(5); +const BOUNDED_READ_KUBO_TIMEOUT: &str = "100ms"; const MAX_BOUNDED_READ_BYTES: u64 = 64 * 1024; const MAX_CAPACITY_REQUIRED_BYTES: u64 = 64 * 1024 * 1024 * 1024; @@ -920,13 +921,13 @@ impl IpfsProvider { // availability. Searching DHT here would spend the whole // deadline on a miss and then still fail over. Kubo path cat // of a missing DAG also keeps the HTTP call open with - // offline=true; a 1ms API timeout fails that miss immediately. - // A local hit still returns the bounded range. + // offline=true; a short API timeout fails that miss. A local + // 64 KiB hit still returns the bounded range after Ready. let mut request = agent .post(&format!("{}/api/v0/cat", self.api_url())) .query("arg", &arg) .query("offline", "true") - .query("timeout", "1ms") + .query("timeout", BOUNDED_READ_KUBO_TIMEOUT) .set("Accept-Encoding", "identity") .timeout(BOUNDED_READ_TIMEOUT); if let Some(range) = &range { @@ -1303,7 +1304,7 @@ impl IpfsProvider { write!(body, "--{}--\r\n", boundary).unwrap(); let url = format!( - "{}/api/v0/add?wrap-with-directory=true&pin={}", + "{}/api/v0/add?wrap-with-directory=true&pin={}&cid-version=1&hash=sha2-256&raw-leaves=true&chunker=size-262144&trickle=false&max-file-links=174&max-directory-links=0&max-hamt-fanout=256&inline=false&inline-limit=32&nocopy=false&fscache=false&preserve-mode=false&preserve-mtime=false&empty-dirs=false&progress=false&fast-provide-root=false&fast-provide-wait=false", self.api_url(), pin ); @@ -2036,7 +2037,7 @@ mod tests { assert!(version_headers.starts_with("POST /api/v0/version")); assert!(headers.starts_with("POST /api/v0/cat?")); assert!(headers.contains("offline=true")); - assert!(headers.contains("timeout=1ms")); + assert!(headers.contains("timeout=100ms")); assert!(headers.contains("offset=8") && headers.contains("length=4")); let Response::Ok { data: Some(data) } = result else { panic!("bounded read failed"); @@ -2384,7 +2385,7 @@ mod tests { for (headers, offset) in [(first, "offset=8"), (second, "offset=12")] { assert!(headers.starts_with("POST /api/v0/cat?")); assert!(headers.contains("offline=true")); - assert!(headers.contains("timeout=1ms")); + assert!(headers.contains("timeout=100ms")); assert!(headers.contains(offset) && headers.contains("length=4")); } let Response::Ok { data: Some(data) } = next else { @@ -2435,7 +2436,7 @@ mod tests { assert_eq!(requests.len(), 1, "bounded failure must not follow, retry or prefetch"); assert!(requests[0].starts_with("POST /api/v0/cat?")); assert!(requests[0].contains("offline=true")); - assert!(requests[0].contains("timeout=1ms")); + assert!(requests[0].contains("timeout=100ms")); assert!(matches!(result, Response::Error { .. })); } } @@ -2569,7 +2570,7 @@ mod tests { ); assert!(requests[0].0.starts_with("POST /api/v0/cat?")); assert!(requests[0].0.contains("offline=true")); - assert!(requests[0].0.contains("timeout=1ms")); + assert!(requests[0].0.contains("timeout=100ms")); assert!(requests[0].1, "bounded Cat must send offset=8 and length=4"); if ignore_range { assert!( From 336749e36849cf750de85b330f369f68c42f5e59 Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Thu, 17 Sep 2026 13:31:45 -0500 Subject: [PATCH 349/535] fix(content): restore a complete local object as a named Carrier holder A pin miss now imports a complete local object directory, announces the replica, and restores those replicas when Carrier comes online. Fetch receipts name the authenticated holder DID. Non-index files use a closed range from the signed object size; max_bytes stays the whole-index bound. Verification: cargo test -p elastos-server --lib content_ensure_imports_complete_local_object_when_pin_fails cargo test -p elastos-server --lib content_restore_imports_complete_local_objects cargo test -p elastos-server --lib content_fetch_rejects_max_bytes_for_a_non_index_path cargo test -p elastos-server --lib fetch_model_part_requires_a_declared_range cargo test -p elastos-server --lib carrier_availability cargo test -p elastos-server --lib live_source_bootstrap_ticket cargo test -p elastos-server --bin elastos live_source_ticket_rejects (cd elastos && cargo fmt --all -- --check) git diff --check --- elastos/crates/elastos-server/src/carrier.rs | 1370 ++++++++++++++++- elastos/crates/elastos-server/src/chat_cmd.rs | 74 +- elastos/crates/elastos-server/src/content.rs | 528 ++++++- .../crates/elastos-server/src/server_infra.rs | 28 +- 4 files changed, 1885 insertions(+), 115 deletions(-) diff --git a/elastos/crates/elastos-server/src/carrier.rs b/elastos/crates/elastos-server/src/carrier.rs index 82c55cfc9..f92049dfb 100644 --- a/elastos/crates/elastos-server/src/carrier.rs +++ b/elastos/crates/elastos-server/src/carrier.rs @@ -64,7 +64,9 @@ use crate::content::{ContentObjectManifest, CONTENT_OBJECT_MANIFEST_PATH}; use crate::operator_control::{ load_operator_control, OperatorHandler, OperatorRuntimeContext, OPERATOR_ALPN, }; -use crate::sources::TrustedSource; +use crate::sources::{ + load_trusted_sources, normalize_gateways, save_trusted_sources, TrustedSource, +}; const CARRIER_ALPN: &[u8] = b"elastos/carrier/1"; #[path = "carrier_file.rs"] @@ -104,6 +106,8 @@ const CONTENT_STORAGE_MARKET_ADMISSION_POLICY_SCHEMA: &str = const CONTENT_BLOCK_GRAPH_PROVIDER: &str = "content-block-graph-provider"; const CONTENT_BLOCK_GRAPH_TARGET: &str = "block-graph"; const CARRIER_PEER_REPUTATION_SCHEMA: &str = "elastos.carrier.peer-reputation/v1"; +const CARRIER_LOCAL_HOLDER_ANNOUNCEMENTS_SCHEMA: &str = + "elastos.carrier.local-holder-announcements/v1"; const CARRIER_PEER_ATTESTATION_EXCHANGE_POLICY_SCHEMA: &str = "elastos.carrier.peer-attestation-exchange-policy/v1"; const CARRIER_PEER_ATTESTATION_EXCHANGE_REQUEST_SCHEMA: &str = @@ -129,6 +133,8 @@ const GOSSIP_JOIN_PEERS_TIMEOUT: Duration = Duration::from_millis(1_500); /// on LAN needed more than 8s (2026-09-16). const CONTENT_AVAILABILITY_DISCOVERY_WAIT: Duration = Duration::from_secs(30); const CONTENT_AVAILABILITY_DISCOVERY_POLL: Duration = Duration::from_millis(100); +const SOURCE_BOOTSTRAP_TIMEOUT: Duration = Duration::from_secs(3); +const SOURCE_BOOTSTRAP_PATH: &str = "/.well-known/elastos/carrier-bootstrap.json?role=publisher"; const GOSSIP_JOIN_EXACT_MAX_PEERS: usize = 16; const GOSSIP_CARRIER_PUSH_MAX_TOPIC_LEN: usize = 256; // Maximum complete JSON GossipMessage admitted to the iroh-gossip wire. @@ -678,6 +684,130 @@ pub(crate) fn public_key_to_did(public_key: &iroh::PublicKey) -> anyhow::Result< .context("iroh public keys must encode as one canonical Ed25519 did:key") } +/// Live publisher bootstrap from `/.well-known/elastos/carrier-bootstrap.json`. +/// The document names schema, role, and current endpoint coordinates. +/// The pinned `publisher_node_id` is the holder identity. +/// Carrier authenticates the replica session after that identity check. +#[derive(Debug, Clone, Default, serde::Deserialize)] +pub struct SourceCarrierBootstrapDocument { + #[serde(default)] + pub schema: String, + #[serde(default)] + pub role: String, + #[serde(default)] + pub ticket: String, + #[serde(default)] + pub node_id: String, + #[serde(default)] + pub did: Option, +} + +/// Accept a publisher bootstrap ticket only for the already-pinned holder. +/// Schema and role must match. Ticket endpoints must match the pinned node. +/// This document has no signature field; Carrier authenticates the session. +pub fn verified_source_bootstrap_ticket( + source: &TrustedSource, + document: &SourceCarrierBootstrapDocument, +) -> Option { + if document.schema != "elastos.carrier.bootstrap/v1" + || document.role != "publisher" + || document.ticket.trim().is_empty() + || document.node_id.trim().is_empty() + { + return None; + } + + let expected_node_id = source.publisher_node_id.trim(); + if !expected_node_id.is_empty() && document.node_id.trim() != expected_node_id { + return None; + } + + let endpoints = decode_ticket_endpoints(&document.ticket); + if endpoints.is_empty() + || endpoints + .iter() + .any(|endpoint| endpoint.id.to_string() != document.node_id.trim()) + { + return None; + } + + if let Some(did) = document + .did + .as_deref() + .map(str::trim) + .filter(|did| !did.is_empty()) + { + let ticket_did = ticket_holder_did(&document.ticket)?; + if did != ticket_did { + return None; + } + } + + Some(document.ticket.trim().to_string()) +} + +/// Fetch current endpoint coordinates from the trusted-source gateway. +pub async fn live_source_bootstrap_ticket(source: &TrustedSource) -> Option { + let client = reqwest::Client::builder() + .timeout(SOURCE_BOOTSTRAP_TIMEOUT) + .build() + .ok()?; + + for gateway in normalize_gateways(&source.gateways) { + let url = format!("{gateway}{SOURCE_BOOTSTRAP_PATH}"); + let Ok(response) = client.get(&url).send().await else { + continue; + }; + if !response.status().is_success() { + continue; + } + let Ok(document) = response.json::().await else { + continue; + }; + if let Some(ticket) = verified_source_bootstrap_ticket(source, &document) { + return Some(ticket); + } + } + None +} + +enum TrustedSourceBootstrapRefresh { + NotConfigured, + Rejected, + Current, + Fresh(String), +} + +async fn refresh_trusted_source_replica_ticket( + data_dir: &std::path::Path, + holder_did: &str, + current_ticket: &str, +) -> TrustedSourceBootstrapRefresh { + let Ok(mut config) = load_trusted_sources(data_dir) else { + return TrustedSourceBootstrapRefresh::NotConfigured; + }; + let Some(source_name) = config.sources.iter().find_map(|source| { + (trusted_source_replica_did(source).as_deref() == Some(holder_did)) + .then(|| source.name.clone()) + }) else { + return TrustedSourceBootstrapRefresh::NotConfigured; + }; + let Some(source) = config.source_named(Some(source_name.as_str())).cloned() else { + return TrustedSourceBootstrapRefresh::NotConfigured; + }; + let Some(live) = live_source_bootstrap_ticket(&source).await else { + return TrustedSourceBootstrapRefresh::Rejected; + }; + if live == current_ticket { + return TrustedSourceBootstrapRefresh::Current; + } + if let Some(stored) = config.source_named_mut(Some(source_name.as_str())) { + stored.connect_ticket = live.clone(); + } + let _ = save_trusted_sources(data_dir, &config); + TrustedSourceBootstrapRefresh::Fresh(live) +} + pub fn decode_ticket_endpoints(ticket: &str) -> Vec { let ticket_bytes = match data_encoding::BASE32_NOPAD.decode(ticket.to_ascii_uppercase().as_bytes()) { @@ -2301,6 +2431,60 @@ impl CarrierAvailabilityProvider { self } + pub async fn restore_persisted_local_holder_announcements(&self) { + let Some(data_dir) = self.data_dir.as_deref() else { + return; + }; + for holder in load_local_holder_announcements(data_dir) { + let mut request = serde_json::json!({ + "op": "ensure", + "cid": holder.cid, + "uri": holder.uri, + "policy": holder.policy, + "local": { + "status": "local_pinned", + "provider": "ipfs-provider", + "replicas": 1 + }, + "requirements": { + "min_replicas": 1, + "max_replicas": 1, + "require_live_multi_peer_proof": false + } + }); + if let Some(object_did) = holder.object_did.as_deref() { + request["object_did"] = serde_json::Value::String(object_did.to_string()); + } + if let Some(publisher_did) = holder.publisher_did.as_deref() { + request["publisher_did"] = serde_json::Value::String(publisher_did.to_string()); + } + match self.announce_availability(&request).await { + Ok(response) + if response.get("status").and_then(|value| value.as_str()) == Some("ok") => + { + info!(cid = %holder.cid, "carrier: restored local holder announcement"); + } + Ok(response) => { + warn!( + cid = %holder.cid, + "carrier: restored local holder announcement returned {}", + response + .get("code") + .and_then(|value| value.as_str()) + .unwrap_or("error") + ); + } + Err(err) => { + warn!( + cid = %holder.cid, + error = %err, + "carrier: restored local holder announcement failed" + ); + } + } + } + } + async fn record_peer_reputation(&self, node_did: &str, success: bool) { let snapshot = { let mut reputation = self.peer_reputation.lock().await; @@ -3088,6 +3272,25 @@ struct CarrierPeerReputationStore { peers: BTreeMap, } +#[derive(Debug, Clone, Serialize, Deserialize)] +struct LocalHolderAnnouncement { + cid: String, + #[serde(default)] + uri: String, + #[serde(default)] + policy: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + object_did: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + publisher_did: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct LocalHolderAnnouncementStore { + schema: String, + holders: Vec, +} + #[async_trait::async_trait] impl Provider for CarrierAvailabilityProvider { async fn handle(&self, _request: ResourceRequest) -> Result { @@ -3221,7 +3424,7 @@ impl CarrierAvailabilityProvider { // add`) are the replica targets this node was told to use; offer // them behind any announced holder of the CID. if let Some(data_dir) = self.data_dir.as_deref() { - append_operator_peer_store_replica_candidates( + append_known_holder_replica_candidates( &mut remote_candidate_pool, data_dir, &node_did, @@ -3331,6 +3534,32 @@ impl CarrierAvailabilityProvider { push_gossip_buffer_message(buffer, msg.clone()); } } + if let Some(data_dir) = self.data_dir.as_deref() { + if replicas > 0 { + if let Err(err) = record_local_holder_announcement( + data_dir, + LocalHolderAnnouncement { + cid: cid.to_string(), + uri: uri.clone(), + policy: policy.to_string(), + object_did: request + .get("object_did") + .and_then(|value| value.as_str()) + .filter(|value| !value.trim().is_empty()) + .map(str::to_string), + publisher_did: request + .get("publisher_did") + .and_then(|value| value.as_str()) + .filter(|value| !value.trim().is_empty()) + .map(str::to_string), + }, + ) { + debug!("carrier: persist local holder announcement failed: {err}"); + } + } else if let Err(err) = forget_local_holder_announcement(data_dir, cid) { + debug!("carrier: forget local holder announcement failed: {err}"); + } + } let delivery = match state.senders.get(&topic_name) { Some(sender) => { @@ -3557,7 +3786,7 @@ impl CarrierAvailabilityProvider { // peers fill the same gap they fill on ensure: a known holder that // has not yet announced this CID can still serve a bounded fetch. if let Some(data_dir) = self.data_dir.as_deref() { - append_operator_peer_store_replica_candidates( + append_known_holder_replica_candidates( &mut replicas, data_dir, &self_did, @@ -3568,7 +3797,7 @@ impl CarrierAvailabilityProvider { let next = replicas .into_iter() .find(|replica| !tried.contains(&replica.node_did)); - if let Some(replica) = next { + if let Some(mut replica) = next { if tried.len() >= MAX_CARRIER_REPLICATION_CANDIDATES { break; } @@ -3595,6 +3824,7 @@ impl CarrierAvailabilityProvider { "policy": "carrier_provider_invoke", "replicas": 1, "transport": "carrier-provider-plane", + "node_did": replica.node_did, "remote_transfer": remote_transfer, "checked_at": now_secs(), } @@ -3602,8 +3832,73 @@ impl CarrierAvailabilityProvider { })); } Err(err) => { + warn!( + holder = %replica.node_did, + error = %err, + "Carrier availability fetch from holder failed" + ); + let mut final_err = err.to_string(); + if let Some(data_dir) = self.data_dir.as_deref() { + match refresh_trusted_source_replica_ticket( + data_dir, + &replica.node_did, + &replica.connect_ticket, + ) + .await + { + TrustedSourceBootstrapRefresh::Fresh(live_ticket) => { + replica.connect_ticket = live_ticket; + match fetch_content_via_carrier_provider_invocation( + ®istry, + &replica.node_did, + &replica.connect_ticket, + cid, + path, + bound, + ) + .await + { + Ok((bytes, remote_transfer)) => { + self.record_peer_reputation(&replica.node_did, true) + .await; + return Ok(serde_json::json!({ + "status": "ok", + "data": { + "data": base64::engine::general_purpose::STANDARD.encode(bytes), + "availability": { + "status": "network_available", + "provider": "carrier-availability", + "policy": "carrier_provider_invoke", + "replicas": 1, + "transport": "carrier-provider-plane", + "node_did": replica.node_did, + "remote_transfer": remote_transfer, + "checked_at": now_secs(), + } + } + })); + } + Err(retry_err) => { + warn!( + holder = %replica.node_did, + error = %retry_err, + "Carrier availability fetch after bootstrap refresh failed" + ); + final_err = retry_err.to_string(); + } + } + } + TrustedSourceBootstrapRefresh::Rejected => { + final_err = format!( + "{final_err}; trusted-source bootstrap refresh rejected" + ); + } + TrustedSourceBootstrapRefresh::Current + | TrustedSourceBootstrapRefresh::NotConfigured => {} + } + } self.record_peer_reputation(&replica.node_did, false).await; - errors.push(err.to_string()); + errors.push(final_err); continue; } } @@ -5707,6 +6002,99 @@ fn append_operator_peer_store_replica_candidates( sort_replica_candidates(pool); } +/// Installer-written trusted sources fill the same Get/Use holder gap as +/// `elastos node peer add`, without marking those tickets as gossip +/// bootstrap peers. The replica DID is the connect-ticket endpoint encoded +/// as `did:key`, so Carrier route filtering matches the ticket. +fn append_trusted_source_replica_candidates( + pool: &mut Vec, + data_dir: &std::path::Path, + self_did: &str, + reputation: &HashMap, + now: u64, +) { + let config = match load_trusted_sources(data_dir) { + Ok(config) => config, + Err(err) => { + debug!("carrier: no trusted sources for replica candidates: {err}"); + return; + } + }; + if config + .sources + .iter() + .all(|source| source.connect_ticket.trim().is_empty()) + { + debug!("carrier: trusted sources have no connect tickets for replica candidates"); + return; + } + for source in &config.sources { + let connect_ticket = source.connect_ticket.trim(); + if connect_ticket.is_empty() { + continue; + } + let Some(node_did) = trusted_source_replica_did(source) else { + warn!( + source = %source.name, + "carrier: trusted source has a connect ticket without a usable holder DID, skipping" + ); + continue; + }; + if node_did == self_did || pool.iter().any(|replica| replica.node_did == node_did) { + continue; + } + let (reputation_score, reputation_reason) = + carrier_reputation_score(reputation.get(&node_did)); + let mut score = 40_u32; + let mut reasons = vec!["trusted_source"]; + if reputation_score > 0 { + score = score.saturating_add(reputation_score as u32); + reasons.push("local_reputation_positive"); + } else if reputation_score < 0 { + score = score.saturating_sub(reputation_score.unsigned_abs()); + reasons.push("local_reputation_negative"); + } else { + reasons.push("local_reputation_neutral"); + } + debug!( + source = %source.name, + "carrier: added trusted-source replica candidate" + ); + pool.push(CarrierAvailabilityReplica { + node_did, + endpoint_id: None, + connect_ticket: connect_ticket.to_string(), + announced_at: now, + score: score.min(100), + selection_reason: reasons.join("+"), + reputation_score, + reputation_reason, + }); + } + sort_replica_candidates(pool); +} + +fn append_known_holder_replica_candidates( + pool: &mut Vec, + data_dir: &std::path::Path, + self_did: &str, + reputation: &HashMap, + now: u64, +) { + append_operator_peer_store_replica_candidates(pool, data_dir, self_did, reputation, now); + append_trusted_source_replica_candidates(pool, data_dir, self_did, reputation, now); +} + +fn ticket_holder_did(ticket: &str) -> Option { + decode_ticket_endpoints(ticket) + .into_iter() + .find_map(|endpoint| public_key_to_did(&endpoint.id).ok()) +} + +fn trusted_source_replica_did(source: &TrustedSource) -> Option { + ticket_holder_did(&source.connect_ticket) +} + fn carrier_replica_candidate_score( endpoint_id: Option<&str>, announced_at: u64, @@ -5812,6 +6200,94 @@ fn save_carrier_peer_reputation( Ok(()) } +fn carrier_local_holder_announcements_path(data_dir: &std::path::Path) -> PathBuf { + data_dir + .join("ElastOS") + .join("SystemServices") + .join("Content") + .join("local-holder-announcements.json") +} + +fn load_local_holder_announcements(data_dir: &std::path::Path) -> Vec { + let path = carrier_local_holder_announcements_path(data_dir); + let Ok(bytes) = std::fs::read(&path) else { + return Vec::new(); + }; + let Ok(store) = serde_json::from_slice::(&bytes) else { + debug!( + "carrier local holder announcements decode failed: {}", + path.display() + ); + return Vec::new(); + }; + if store.schema != CARRIER_LOCAL_HOLDER_ANNOUNCEMENTS_SCHEMA { + debug!( + "carrier local holder announcements schema mismatch at {}: {}", + path.display(), + store.schema + ); + return Vec::new(); + } + store + .holders + .into_iter() + .filter(|holder| validate_content_cid(&holder.cid).is_ok()) + .take(MAX_TOPICS) + .collect() +} + +fn save_local_holder_announcements( + data_dir: &std::path::Path, + store: &LocalHolderAnnouncementStore, +) -> Result<()> { + let path = carrier_local_holder_announcements_path(data_dir); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + let bytes = serde_json::to_vec_pretty(store)?; + std::fs::write(path, bytes)?; + Ok(()) +} + +fn record_local_holder_announcement( + data_dir: &std::path::Path, + announcement: LocalHolderAnnouncement, +) -> Result<()> { + if validate_content_cid(&announcement.cid).is_err() { + anyhow::bail!("local holder announcement requires a valid CID"); + } + let mut holders = load_local_holder_announcements(data_dir); + holders.retain(|holder| holder.cid != announcement.cid); + holders.push(announcement); + if holders.len() > MAX_TOPICS { + let drop_count = holders.len() - MAX_TOPICS; + holders.drain(0..drop_count); + } + save_local_holder_announcements( + data_dir, + &LocalHolderAnnouncementStore { + schema: CARRIER_LOCAL_HOLDER_ANNOUNCEMENTS_SCHEMA.to_string(), + holders, + }, + ) +} + +fn forget_local_holder_announcement(data_dir: &std::path::Path, cid: &str) -> Result<()> { + let mut holders = load_local_holder_announcements(data_dir); + let before = holders.len(); + holders.retain(|holder| holder.cid != cid); + if holders.len() == before { + return Ok(()); + } + save_local_holder_announcements( + data_dir, + &LocalHolderAnnouncementStore { + schema: CARRIER_LOCAL_HOLDER_ANNOUNCEMENTS_SCHEMA.to_string(), + holders, + }, + ) +} + fn gossip_cursor_error(code: &str, message: &str) -> serde_json::Value { serde_json::json!({"status":"error","code":code,"message":message}) } @@ -6978,7 +7454,7 @@ impl CarrierProviderInvoker { } fn carrier_provider_public_connect_error(index: usize, err: &anyhow::Error) -> String { - tracing::debug!( + tracing::warn!( ticket_index = index, error = %format_args!("{err:#}"), "Carrier provider connect failed" @@ -6987,7 +7463,7 @@ fn carrier_provider_public_connect_error(index: usize, err: &anyhow::Error) -> S } fn carrier_provider_public_invoke_error(index: usize, err: &anyhow::Error) -> String { - tracing::debug!( + tracing::warn!( ticket_index = index, error = %format_args!("{err:#}"), "Carrier provider invocation failed" @@ -12815,13 +13291,13 @@ pub(crate) mod tests { } #[tokio::test] - async fn test_carrier_availability_fetch_waits_for_a_late_signed_announcement() { - // A consumer that joins after the holder announced still has an empty - // local buffer. The fetch waits for the signed announcement instead of - // returning carrier_fetch_unavailable before gossip delivers it. + async fn test_carrier_availability_fetch_uses_trusted_source_without_announcements() { + // Ordinary install writes sources.json, not `elastos node peer add`. + // Get still has to invoke that holder before gossip announces the CID. let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; let topic_name = content_availability_topic_name(cid); - let (local_sk, local_did) = elastos_identity::derive_did(&[25u8; 32]); + let (remote_sk, expected_holder_did) = elastos_identity::derive_did(&[42u8; 32]); + let (local_sk, local_did) = elastos_identity::derive_did(&[43u8; 32]); let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) .bind() @@ -12833,7 +13309,11 @@ pub(crate) mod tests { .unwrap() .add(memory_lookup.clone()); let gossip = spawn_carrier_gossip(&endpoint); - let holder_ticket = carrier_connect_ticket(&endpoint); + let remote_ticket = encode_ticket_for(iroh::EndpointAddr::from( + iroh::SecretKey::from_bytes(&remote_sk.to_bytes()).public(), + )); + let catalog_publisher_did = + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string(); let state = Arc::new(Mutex::new(GossipState::new( endpoint.clone(), gossip, @@ -12848,20 +13328,32 @@ pub(crate) mod tests { .buffers .lock() .await - .insert(topic_name.clone(), test_topic_buffer([], 0)); + .insert(topic_name, test_topic_buffer([], 0)); } - let buffers = state.lock().await.buffers.clone(); - let ticket = holder_ticket.clone(); - let topic = topic_name.clone(); - tokio::spawn(async move { - tokio::time::sleep(Duration::from_millis(200)).await; - let (message, _) = - signed_content_availability_message(cid, [26u8; 32], &ticket, "", now_secs()); - let mut buffers = buffers.lock().await; - let buffer = buffers.get_mut(&topic).expect("topic buffer"); - assert!(push_gossip_buffer_message(buffer, message)); - }); let data_dir = tempfile::tempdir().unwrap(); + crate::sources::save_trusted_sources( + data_dir.path(), + &crate::sources::TrustedSourcesConfig { + schema: "elastos.trusted-sources/v1".to_string(), + default_source: "official".to_string(), + sources: vec![TrustedSource { + name: "official".to_string(), + publisher_dids: vec![catalog_publisher_did.clone()], + channel: "stable".to_string(), + discovery_uri: "https://example.invalid/discovery".to_string(), + connect_ticket: remote_ticket.clone(), + gateways: Vec::new(), + install_path: String::new(), + installed_version: String::new(), + head_cid: String::new(), + publisher_node_id: iroh::SecretKey::from_bytes(&remote_sk.to_bytes()) + .public() + .to_string(), + ipns_name: String::new(), + }], + }, + ) + .unwrap(); let registry = Arc::new(ProviderRegistry::new()); let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); registry.set_carrier_invoker(invoker.clone()).await; @@ -12870,7 +13362,6 @@ pub(crate) mod tests { Arc::downgrade(®istry), data_dir.path().to_path_buf(), ); - let started = std::time::Instant::now(); let response = provider .send_raw(&serde_json::json!({ "op": "fetch", @@ -12883,24 +13374,41 @@ pub(crate) mod tests { .await .unwrap(); assert_eq!(response["status"], "ok", "{response}"); - assert!(started.elapsed() >= Duration::from_millis(200)); - assert!(started.elapsed() < Duration::from_secs(2)); + assert_eq!( + response["data"]["availability"]["policy"], + "carrier_provider_invoke" + ); let requests = invoker.requests.lock().await; assert!( requests .iter() - .all(|request| request["ticket"] == holder_ticket), + .all(|request| request["ticket"] == remote_ticket), "{requests:?}" ); assert!(requests.iter().any(|request| request["op"] == "fetch")); assert_eq!(requests.len(), 1); + let mut pool = Vec::new(); + append_trusted_source_replica_candidates( + &mut pool, + data_dir.path(), + &local_did, + &HashMap::new(), + 1, + ); + assert_eq!(pool.len(), 1); + assert_eq!(pool[0].node_did, expected_holder_did); + assert_ne!(pool[0].node_did, catalog_publisher_did); + assert!(pool[0].selection_reason.starts_with("trusted_source")); } #[tokio::test] - async fn test_carrier_availability_fetch_waits_past_a_non_holder_neighbor_for_a_later_holder() { + async fn test_carrier_availability_fetch_waits_for_a_late_signed_announcement() { + // A consumer that joins after the holder announced still has an empty + // local buffer. The fetch waits for the signed announcement instead of + // returning carrier_fetch_unavailable before gossip delivers it. let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; let topic_name = content_availability_topic_name(cid); - let (local_sk, local_did) = elastos_identity::derive_did(&[31u8; 32]); + let (local_sk, local_did) = elastos_identity::derive_did(&[25u8; 32]); let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) .bind() @@ -12928,8 +13436,87 @@ pub(crate) mod tests { .lock() .await .insert(topic_name.clone(), test_topic_buffer([], 0)); - guard - .topic_peers + } + let buffers = state.lock().await.buffers.clone(); + let ticket = holder_ticket.clone(); + let topic = topic_name.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(200)).await; + let (message, _) = + signed_content_availability_message(cid, [26u8; 32], &ticket, "", now_secs()); + let mut buffers = buffers.lock().await; + let buffer = buffers.get_mut(&topic).expect("topic buffer"); + assert!(push_gossip_buffer_message(buffer, message)); + }); + let data_dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + let started = std::time::Instant::now(); + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + assert!(started.elapsed() >= Duration::from_millis(200)); + assert!(started.elapsed() < Duration::from_secs(2)); + let requests = invoker.requests.lock().await; + assert!( + requests + .iter() + .all(|request| request["ticket"] == holder_ticket), + "{requests:?}" + ); + assert!(requests.iter().any(|request| request["op"] == "fetch")); + assert_eq!(requests.len(), 1); + } + + #[tokio::test] + async fn test_carrier_availability_fetch_waits_past_a_non_holder_neighbor_for_a_later_holder() { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[31u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let holder_ticket = carrier_connect_ticket(&endpoint); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name.clone(), test_topic_buffer([], 0)); + guard + .topic_peers .lock() .await .entry(topic_name.clone()) @@ -13068,6 +13655,311 @@ pub(crate) mod tests { ); } + #[tokio::test] + async fn test_carrier_availability_fetch_walks_a_dead_trusted_source_then_a_later_independent_holder( + ) { + // Exclusive-holder miss shape: the installer replica is unreachable, + // then a later independent signed announcement is the live holder. + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[61u8; 32]); + let (source_sk, dead_source_did) = elastos_identity::derive_did(&[62u8; 32]); + let (_, independent_holder_did) = elastos_identity::derive_did(&[63u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let live_ticket = "ticket:independent-holder"; + let dead_ticket = encode_ticket_for(iroh::EndpointAddr::from( + iroh::SecretKey::from_bytes(&source_sk.to_bytes()).public(), + )); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name.clone(), test_topic_buffer([], 0)); + } + let buffers = state.lock().await.buffers.clone(); + let topic = topic_name.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(250)).await; + let (message, _) = + signed_content_availability_message(cid, [63u8; 32], live_ticket, "", now_secs()); + let mut buffers = buffers.lock().await; + let buffer = buffers.get_mut(&topic).expect("topic buffer"); + assert!(push_gossip_buffer_message(buffer, message)); + }); + let data_dir = tempfile::tempdir().unwrap(); + crate::sources::save_trusted_sources( + data_dir.path(), + &crate::sources::TrustedSourcesConfig { + schema: "elastos.trusted-sources/v1".to_string(), + default_source: "official".to_string(), + sources: vec![TrustedSource { + name: "official".to_string(), + publisher_dids: vec![ + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string(), + ], + channel: "stable".to_string(), + discovery_uri: String::new(), + connect_ticket: dead_ticket.clone(), + gateways: Vec::new(), + install_path: String::new(), + installed_version: String::new(), + head_cid: String::new(), + publisher_node_id: iroh::SecretKey::from_bytes(&source_sk.to_bytes()) + .public() + .to_string(), + ipns_name: String::new(), + }], + }, + ) + .unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker { + fail_tickets: vec![dead_ticket.clone()], + ..Default::default() + }); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ) + .with_discovery_wait(Duration::from_secs(2)); + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + let requests = invoker.requests.lock().await; + assert!( + requests + .iter() + .any(|request| request["ticket"] == dead_ticket), + "{requests:?}" + ); + assert_eq!(requests.last().unwrap()["ticket"], live_ticket); + assert_eq!( + response["data"]["availability"]["node_did"].as_str(), + Some(independent_holder_did.as_str()), + "{response}" + ); + assert_ne!( + response["data"]["availability"]["node_did"].as_str(), + Some(dead_source_did.as_str()) + ); + } + + #[tokio::test] + async fn test_carrier_availability_announce_records_a_durable_local_holder() { + // A complete local pin must survive Carrier restart as a CID-topic + // holder. The in-memory gossip buffer is empty after restart. + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[64u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name, test_topic_buffer([], 0)); + } + let data_dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); + registry.set_carrier_invoker(invoker).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + let response = provider + .send_raw(&serde_json::json!({ + "op": "ensure", + "cid": cid, + "uri": format!("elastos://{cid}"), + "policy": "network_default", + "local": { + "status": "local_pinned", + "provider": "ipfs-provider", + "replicas": 1 + }, + "requirements": { + "min_replicas": 1, + "max_replicas": 1, + "require_live_multi_peer_proof": false + } + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + let path = data_dir + .path() + .join("ElastOS") + .join("SystemServices") + .join("Content") + .join("local-holder-announcements.json"); + assert!( + path.is_file(), + "local holder announce must persist the CID at {}", + path.display() + ); + let store: serde_json::Value = + serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + assert_eq!( + store["schema"], + "elastos.carrier.local-holder-announcements/v1" + ); + assert_eq!(store["holders"][0]["cid"], cid); + } + + #[tokio::test] + async fn test_carrier_availability_restore_reannounces_a_persisted_holder_into_an_empty_buffer() + { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let (local_sk, local_did) = elastos_identity::derive_did(&[65u8; 32]); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let first_state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup.clone(), + Some(elastos_identity::derive_did(&[65u8; 32]).0), + Some(local_did.clone()), + ))); + { + let mut guard = first_state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name.clone(), test_topic_buffer([], 0)); + } + let data_dir = tempfile::tempdir().unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker::default()); + registry.set_carrier_invoker(invoker).await; + let first = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + first_state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + let response = first + .send_raw(&serde_json::json!({ + "op": "ensure", + "cid": cid, + "uri": format!("elastos://{cid}"), + "policy": "network_default", + "local": { + "status": "local_pinned", + "provider": "ipfs-provider", + "replicas": 1 + }, + "requirements": { + "min_replicas": 1, + "max_replicas": 1, + "require_live_multi_peer_proof": false + } + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + + let restarted_gossip = spawn_carrier_gossip(&endpoint); + let restarted_state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + restarted_gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = restarted_state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name.clone(), test_topic_buffer([], 0)); + } + let restarted = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + restarted_state.clone(), + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ); + restarted + .restore_persisted_local_holder_announcements() + .await; + let messages = { + let state = restarted_state.lock().await; + let buffers = state.buffers.lock().await; + buffers + .get(&topic_name) + .map(|buffer| buffer.messages.iter().cloned().collect::>()) + .unwrap_or_default() + }; + let catchup = local_content_availability_catchup_message(&messages, &local_did) + .expect("restarted holder must reannounce a fetch ticket"); + assert_eq!(catchup.sender_id, local_did); + assert!(catchup.content.contains("connect_ticket")); + } + #[tokio::test] async fn test_carrier_availability_fetch_ends_at_the_deadline_when_holders_stay_unavailable() { let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; @@ -13629,6 +14521,418 @@ pub(crate) mod tests { ); } + #[test] + fn trusted_source_replica_did_comes_from_the_connect_ticket() { + let secret = iroh::SecretKey::from_bytes(&[44u8; 32]); + let ticket = encode_ticket_for(iroh::EndpointAddr::from(secret.public())); + let expected = public_key_to_did(&secret.public()).unwrap(); + let source = TrustedSource { + name: "official".to_string(), + publisher_dids: vec![ + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string() + ], + channel: "stable".to_string(), + discovery_uri: String::new(), + connect_ticket: ticket, + gateways: Vec::new(), + install_path: String::new(), + installed_version: String::new(), + head_cid: String::new(), + publisher_node_id: secret.public().to_string(), + ipns_name: String::new(), + }; + assert_eq!( + trusted_source_replica_did(&source).as_deref(), + Some(expected.as_str()) + ); + assert_ne!(source.publisher_dids[0], expected); + } + + fn publisher_bootstrap_document( + ticket: &str, + node_id: &str, + role: &str, + schema: &str, + ) -> serde_json::Value { + serde_json::json!({ + "schema": schema, + "role": role, + "ticket": ticket, + "node_id": node_id, + "transport": "carrier", + }) + } + + async fn serve_publisher_bootstrap( + document: serde_json::Value, + ) -> (String, tokio::task::JoinHandle<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let app = axum::Router::new().route( + "/.well-known/elastos/carrier-bootstrap.json", + axum::routing::get({ + let document = document.clone(); + move || { + let document = document.clone(); + async move { axum::Json(document) } + } + }), + ); + let handle = tokio::spawn(async move { + axum::serve(listener, app).await.ok(); + }); + (format!("http://{addr}"), handle) + } + + fn holder_tickets_for_secret( + seed: [u8; 32], + ) -> (ed25519_dalek::SigningKey, String, String, String) { + let (remote_sk, holder_did) = elastos_identity::derive_did(&seed); + let public = iroh::SecretKey::from_bytes(&remote_sk.to_bytes()).public(); + let stale = encode_ticket_for( + iroh::EndpointAddr::from(public) + .with_addrs([iroh::TransportAddr::Ip("127.0.0.1:1".parse().unwrap())]), + ); + let live = encode_ticket_for(iroh::EndpointAddr::from(public)); + (remote_sk, holder_did, stale, live) + } + + #[test] + fn verified_source_bootstrap_ticket_rejects_missing_schema_and_identity_mismatch() { + let secret = iroh::SecretKey::from_bytes(&[46u8; 32]); + let ticket = encode_ticket_for(iroh::EndpointAddr::from(secret.public())); + let node_id = secret.public().to_string(); + let source = TrustedSource { + name: "official".to_string(), + publisher_dids: vec![ + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string() + ], + channel: "stable".to_string(), + discovery_uri: String::new(), + connect_ticket: "stale".to_string(), + gateways: Vec::new(), + install_path: String::new(), + installed_version: String::new(), + head_cid: "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".to_string(), + publisher_node_id: node_id.clone(), + ipns_name: String::new(), + }; + let matching = SourceCarrierBootstrapDocument { + schema: "elastos.carrier.bootstrap/v1".to_string(), + role: "publisher".to_string(), + ticket: ticket.clone(), + node_id: node_id.clone(), + did: None, + }; + assert_eq!( + verified_source_bootstrap_ticket(&source, &matching), + Some(ticket.clone()) + ); + let missing_schema = SourceCarrierBootstrapDocument { + schema: String::new(), + role: "publisher".to_string(), + ticket: ticket.clone(), + node_id: node_id.clone(), + did: None, + }; + assert_eq!( + verified_source_bootstrap_ticket(&source, &missing_schema), + None + ); + let other = iroh::SecretKey::from_bytes(&[47u8; 32]); + let foreign = SourceCarrierBootstrapDocument { + schema: "elastos.carrier.bootstrap/v1".to_string(), + role: "publisher".to_string(), + ticket: encode_ticket_for(iroh::EndpointAddr::from(other.public())), + node_id: other.public().to_string(), + did: None, + }; + assert_eq!(verified_source_bootstrap_ticket(&source, &foreign), None); + } + + #[tokio::test] + async fn live_source_bootstrap_ticket_rejects_http_refresh_without_bootstrap_schema() { + let secret = iroh::SecretKey::from_bytes(&[53u8; 32]); + let ticket = encode_ticket_for(iroh::EndpointAddr::from(secret.public())); + let node_id = secret.public().to_string(); + let (gateway, _server) = serve_publisher_bootstrap(publisher_bootstrap_document( + &ticket, + &node_id, + "publisher", + "", + )) + .await; + let source = TrustedSource { + name: "official".to_string(), + publisher_dids: vec![ + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string() + ], + channel: "stable".to_string(), + discovery_uri: String::new(), + connect_ticket: ticket, + gateways: vec![gateway], + install_path: String::new(), + installed_version: String::new(), + head_cid: "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".to_string(), + publisher_node_id: node_id, + ipns_name: String::new(), + }; + assert_eq!(live_source_bootstrap_ticket(&source).await, None); + } + + async fn availability_provider_with_trusted_source( + stale_ticket: String, + publisher_node_id: String, + catalog_publisher_did: String, + gateway: String, + local_sk: ed25519_dalek::SigningKey, + local_did: String, + fail_tickets: Vec, + ) -> ( + CarrierAvailabilityProvider, + Arc, + Arc, + tempfile::TempDir, + String, + ) { + let cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"; + let topic_name = content_availability_topic_name(cid); + let endpoint = Endpoint::builder(iroh::endpoint::presets::N0) + .secret_key(iroh::SecretKey::from_bytes(&local_sk.to_bytes())) + .bind() + .await + .unwrap(); + let memory_lookup = MemoryLookup::new(); + endpoint + .address_lookup() + .unwrap() + .add(memory_lookup.clone()); + let gossip = spawn_carrier_gossip(&endpoint); + let state = Arc::new(Mutex::new(GossipState::new( + endpoint.clone(), + gossip, + memory_lookup, + Some(local_sk), + Some(local_did.clone()), + ))); + { + let mut guard = state.lock().await; + guard.joined_topics.insert(topic_name.clone()); + guard + .buffers + .lock() + .await + .insert(topic_name, test_topic_buffer([], 0)); + } + let data_dir = tempfile::tempdir().unwrap(); + crate::sources::save_trusted_sources( + data_dir.path(), + &crate::sources::TrustedSourcesConfig { + schema: "elastos.trusted-sources/v1".to_string(), + default_source: "official".to_string(), + sources: vec![TrustedSource { + name: "official".to_string(), + publisher_dids: vec![catalog_publisher_did], + channel: "stable".to_string(), + discovery_uri: "https://example.invalid/discovery".to_string(), + connect_ticket: stale_ticket, + gateways: vec![gateway], + install_path: String::new(), + installed_version: String::new(), + head_cid: cid.to_string(), + publisher_node_id, + ipns_name: String::new(), + }], + }, + ) + .unwrap(); + let registry = Arc::new(ProviderRegistry::new()); + let invoker = Arc::new(MockCarrierProviderPlaneInvoker { + fail_tickets, + ..Default::default() + }); + registry.set_carrier_invoker(invoker.clone()).await; + let provider = CarrierAvailabilityProvider::with_provider_registry_and_data_dir( + state, + Arc::downgrade(®istry), + data_dir.path().to_path_buf(), + ) + .with_discovery_wait(Duration::from_millis(400)); + (provider, invoker, registry, data_dir, cid.to_string()) + } + + #[tokio::test] + async fn test_carrier_availability_fetch_refreshes_stale_ticket_after_holder_restart() { + let (local_sk, local_did) = elastos_identity::derive_did(&[48u8; 32]); + let (_remote_sk, _holder_did, stale_ticket, live_ticket) = + holder_tickets_for_secret([49u8; 32]); + let public = + iroh::SecretKey::from_bytes(&elastos_identity::derive_did(&[49u8; 32]).0.to_bytes()) + .public(); + let catalog_did = "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string(); + let (gateway, _server) = serve_publisher_bootstrap(publisher_bootstrap_document( + &live_ticket, + &public.to_string(), + "publisher", + "elastos.carrier.bootstrap/v1", + )) + .await; + let (provider, invoker, _registry, data_dir, cid) = + availability_provider_with_trusted_source( + stale_ticket.clone(), + public.to_string(), + catalog_did.clone(), + gateway, + local_sk, + local_did, + vec![stale_ticket.clone()], + ) + .await; + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "ok", "{response}"); + let requests = invoker.requests.lock().await; + assert!( + requests + .iter() + .any(|request| request["ticket"] == stale_ticket), + "{requests:?}" + ); + assert!( + requests + .iter() + .any(|request| request["ticket"] == live_ticket), + "{requests:?}" + ); + let stored = crate::sources::load_trusted_sources(data_dir.path()).unwrap(); + let source = stored.default_source().unwrap(); + assert_eq!(source.connect_ticket, live_ticket); + assert_eq!(source.publisher_dids, vec![catalog_did]); + assert_eq!( + source.head_cid, + "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" + ); + } + + #[tokio::test] + async fn test_carrier_availability_fetch_rejects_identity_mismatch_and_malformed_refresh() { + let (local_sk, local_did) = elastos_identity::derive_did(&[50u8; 32]); + let (_remote_sk, _holder_did, stale_ticket, _live_ticket) = + holder_tickets_for_secret([51u8; 32]); + let public = + iroh::SecretKey::from_bytes(&elastos_identity::derive_did(&[51u8; 32]).0.to_bytes()) + .public(); + let foreign = iroh::SecretKey::from_bytes(&[52u8; 32]).public(); + let foreign_ticket = encode_ticket_for(iroh::EndpointAddr::from(foreign)); + let catalog_did = "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string(); + let (gateway, _server) = serve_publisher_bootstrap(publisher_bootstrap_document( + &foreign_ticket, + &foreign.to_string(), + "publisher", + "elastos.carrier.bootstrap/v1", + )) + .await; + let (provider, _, _registry, data_dir, cid) = availability_provider_with_trusted_source( + stale_ticket.clone(), + public.to_string(), + catalog_did.clone(), + gateway, + local_sk, + local_did, + vec![stale_ticket.clone()], + ) + .await; + let response = provider + .send_raw(&serde_json::json!({ + "op": "fetch", + "cid": cid, + "path": crate::content::CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .unwrap(); + assert_eq!(response["status"], "error", "{response}"); + assert_eq!(response["code"], "carrier_fetch_failed"); + assert!( + response["message"] + .as_str() + .unwrap_or_default() + .contains("trusted-source bootstrap refresh rejected"), + "{response}" + ); + let stored = crate::sources::load_trusted_sources(data_dir.path()).unwrap(); + let source = stored.default_source().unwrap(); + assert_eq!(source.connect_ticket, stale_ticket); + assert_eq!(source.publisher_dids, vec![catalog_did]); + + let missing_schema = SourceCarrierBootstrapDocument { + schema: String::new(), + role: String::new(), + ticket: foreign_ticket, + node_id: foreign.to_string(), + did: None, + }; + assert_eq!( + verified_source_bootstrap_ticket(source, &missing_schema), + None + ); + } + + #[test] + fn decode_ticket_endpoints_accepts_installer_n0_relay_shape() { + let secret = iroh::SecretKey::from_bytes(&[45u8; 32]); + let ticket_json = serde_json::json!({ + "topic": serde_json::Value::Null, + "endpoints": [{ + "id": secret.public().to_string(), + "addrs": [ + { "Relay": "https://euc1-1.relay.n0.iroh.link./" }, + { "Ip": "127.0.0.1:9" } + ] + }] + }); + let mut encoded = + data_encoding::BASE32_NOPAD.encode(&serde_json::to_vec(&ticket_json).unwrap()); + encoded.make_ascii_lowercase(); + let endpoints = decode_ticket_endpoints(&encoded); + assert_eq!( + endpoints.len(), + 1, + "installer connect tickets must decode to one Carrier endpoint" + ); + assert_eq!(endpoints[0].id, secret.public()); + assert_eq!( + trusted_source_replica_did(&TrustedSource { + name: "official".to_string(), + publisher_dids: vec![ + "did:key:z6Mkjg9duxEF2nskEPR9F38eSfrY6F1GyWUq5aDbgsMqgcjA".to_string() + ], + channel: "stable".to_string(), + discovery_uri: String::new(), + connect_ticket: encoded, + gateways: Vec::new(), + install_path: String::new(), + installed_version: String::new(), + head_cid: String::new(), + publisher_node_id: secret.public().to_string(), + ipns_name: String::new(), + }) + .as_deref(), + Some(public_key_to_did(&secret.public()).unwrap().as_str()) + ); + } + #[test] fn test_connect_marks_bootstrap_join_peers() { let secret = iroh::SecretKey::from_bytes(&[12u8; 32]); diff --git a/elastos/crates/elastos-server/src/chat_cmd.rs b/elastos/crates/elastos-server/src/chat_cmd.rs index 36614605b..f52afcb56 100644 --- a/elastos/crates/elastos-server/src/chat_cmd.rs +++ b/elastos/crates/elastos-server/src/chat_cmd.rs @@ -3,14 +3,12 @@ use std::sync::Arc; use std::time::{Duration, Instant}; use std::{io::IsTerminal, io::Read, io::Write}; -use elastos_server::sources::{ - default_data_dir, load_trusted_sources, normalize_gateways, TrustedSource, TrustedSourcesConfig, -}; +use elastos_server::carrier::live_source_bootstrap_ticket; +use elastos_server::sources::{default_data_dir, load_trusted_sources, TrustedSourcesConfig}; use sha2::{Digest, Sha256}; const CHAT_TOPIC: &str = "#general"; const PRESENCE_ATTACH_RETRY_BACKOFF: Duration = Duration::from_secs(12); -const SOURCE_BOOTSTRAP_TIMEOUT: Duration = Duration::from_secs(3); #[derive(Clone, Debug, Default)] struct AttachedChatIdentity { @@ -47,15 +45,6 @@ struct ChatPresenceAnnouncement { ts: u64, } -#[derive(Debug, serde::Deserialize)] -struct SourceCarrierBootstrapDocument { - schema: String, - role: String, - ticket: String, - node_id: String, -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq)] enum NativeChatInputMode { Line, StdinTty, @@ -980,59 +969,6 @@ async fn resolve_chat_bootstrap( .map(ChatBootstrap::TrustedSourceSeed) } -async fn live_source_bootstrap_ticket(source: &TrustedSource) -> Option { - let client = reqwest::Client::builder() - .timeout(SOURCE_BOOTSTRAP_TIMEOUT) - .build() - .ok()?; - - for gateway in normalize_gateways(&source.gateways) { - let url = format!("{gateway}/.well-known/elastos/carrier-bootstrap.json?role=publisher"); - let Ok(response) = client.get(url).send().await else { - continue; - }; - if !response.status().is_success() { - continue; - } - let Ok(document) = response.json::().await else { - continue; - }; - if let Some(ticket) = verified_source_bootstrap_ticket(source, &document) { - return Some(ticket); - } - } - None -} - -fn verified_source_bootstrap_ticket( - source: &TrustedSource, - document: &SourceCarrierBootstrapDocument, -) -> Option { - if document.schema != "elastos.carrier.bootstrap/v1" - || document.role != "publisher" - || document.ticket.trim().is_empty() - || document.node_id.trim().is_empty() - { - return None; - } - - let expected_node_id = source.publisher_node_id.trim(); - if !expected_node_id.is_empty() && document.node_id != expected_node_id { - return None; - } - - let endpoints = elastos_server::carrier::decode_ticket_endpoints(&document.ticket); - if endpoints.is_empty() - || endpoints - .iter() - .any(|endpoint| endpoint.id.to_string() != document.node_id) - { - return None; - } - - Some(document.ticket.trim().to_string()) -} - fn presence_attach_retry_pending( retry_after: &HashMap, did: &str, @@ -1859,6 +1795,9 @@ pub(crate) async fn request_attached_capability( #[cfg(test)] mod tests { use super::*; + use elastos_server::carrier::{ + verified_source_bootstrap_ticket, SourceCarrierBootstrapDocument, + }; use elastos_server::sources::{TrustedSource, TrustedSourcesConfig}; use std::io::Cursor; use std::sync::Mutex; @@ -1902,6 +1841,7 @@ mod tests { role: "publisher".to_string(), ticket: ticket.clone(), node_id, + ..Default::default() }; assert_eq!( @@ -1920,6 +1860,7 @@ mod tests { role: "publisher".to_string(), ticket, node_id, + ..Default::default() }; assert_eq!(verified_source_bootstrap_ticket(&source, &document), None); @@ -1934,6 +1875,7 @@ mod tests { role: "runtime".to_string(), ticket, node_id, + ..Default::default() }; assert_eq!(verified_source_bootstrap_ticket(&source, &document), None); diff --git a/elastos/crates/elastos-server/src/content.rs b/elastos/crates/elastos-server/src/content.rs index ef9b7a662..41fb470ed 100644 --- a/elastos/crates/elastos-server/src/content.rs +++ b/elastos/crates/elastos-server/src/content.rs @@ -100,6 +100,16 @@ const OBJECT_MANIFEST_PATH: &str = "_elastos_object.json"; const SEALED_OBJECT_PATH: &str = "sealed.json"; pub const CONTENT_OBJECT_MANIFEST_PATH: &str = OBJECT_MANIFEST_PATH; +const MAX_LOCAL_OBJECT_DIRS: usize = 16; +const MAX_LOCAL_OBJECT_WALK_DEPTH: usize = 8; +const LOCAL_OBJECT_WALK_SKIP: &[&str] = &[ + "ipfs-repo", + "logs", + "bin", + "managed-runtimes", + "target", + ".git", +]; pub struct ContentProvider { data_dir: PathBuf, @@ -3000,6 +3010,11 @@ impl Provider for ContentProvider { } } +fn local_backend_needs_prepare(err: &ProviderError) -> bool { + let text = err.to_string(); + text.contains("Bounded content read failed") || text.contains("bounded_read_failed") +} + impl ContentProvider { async fn invoke_provider( &self, @@ -3068,15 +3083,26 @@ impl ContentProvider { let registry = self.registry()?; let transfer = ContentFetchTransfer::from_request(request)?; + let local_only = request.get("local_only").and_then(|value| value.as_bool()) == Some(true); let result = match self .fetch_from_local_backend(®istry, cid, path, &transfer) .await { Ok(result) => result, - Err(local_err) - if request.get("local_only").and_then(|value| value.as_bool()) == Some(true) => - { - return Err(local_err); + Err(local_err) if local_only => { + if local_backend_needs_prepare(&local_err) + && registry.prepare_local_ipfs_backend().await.is_ok() + { + match self + .fetch_from_local_backend(®istry, cid, path, &transfer) + .await + { + Ok(result) => result, + Err(err) => return Err(err), + } + } else { + return Err(local_err); + } } Err(local_err) => match self .fetch_from_availability_provider(®istry, cid, path, &transfer) @@ -3778,7 +3804,7 @@ impl ContentProvider { )); } - let outcome = AvailabilityOutcome { + let local_outcome = AvailabilityOutcome { provider: "ipfs-provider".to_string(), policy: "carrier_object_import".to_string(), status: "local_pinned".to_string(), @@ -3791,6 +3817,23 @@ impl ContentProvider { repair_graph: local_repair_graph_json(), abuse_controls: local_abuse_controls_json(), }; + let outcome = self + .ensure_network_availability( + ®istry, + cid, + request, + &local_outcome, + AvailabilityRequestContext { + object_did: request.get("object_did").and_then(|value| value.as_str()), + publisher_did: publisher_did.as_deref(), + accounting_observation: ContentAccountingObservation { + files: Some(file_count as u64), + bytes: Some(total_bytes as u64), + }, + }, + ) + .await? + .unwrap_or(local_outcome); let object_did = request .get("object_did") .and_then(|value| value.as_str()) @@ -3973,11 +4016,17 @@ impl ContentProvider { ) .await?; - let (status, policy, replicas, reason) = if ipfs_response + let pin_failed = ipfs_response .get("status") .and_then(|status| status.as_str()) - == Some("error") - { + == Some("error"); + let imported_local_object = if pin_failed { + self.import_complete_local_object_matching(®istry, cid) + .await? + } else { + false + }; + let (status, policy, replicas, reason) = if pin_failed && !imported_local_object { ( "repair_needed", failure_policy, @@ -3987,6 +4036,8 @@ impl ContentProvider { .and_then(|message| message.as_str()) .map(str::to_string), ) + } else if imported_local_object { + ("local_pinned", "local_object_import", 1, None) } else { ("local_pinned", success_policy, 1, None) }; @@ -4075,6 +4126,101 @@ impl ContentProvider { }))) } + async fn import_complete_local_object_matching( + &self, + registry: &ProviderRegistry, + cid: &str, + ) -> Result { + for dir in complete_local_object_dirs(&self.data_dir) { + match self.import_local_object_dir(registry, &dir).await { + Ok(imported_cid) if imported_cid == cid => return Ok(true), + Ok(imported_cid) => { + let _ = self + .invoke_provider( + registry, + "ipfs", + "unpin", + json!({ + "op": "unpin", + "cid": imported_cid, + }), + ProviderTransfer::Json, + ) + .await; + } + Err(err) => { + tracing::debug!( + dir = %dir.display(), + error = %err, + "content: local object import skipped" + ); + } + } + } + Ok(false) + } + + async fn import_local_object_dir( + &self, + registry: &ProviderRegistry, + dir: &Path, + ) -> Result { + let files = local_object_dir_files(dir)?; + let ipfs_response = self + .invoke_provider( + registry, + "ipfs", + "add_directory", + json!({ + "op": "add_directory", + "files": files, + "pin": true, + }), + ProviderTransfer::Json, + ) + .await?; + provider_response_ok(&ipfs_response, "content local object import")?; + provider_response_cid(&ipfs_response).map_err(|err| { + ProviderError::Provider(format!("content local object import missing CID: {err}")) + }) + } + + pub async fn restore_complete_local_objects(&self) { + let Ok(registry) = self.registry() else { + return; + }; + for dir in complete_local_object_dirs(&self.data_dir) { + match self.import_local_object_dir(®istry, &dir).await { + Ok(cid) => match self + .ensure(&json!({ + "cid": cid, + "availability_requirements": { + "min_replicas": 1, + "max_replicas": 1, + "require_live_multi_peer_proof": false + } + })) + .await + { + Ok(_) => tracing::info!( + cid = %cid, + "content: restored complete local object replica" + ), + Err(err) => tracing::warn!( + cid = %cid, + error = %err, + "content: restore local object replica failed" + ), + }, + Err(err) => tracing::debug!( + dir = %dir.display(), + error = %err, + "content: restore local object import skipped" + ), + } + } + } + async fn ensure_network_availability( &self, registry: &ProviderRegistry, @@ -8476,6 +8622,120 @@ fn validate_content_path(path: &str) -> Result<(), String> { Ok(()) } +fn complete_local_object_dirs(data_dir: &Path) -> Vec { + let mut found = Vec::new(); + walk_local_object_dirs(data_dir, 0, &mut found); + if let Some(owner) = managed_home_owner_data_dir() { + if owner != data_dir { + walk_local_object_dirs(&owner, 0, &mut found); + } + } + found +} + +fn managed_home_owner_data_dir() -> Option { + if let Ok(raw) = std::env::var("ELASTOS_MANAGED_GATEWAY_OWNER") { + if let Ok(value) = serde_json::from_str::(&raw) { + if let Some(dir) = value.get("data_dir").and_then(|v| v.as_str()) { + let path = PathBuf::from(dir); + if path.is_absolute() && path.is_dir() { + return Some(path); + } + } + } + } + std::env::var("ELASTOS_HOME_LAUNCH_TRUSTED_AUTH_DATA_DIR") + .ok() + .map(PathBuf::from) + .filter(|path| path.is_absolute() && path.is_dir()) +} + +fn walk_local_object_dirs(dir: &Path, depth: usize, found: &mut Vec) { + if found.len() >= MAX_LOCAL_OBJECT_DIRS || depth > MAX_LOCAL_OBJECT_WALK_DEPTH { + return; + } + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + for entry in entries.flatten() { + if found.len() >= MAX_LOCAL_OBJECT_DIRS { + return; + } + let path = entry.path(); + let name = entry.file_name(); + let name = name.to_string_lossy(); + if LOCAL_OBJECT_WALK_SKIP + .iter() + .any(|skip| *skip == name.as_ref()) + { + continue; + } + if !path.is_dir() { + continue; + } + if path.join(OBJECT_MANIFEST_PATH).is_file() { + if local_object_dir_is_complete(&path) { + found.push(path); + } + continue; + } + walk_local_object_dirs(&path, depth + 1, found); + } +} + +fn local_object_dir_is_complete(dir: &Path) -> bool { + let Ok(bytes) = std::fs::read(dir.join(OBJECT_MANIFEST_PATH)) else { + return false; + }; + let Ok(manifest) = serde_json::from_slice::(&bytes) else { + return false; + }; + if manifest.schema != OBJECT_MANIFEST_SCHEMA || manifest.files.is_empty() { + return false; + } + manifest.files.iter().all(|file| { + if validate_content_path(&file.path).is_err() { + return false; + } + let path = dir.join(&file.path); + let Ok(bytes) = std::fs::read(&path) else { + return false; + }; + bytes.len() as u64 == file.size + && format!("{:x}", sha2::Sha256::digest(&bytes)) == file.sha256 + }) +} + +fn local_object_dir_files(dir: &Path) -> Result, ProviderError> { + let manifest_path = dir.join(OBJECT_MANIFEST_PATH); + let manifest_bytes = std::fs::read(&manifest_path)?; + let manifest: ContentObjectManifest = + serde_json::from_slice(&manifest_bytes).map_err(|err| { + ProviderError::Provider(format!( + "content local object manifest decode failed: {err}" + )) + })?; + let mut files = Vec::with_capacity(manifest.files.len().saturating_add(1)); + for file in &manifest.files { + if validate_content_path(&file.path).is_err() { + return Err(ProviderError::Provider(format!( + "content local object path {} is invalid", + file.path + ))); + } + let bytes = std::fs::read(dir.join(&file.path))?; + files.push(json!({ + "path": file.path, + "data": base64::engine::general_purpose::STANDARD.encode(bytes), + })); + } + files.push(json!({ + "path": OBJECT_MANIFEST_PATH, + "data": base64::engine::general_purpose::STANDARD.encode(manifest_bytes), + })); + Ok(files) +} + fn with_directory_object_manifest( files: Value, kind: &str, @@ -8885,6 +9145,7 @@ mod tests { pinned: Mutex>, pin_error: Mutex>, unpinned: Mutex>, + bounded_read_not_ready: Mutex, } struct MockAvailabilityProvider { @@ -8937,6 +9198,17 @@ mod tests { .and_then(|path| path.as_str()) .unwrap_or("") .to_string(); + if request + .get("bounded_read") + .and_then(|value| value.as_bool()) + == Some(true) + && *self.bounded_read_not_ready.lock().await + { + return Ok(provider_error( + "bounded_read_failed", + "Bounded content read failed", + )); + } if self .missing_paths .lock() @@ -9201,6 +9473,7 @@ mod tests { pinned: Mutex::new(Vec::new()), pin_error: Mutex::new(None), unpinned: Mutex::new(Vec::new()), + bounded_read_not_ready: Mutex::new(false), }); registry .register_sub_provider("ipfs", ipfs.clone()) @@ -11262,6 +11535,112 @@ mod tests { ); } + #[tokio::test] + async fn content_ensure_imports_complete_local_object_when_pin_fails() { + let (data_dir, registry, ipfs, content) = registry_with_content_and_ipfs().await; + *ipfs.pin_error.lock().await = Some("not in local repo".to_string()); + let body = b"hello object"; + let digest = format!("{:x}", sha2::Sha256::digest(body)); + let object_dir = data_dir.path().join("held-object"); + std::fs::create_dir_all(&object_dir).unwrap(); + std::fs::write(object_dir.join("index.md"), body).unwrap(); + std::fs::write( + object_dir.join(OBJECT_MANIFEST_PATH), + serde_json::to_vec(&json!({ + "schema": OBJECT_MANIFEST_SCHEMA, + "kind": "directory", + "content_digest": format!("sha256:{digest}"), + "files": [{ + "path": "index.md", + "sha256": digest, + "size": body.len() as u64 + }], + })) + .unwrap(), + ) + .unwrap(); + let availability = Arc::new(MockAvailabilityProvider { + requests: Mutex::new(Vec::new()), + response: Mutex::new(provider_ok(json!({ + "availability": { + "status": "carrier_announced", + "provider": "carrier-availability", + "policy": "network_default", + "replicas": 1, + "peer_selection": { + "mode": "carrier_topic", + "strategy": "local_holder", + "topic": "elastos.content.availability" + }, + "quota": { + "policy": "local_holder", + "status": "within_quota" + }, + "repair_worker": { + "scheduled": false, + "status": "idle" + } + } + }))), + }); + registry.register(availability.clone()).await; + + let response = content + .send_raw(&json!({ + "op": "ensure", + "cid": TEST_CID, + })) + .await + .unwrap(); + + assert_eq!(response["status"], "ok", "{response}"); + assert_ne!( + response["data"]["availability"]["status"], "repair_needed", + "complete local object must become a Content replica: {response}" + ); + assert_eq!(response["data"]["availability"]["replicas"], 1); + assert_eq!(ipfs.added_directories.lock().await.len(), 1); + let sent = availability.requests.lock().await; + assert_eq!(sent.len(), 1); + assert_eq!(sent[0]["op"], "ensure"); + assert_eq!(sent[0]["local"]["replicas"], 1); + assert_eq!(sent[0]["local"]["policy"], "local_object_import"); + assert_eq!( + response["data"]["availability"]["status"], + "carrier_announced" + ); + } + + #[tokio::test] + async fn content_restore_imports_complete_local_objects() { + let (data_dir, _registry, ipfs, content) = registry_with_content_and_ipfs().await; + let body = b"hello object"; + let digest = format!("{:x}", sha2::Sha256::digest(body)); + let object_dir = data_dir.path().join("held-object"); + std::fs::create_dir_all(&object_dir).unwrap(); + std::fs::write(object_dir.join("index.md"), body).unwrap(); + std::fs::write( + object_dir.join(OBJECT_MANIFEST_PATH), + serde_json::to_vec(&json!({ + "schema": OBJECT_MANIFEST_SCHEMA, + "kind": "directory", + "content_digest": format!("sha256:{digest}"), + "files": [{ + "path": "index.md", + "sha256": digest, + "size": body.len() as u64 + }], + })) + .unwrap(), + ) + .unwrap(); + + content.restore_complete_local_objects().await; + + assert_eq!(ipfs.added_directories.lock().await.len(), 1); + assert_eq!(ipfs.pinned.lock().await.as_slice(), [TEST_CID.to_string()]); + } + #[tokio::test] async fn content_publish_file_wraps_ipfs_bytes_with_receipt() { let (_data_dir, registry, ipfs, content) = registry_with_content_and_ipfs().await; @@ -11882,6 +12261,93 @@ mod tests { assert!(registry.get("availability").await.is_none()); } + #[tokio::test] + async fn content_fetch_rejects_max_bytes_for_a_non_index_path_and_forwards_declared_range() { + // Exclusive LICENSE miss: max_bytes is the whole-index metadata bound. + // A non-index file uses the signed size as a closed 64 KiB range. + let (_root, registry, ipfs, content) = registry_with_content_and_ipfs().await; + let err = content + .send_raw(&json!({ + "op": "fetch", + "cid": TEST_CID, + "path": "docs/readme.md", + "bounded_read": true, + "max_bytes": 65536, + "transfer": "stream" + })) + .await + .expect_err("max_bytes stays the whole-index bound"); + assert!( + err.to_string() + .contains("complete metadata requires a local bounded whole index"), + "{err}" + ); + assert!(ipfs.requests.lock().await.is_empty()); + assert!(registry.get("availability").await.is_none()); + + ipfs.missing_paths + .lock() + .await + .push("docs/readme.md".to_string()); + let encode = |bytes: &[u8]| base64::engine::general_purpose::STANDARD.encode(bytes); + let availability = Arc::new(MockAvailabilityProvider { + requests: Mutex::new(Vec::new()), + response: Mutex::new(provider_ok(json!({ + "data": encode(b"89ab"), + "availability": { + "status": "network_available", + "provider": "mock-availability", + "policy": "carrier_provider_invoke", + "replicas": 1 + } + }))), + }); + registry.register(availability.clone()).await; + let response = registry + .invoke_provider(ProviderInvocation { + source: "runtime-model-preparation".into(), + target: "content".into(), + op: "fetch".into(), + request: json!({ + "op": "fetch", + "cid": TEST_CID, + "path": "docs/readme.md", + "bounded_read": true, + "range": { "start": 0, "end": 3 }, + "transfer": "stream" + }), + transfer: ProviderTransfer::Stream, + range: None, + progress: None, + transport: ProviderInvocationTransport::Local, + }) + .await + .unwrap(); + assert_eq!( + decode_test_stream_payload(&response["data"]["stream"]), + b"89ab" + ); + let sent = availability.requests.lock().await; + assert_eq!(sent.len(), 1); + assert_eq!(sent[0]["path"], "docs/readme.md"); + assert_eq!(sent[0]["bounded_read"], true); + assert_eq!(sent[0]["range"], json!({ "start": 0, "end": 3 })); + assert!(sent[0].get("max_bytes").is_none()); + assert!(sent[0]["_runtime_invocation"]["range"].is_null()); + } + + #[tokio::test] + async fn fetch_model_part_requires_a_declared_range_for_a_non_index_path() { + let (_root, registry, _ipfs, _content) = registry_with_content_and_ipfs().await; + let err = fetch_model_part(®istry, TEST_CID, "docs/readme.md", None) + .await + .expect_err("non-index files use a declared range"); + assert!( + err.to_string().contains("complete model index required"), + "{err}" + ); + } + #[tokio::test] async fn content_fetch_ranges_availability_provider_when_local_backend_misses() { let (_data_dir, registry, ipfs, content) = registry_with_content_and_ipfs().await; @@ -12049,6 +12515,52 @@ mod tests { assert!(availability.requests.lock().await.is_empty()); } + #[test] + fn local_backend_needs_prepare_matches_holder_cold_bounded_read() { + assert!(local_backend_needs_prepare(&ProviderError::Provider( + "content fetch failed: Bounded content read failed".into(), + ))); + assert!(local_backend_needs_prepare(&ProviderError::Provider( + "bounded_read_failed".into(), + ))); + assert!(!local_backend_needs_prepare(&ProviderError::Provider( + "content fetch failed: mock content path missing".into(), + ))); + } + + #[tokio::test] + async fn content_fetch_local_only_cold_bounded_read_does_not_use_availability() { + let (_data_dir, registry, ipfs, content) = registry_with_content_and_ipfs().await; + let availability = Arc::new(MockAvailabilityProvider { + requests: Mutex::new(Vec::new()), + response: Mutex::new(provider_ok(json!({ + "data": base64::engine::general_purpose::STANDARD.encode(b"remote content"), + }))), + }); + registry.register(availability.clone()).await; + *ipfs.bounded_read_not_ready.lock().await = true; + let err = content + .send_raw(&json!({ + "op": "fetch", + "cid": TEST_CID, + "path": CONTENT_OBJECT_MANIFEST_PATH, + "bounded_read": true, + "max_bytes": 65536, + "local_only": true, + })) + .await + .expect_err("cold bounded local_only fetch must fail closed"); + assert!( + err.to_string().contains("Bounded content read failed"), + "{err}" + ); + assert!(availability.requests.lock().await.is_empty()); + assert!( + ipfs.requests.lock().await.len() <= 2, + "one bounded miss plus at most one prepare retry" + ); + } + #[tokio::test] async fn content_prepare_data_capsule_materializes_verified_manifest_files() { let (_data_dir, registry, ipfs, _content) = registry_with_content_and_ipfs().await; diff --git a/elastos/crates/elastos-server/src/server_infra.rs b/elastos/crates/elastos-server/src/server_infra.rs index 9a63fe47a..aff57e7cd 100644 --- a/elastos/crates/elastos-server/src/server_infra.rs +++ b/elastos/crates/elastos-server/src/server_infra.rs @@ -452,16 +452,19 @@ pub(crate) async fn setup_control_plane_infrastructure() -> anyhow::Result, data_dir: &Path, -) -> anyhow::Result<()> { +) -> anyhow::Result> { let content_provider = Arc::new(ContentProvider::new( data_dir.to_path_buf(), Arc::downgrade(provider_registry), )); provider_registry.register(content_provider.clone()).await; provider_registry - .register_sub_provider("content", content_provider) + .register_sub_provider("content", content_provider.clone()) .await - .map_err(|err| anyhow::anyhow!("failed to register elastos://content sub-provider: {err}")) + .map_err(|err| { + anyhow::anyhow!("failed to register elastos://content sub-provider: {err}") + })?; + Ok(content_provider) } /// Spawn the ipfs-provider capsule and register `elastos://ipfs`, the block @@ -653,9 +656,13 @@ async fn setup_server_infrastructure_impl( let mut managed_host_processes = Vec::new(); let mut external_availability_registered = false; let mut carrier_service = None; - if let Err(err) = register_content_plane(&provider_registry, &data_dir).await { - tracing::warn!("{}", err); - } + let content_plane = match register_content_plane(&provider_registry, &data_dir).await { + Ok(provider) => Some(provider), + Err(err) => { + tracing::warn!("{}", err); + None + } + }; provider_registry .register(Arc::new(DocumentsProvider::new( data_dir.clone(), @@ -1447,20 +1454,25 @@ async fn setup_server_infrastructure_impl( tracing::warn!("Failed to register Carrier gossip provider: {}", e); } if !external_availability_registered { - let availability_provider: Arc = - Arc::new( + let availability_provider = Arc::new( elastos_server::carrier::CarrierAvailabilityProvider::with_provider_registry_data_dir_and_peer_attestation_exchange_config( carrier_node.gossip_state.clone(), Arc::downgrade(&provider_registry), data_dir.clone(), carrier_peer_attestation_exchange_config_from_env(), )); + availability_provider + .restore_persisted_local_holder_announcements() + .await; if let Err(e) = provider_registry .register_sub_provider("availability", availability_provider) .await { tracing::warn!("Failed to register Carrier availability provider: {}", e); } + if let Some(content) = content_plane.as_ref() { + content.restore_complete_local_objects().await; + } } carrier_service = Some(elastos_server::carrier::CarrierRuntimeService::new( carrier_node, From 20ac3f628aea683e2cefff7cf7c056852af3365c Mon Sep 17 00:00:00 2001 From: Anders Alm Date: Thu, 17 Sep 2026 13:41:39 -0500 Subject: [PATCH 350/535] fix(setup): clear group-write on model-engine install parents Install and fetch strip group/other write from each install_path directory under the data dir. Identity and verification stay read-only. Verification derives the receipt bundle root from configured install_path so a nested binary_path such as bin/llama-server still matches the receipt. Private activation stays on debug. Verification: cargo test -p elastos-server --lib verified_local_model_engine_accepts_a_nested_binary_path cargo test -p elastos-server --lib group_writable cargo test -p elastos-server --lib protect_engine_parents_rejects_symlink python3 scripts/fetch/fetch-model-smoke.py git diff --check --- .../src/api/capsule_inventory/preparation.rs | 10 + elastos/crates/elastos-server/src/setup.rs | 336 +++++++++++++++++- scripts/fetch/fetch-model.sh | 49 +++ 3 files changed, 389 insertions(+), 6 deletions(-) diff --git a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs index 09d462406..327fb2233 100644 --- a/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs +++ b/elastos/crates/elastos-server/src/api/capsule_inventory/preparation.rs @@ -3040,6 +3040,16 @@ mod tests { std::fs::set_permissions(bundle.join(name), std::fs::Permissions::from_mode(mode)) .unwrap(); } + let mut parent = bundle.clone(); + for _ in Path::new(relative).components() { + let mode = std::fs::metadata(&parent).unwrap().permissions().mode(); + std::fs::set_permissions( + &parent, + std::fs::Permissions::from_mode((mode & 0o7777) & !0o022), + ) + .unwrap(); + parent = parent.parent().unwrap().to_path_buf(); + } std::fs::set_permissions(&bundle, std::fs::Permissions::from_mode(0o500)).unwrap(); EngineFixture(bundle) } diff --git a/elastos/crates/elastos-server/src/setup.rs b/elastos/crates/elastos-server/src/setup.rs index 3e461a41f..e3ce2e899 100644 --- a/elastos/crates/elastos-server/src/setup.rs +++ b/elastos/crates/elastos-server/src/setup.rs @@ -1408,6 +1408,9 @@ fn write_cache_metadata( let (version, checksum, binary) = local_model_engine_receipt_args(component, platform_info.unwrap())?; local_model_engine_receipt::write(dest, version, platform, checksum, binary)?; + let install_path = resolve_install_path(component, platform_info) + .ok_or_else(|| anyhow::anyhow!("local model engine install path is unavailable"))?; + protect_local_model_engine_install_parents(dest, install_path)?; return Ok(()); } @@ -1464,6 +1467,74 @@ pub(crate) struct LocalModelEngineIdentity { pub receipt_sha256: String, } +/// Clear group/other write on install_path directories under the data dir. +/// Call this from the owned install/repair path only. Verification stays read-only. +#[cfg(unix)] +fn protect_local_model_engine_install_parents( + bundle: &Path, + install_path: &str, +) -> anyhow::Result<()> { + use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _}; + use std::os::unix::io::AsRawFd; + let relative = Path::new(install_path); + anyhow::ensure!( + !install_path.is_empty() + && install_path.len() <= 4096 + && relative + .components() + .all(|part| matches!(part, std::path::Component::Normal(_))), + "local model engine install path is invalid" + ); + let mut data_dir = if bundle.is_absolute() { + bundle.to_path_buf() + } else { + std::env::current_dir()?.join(bundle) + }; + anyhow::ensure!( + data_dir.ends_with(relative), + "model engine parent is not protected" + ); + for _ in relative.components() { + data_dir = data_dir + .parent() + .map(Path::to_path_buf) + .ok_or_else(|| anyhow::anyhow!("model engine parent is not protected"))?; + } + let data_dir = fs::canonicalize(data_dir)?; + let mut path = data_dir.clone(); + for part in relative.components() { + path.push(part.as_os_str()); + anyhow::ensure!( + path.starts_with(&data_dir), + "model engine parent is not protected" + ); + let dir = fs::OpenOptions::new() + .read(true) + .custom_flags(libc::O_CLOEXEC | libc::O_DIRECTORY | libc::O_NOFOLLOW) + .open(&path) + .map_err(|_| anyhow::anyhow!("model engine parent is not protected"))?; + let metadata = dir.metadata()?; + anyhow::ensure!( + metadata.is_dir() && metadata.uid() == unsafe { libc::geteuid() }, + "model engine parent is not protected" + ); + let mode = metadata.mode(); + if mode & 0o022 != 0 { + let rc = + unsafe { libc::fchmod(dir.as_raw_fd(), (mode as libc::mode_t & 0o7777) & !0o022) }; + anyhow::ensure!(rc == 0, "model engine parent is not protected"); + } + let after = dir.metadata()?; + anyhow::ensure!( + after.is_dir() + && after.uid() == unsafe { libc::geteuid() } + && after.mode() & 0o022 == 0, + "model engine parent is not protected" + ); + } + Ok(()) +} + #[cfg(unix)] pub(crate) fn local_model_engine_receipt_identity( data_dir: &Path, @@ -1495,7 +1566,8 @@ pub(crate) fn local_model_engine_receipt_identity( bundle.push(part.as_os_str()); let metadata = fs::symlink_metadata(&bundle)?; anyhow::ensure!( - metadata.is_dir() + !metadata.file_type().is_symlink() + && metadata.is_dir() && metadata.uid() == unsafe { libc::geteuid() } && metadata.mode() & 0o022 == 0, "model engine parent is not protected" @@ -1526,13 +1598,34 @@ pub(crate) fn verified_local_model_engine( manifest: &ComponentsManifest, ) -> anyhow::Result { let identity = local_model_engine_receipt_identity(data_dir, manifest)?; - let component = manifest.external.get("llama-server").unwrap(); + let component = manifest + .external + .get("llama-server") + .ok_or_else(|| anyhow::anyhow!("local model engine is unavailable"))?; let platform = detect_platform(); - let info = component.platforms.get(&platform).unwrap(); + let info = component + .platforms + .get(&platform) + .ok_or_else(|| anyhow::anyhow!("local model engine platform is unavailable"))?; let (version, archive, binary) = local_model_engine_receipt_args(component, info)?; - let bundle = data_dir - .canonicalize()? - .join(resolve_install_path(component, Some(info)).unwrap()); + let install_path = resolve_install_path(component, Some(info)) + .ok_or_else(|| anyhow::anyhow!("local model engine install path is unavailable"))?; + let relative = Path::new(install_path); + anyhow::ensure!( + !install_path.is_empty() + && install_path.len() <= 4096 + && relative + .components() + .all(|part| matches!(part, std::path::Component::Normal(_))), + "local model engine install path is invalid" + ); + let bundle = data_dir.canonicalize()?.join(relative); + anyhow::ensure!( + identity.path == bundle.join(binary) + && bundle.canonicalize()? == bundle + && identity.path.canonicalize()? == identity.path, + "local model engine install path is unavailable" + ); local_model_engine_receipt::verify(&bundle, version, &platform, archive, binary)?; anyhow::ensure!( local_model_engine_receipt_identity(data_dir, manifest)? == identity @@ -4229,6 +4322,237 @@ mod tests { ); } + #[cfg(unix)] + fn restore_engine_tree_for_cleanup(path: &Path) { + use std::os::unix::fs::PermissionsExt; + let _ = fs::set_permissions(path, fs::Permissions::from_mode(0o700)); + if let Ok(entries) = fs::read_dir(path) { + for entry in entries.flatten() { + restore_engine_tree_for_cleanup(&entry.path()); + } + } + } + + #[cfg(unix)] + fn unix_engine_identity_fixture(root: &Path) -> (ComponentsManifest, PathBuf, PathBuf) { + unix_engine_identity_fixture_with_binary(root, "llama-server") + } + + #[cfg(unix)] + fn unix_engine_identity_fixture_with_binary( + root: &Path, + binary_path: &str, + ) -> (ComponentsManifest, PathBuf, PathBuf) { + use std::os::unix::fs::PermissionsExt; + let data = root.join("data"); + let platform = detect_platform(); + let relative = format!("libexec/llama.cpp/fixture-v1/{platform}"); + let bundle = data.join(&relative); + let executable = bundle.join(binary_path); + fs::create_dir_all(executable.parent().unwrap()).unwrap(); + fs::write(&executable, b"fixture llama-server\n").unwrap(); + fs::set_permissions(&executable, fs::Permissions::from_mode(0o755)).unwrap(); + let archive = format!("sha256:{}", "a".repeat(64)); + let manifest = serde_json::from_value(serde_json::json!({ + "external": { + "llama-server": { + "version": "fixture-v1", + "platforms": { platform: { + "url": "https://fixture.invalid/llama.tar.gz", + "checksum": archive, + "extract_path": "llama-fixture-v1", + "install_path": relative, + "binary_path": binary_path + }} + } + }, + "profiles": {} + })) + .unwrap(); + (manifest, data, bundle) + } + + #[cfg(unix)] + fn mark_engine_parents_group_writable(data: &Path, bundle: &Path) { + use std::os::unix::fs::PermissionsExt; + let mut path = bundle.parent().unwrap().to_path_buf(); + while path.starts_with(data) && path != data { + fs::set_permissions(&path, fs::Permissions::from_mode(0o775)).unwrap(); + path = path.parent().unwrap().to_path_buf(); + } + fs::set_permissions(data.join("libexec"), fs::Permissions::from_mode(0o775)).unwrap(); + } + + #[cfg(unix)] + #[test] + fn local_model_engine_identity_rejects_group_writable_install_parents() { + use std::os::unix::fs::PermissionsExt; + let tmp = tempfile::tempdir().unwrap(); + let (manifest, data, bundle) = unix_engine_identity_fixture(tmp.path()); + let component = &manifest.external["llama-server"]; + let platform = detect_platform(); + write_cache_metadata( + &manifest, + resolve_platform_info(component, &platform), + &platform, + "llama-server", + &bundle, + ) + .unwrap(); + restore_engine_tree_for_cleanup(&data); + mark_engine_parents_group_writable(&data, &bundle); + fs::set_permissions(&bundle, fs::Permissions::from_mode(0o500)).unwrap(); + fs::set_permissions( + bundle.join("llama-server"), + fs::Permissions::from_mode(0o500), + ) + .unwrap(); + fs::set_permissions( + bundle.join(".elastos-engine.json"), + fs::Permissions::from_mode(0o400), + ) + .unwrap(); + let error = local_model_engine_receipt_identity(&data, &manifest) + .err() + .expect("identity must fail") + .to_string(); + restore_engine_tree_for_cleanup(&data); + assert!( + error.contains("model engine parent is not protected"), + "{error}" + ); + } + + #[cfg(unix)] + #[test] + fn write_cache_metadata_clears_group_writable_parents() { + use std::os::unix::fs::PermissionsExt; + let tmp = tempfile::tempdir().unwrap(); + let (manifest, data, bundle) = unix_engine_identity_fixture(tmp.path()); + mark_engine_parents_group_writable(&data, &bundle); + let component = &manifest.external["llama-server"]; + let platform = detect_platform(); + write_cache_metadata( + &manifest, + resolve_platform_info(component, &platform), + &platform, + "llama-server", + &bundle, + ) + .unwrap(); + for path in [ + data.join("libexec"), + data.join("libexec/llama.cpp"), + data.join("libexec/llama.cpp/fixture-v1"), + ] { + assert_eq!( + fs::metadata(&path).unwrap().permissions().mode() & 0o022, + 0, + "{}", + path.display() + ); + } + restore_engine_tree_for_cleanup(&data); + } + + #[cfg(unix)] + #[test] + fn verified_local_model_engine_rejects_group_writable_parents_without_chmod() { + use std::os::unix::fs::PermissionsExt; + let tmp = tempfile::tempdir().unwrap(); + let (manifest, data, bundle) = unix_engine_identity_fixture(tmp.path()); + let component = &manifest.external["llama-server"]; + let platform = detect_platform(); + write_cache_metadata( + &manifest, + resolve_platform_info(component, &platform), + &platform, + "llama-server", + &bundle, + ) + .unwrap(); + restore_engine_tree_for_cleanup(&data); + mark_engine_parents_group_writable(&data, &bundle); + fs::set_permissions(&bundle, fs::Permissions::from_mode(0o500)).unwrap(); + fs::set_permissions( + bundle.join("llama-server"), + fs::Permissions::from_mode(0o500), + ) + .unwrap(); + fs::set_permissions( + bundle.join(".elastos-engine.json"), + fs::Permissions::from_mode(0o400), + ) + .unwrap(); + let libexec = data.join("libexec"); + let before = fs::metadata(&libexec).unwrap().permissions().mode() & 0o777; + let error = verified_local_model_engine(&data, &manifest) + .err() + .expect("verification must stay read-only") + .to_string(); + let after = fs::metadata(&libexec).unwrap().permissions().mode() & 0o777; + restore_engine_tree_for_cleanup(&data); + assert!( + error.contains("model engine parent is not protected"), + "{error}" + ); + assert_eq!(before, 0o775); + assert_eq!(after, 0o775); + } + + #[cfg(unix)] + #[test] + fn verified_local_model_engine_accepts_a_nested_binary_path() { + let tmp = tempfile::tempdir().unwrap(); + let (manifest, data, bundle) = + unix_engine_identity_fixture_with_binary(tmp.path(), "bin/llama-server"); + let component = &manifest.external["llama-server"]; + let platform = detect_platform(); + write_cache_metadata( + &manifest, + resolve_platform_info(component, &platform), + &platform, + "llama-server", + &bundle, + ) + .unwrap(); + let identity = verified_local_model_engine(&data, &manifest) + .expect("configured install_path remains the receipt bundle root"); + let expected = bundle.canonicalize().unwrap().join("bin/llama-server"); + restore_engine_tree_for_cleanup(&data); + assert_eq!(identity.path, expected); + } + + #[cfg(unix)] + #[test] + fn protect_engine_parents_rejects_symlink_escape_without_mutating_outside() { + use std::os::unix::fs::{symlink, PermissionsExt}; + let tmp = tempfile::tempdir().unwrap(); + let data = tmp.path().join("data"); + let victim = tmp.path().join("victim"); + fs::create_dir_all(&data).unwrap(); + fs::create_dir_all(&victim).unwrap(); + let sentinel = victim.join("sentinel"); + fs::write(&sentinel, b"outside-bytes").unwrap(); + fs::set_permissions(&victim, fs::Permissions::from_mode(0o775)).unwrap(); + symlink(&victim, data.join("libexec")).unwrap(); + let platform = detect_platform(); + let install_path = format!("libexec/llama.cpp/fixture-v1/{platform}"); + let dest = data.join(&install_path); + let error = protect_local_model_engine_install_parents(&dest, &install_path) + .err() + .expect("protect must reject an aliased install parent") + .to_string(); + let victim_mode = fs::metadata(&victim).unwrap().permissions().mode() & 0o777; + let sentinel_bytes = fs::read(&sentinel).unwrap(); + assert!( + error.contains("model engine parent is not protected"), + "{error}" + ); + assert_eq!(victim_mode, 0o775); + assert_eq!(sentinel_bytes, b"outside-bytes"); + } + #[cfg(target_os = "macos")] fn llama_bundle_fixture(root: &Path) -> (ComponentsManifest, PathBuf, PathBuf) { use std::os::unix::fs::{symlink, PermissionsExt}; diff --git a/scripts/fetch/fetch-model.sh b/scripts/fetch/fetch-model.sh index 3bbc82c0b..133c1ea3c 100755 --- a/scripts/fetch/fetch-model.sh +++ b/scripts/fetch/fetch-model.sh @@ -349,6 +349,52 @@ except (OSError, ValueError) as error: PY } +protect_engine_install_parents() { + python3 - "$1" "$2" <<'PY' +import os +import stat +import sys +from pathlib import Path + +root = Path(sys.argv[1]) +if not root.is_absolute(): + root = Path.cwd() / root +relative = Path(sys.argv[2]) +if relative.is_absolute() or any(part in (os.pardir, os.curdir) for part in relative.parts): + raise SystemExit("llama-server bundle install path is invalid") +if not root.is_dir(): + raise SystemExit(f"model engine parent is not protected: {root}") +root = root.resolve() +uid = os.getuid() +flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW +if hasattr(os, "O_CLOEXEC"): + flags |= os.O_CLOEXEC +path = root +for part in relative.parts: + path = path / part + try: + path.relative_to(root) + except ValueError as exc: + raise SystemExit(f"model engine parent is not protected: {path}") from exc + try: + fd = os.open(path, flags) + except OSError as exc: + raise SystemExit(f"model engine parent is not protected: {path}") from exc + try: + metadata = os.fstat(fd) + if metadata.st_uid != uid: + raise SystemExit(f"model engine parent is not protected: {path}") + mode = stat.S_IMODE(metadata.st_mode) + if mode & 0o022: + os.fchmod(fd, mode & ~0o022) + after = os.fstat(fd) + if after.st_uid != uid or stat.S_IMODE(after.st_mode) & 0o022: + raise SystemExit(f"model engine parent is not protected: {path}") + finally: + os.close(fd) +PY +} + install_prebuilt_engine() { local platform="$1" version url checksum extract_path install_path binary_path local archive engine_dir engine_binary receipt stage staged_binary install_temp link_temp @@ -436,6 +482,9 @@ PY fi ok "llama-server ${version} installed and verified" fi + if [[ -n "$binary_path" ]]; then + protect_engine_install_parents "$INSTALL_DIR" "$install_path" + fi if [[ -n "$binary_path" ]]; then if [[ (-e "${BIN_DIR}/llama-server" || -L "${BIN_DIR}/llama-server") \ From 5894b2f9294dd82280249aa3ae4593a7f5cf6f42 Mon Sep 17 00:00:00 2001 From: SashaMIT Date: Sun, 13 Sep 2026 13:44:48 +0800 Subject: [PATCH 351/535] assistant: mode controls move to the sidebar and show only when backed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Chat / Build / Studio bar above the transcript (plus Copy conversation) is removed. The same data-assistant-mode buttons now live in the Sash sidebar: Chat|Build as the URUX segmented control, Studio as a nav row between Home and Settings. Anders' typed controller, workspace fields and string gates are unchanged. UI ≠ authority: a mode control is visible only when the Runtime backs it. Build is a saved label today (no run reads it), so the segment stays hidden behind MODE_CAPABILITY. Studio appears once offers_list advertises an image.generate or video.generate offer (eligibleStudioOffers, exported from assistant.js). A session saved in a mode the Home no longer backs opens as chat unless it still owns an active Studio run. Because the segment is hidden, the Studio row toggles back to chat. Copy conversation is dropped; per-message and code-block copy in the transcript remain. Gates: convergence smoke asserts honest absence on the text-only fixture; shell smoke bans the bar, the copy button, and requires capability gating. Co-authored-by: Cursor (cherry picked from commit d6fb946334c600c81fd250f6fcfd2d0117578832) --- capsules/assistant/browser/agent-harness.css | 40 +++++++++++ .../assistant/browser/assistant-modes.css | 5 +- capsules/assistant/browser/assistant-modes.js | 69 +++++++++++++------ capsules/assistant/browser/assistant.js | 2 +- capsules/assistant/browser/index.html | 16 +++-- .../assistant-convergence-browser-smoke.mjs | 11 +-- scripts/home-agent-shell-smoke.mjs | 3 + 7 files changed, 109 insertions(+), 37 deletions(-) diff --git a/capsules/assistant/browser/agent-harness.css b/capsules/assistant/browser/agent-harness.css index 20e6be40a..50d5f0179 100644 --- a/capsules/assistant/browser/agent-harness.css +++ b/capsules/assistant/browser/agent-harness.css @@ -693,6 +693,42 @@ body.agent-harness-sidebar-collapsed #agent-harness-search-open-main { min-height: 32px; } +/* Chat | Build mode segment (URUX sidebar control). */ +.agent-harness-segment { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 2px; + flex: 0 0 auto; + margin: 0 0 6px; + padding: 3px; + border-radius: 10px; + background: color-mix(in srgb, var(--text) 7%, transparent); +} + +.agent-harness-segment[hidden] { + display: none; +} + +.agent-harness-segment-btn { + appearance: none; + min-height: 28px; + margin: 0; + padding: 0 8px; + border: 0; + border-radius: 8px; + background: transparent; + color: color-mix(in srgb, var(--text) 62%, transparent); + font: inherit; + font-size: 12px; + font-weight: 600; + cursor: pointer; +} + +.agent-harness-segment-btn.is-active { + background: color-mix(in srgb, var(--text) 12%, transparent); + color: var(--text); +} + .agent-harness-nav-row { appearance: none; display: flex; @@ -719,6 +755,10 @@ body.agent-harness-sidebar-collapsed #agent-harness-search-open-main { cursor: pointer; } +.agent-harness-nav-row[hidden] { + display: none; +} + .agent-harness-nav-row:hover { background: color-mix(in srgb, var(--text) 8%, transparent); color: var(--harness-sidebar-fg-active); diff --git a/capsules/assistant/browser/assistant-modes.css b/capsules/assistant/browser/assistant-modes.css index 5a973bf0a..8bb0d5b7f 100644 --- a/capsules/assistant/browser/assistant-modes.css +++ b/capsules/assistant/browser/assistant-modes.css @@ -1,7 +1,4 @@ -.assistant-modes { display:flex; align-items:center; gap:6px; padding:12px 20px; flex-wrap:wrap; } -.assistant-modes button, #assistant-studio button, #assistant-studio select { color:var(--text); background:var(--panel); border:1px solid var(--hairline); border-radius:9px; padding:8px 14px; font:inherit; } -.assistant-modes button[aria-pressed="true"] { background:var(--brand); color:#151515; } -#assistant-copy-conversation { margin-left:auto; } +#assistant-studio button, #assistant-studio select { color:var(--text); background:var(--panel); border:1px solid var(--hairline); border-radius:9px; padding:8px 14px; font:inherit; } #assistant-workspace-notice { margin:4px 20px; padding:10px; border:1px solid var(--hairline); border-radius:8px; cursor:pointer; } #assistant-studio { padding:20px; overflow:auto; max-height:calc(100vh - 150px); } #assistant-studio label { display:block; margin:16px 0 8px; } diff --git a/capsules/assistant/browser/assistant-modes.js b/capsules/assistant/browser/assistant-modes.js index 8c92ec2e5..2277c6ff6 100644 --- a/capsules/assistant/browser/assistant-modes.js +++ b/capsules/assistant/browser/assistant-modes.js @@ -1,13 +1,20 @@ -import { createAssistantApp } from "./assistant.js"; +import { createAssistantApp, eligibleStudioOffers } from "./assistant.js"; import { getAgentWorkspaceSnapshot, setAssistantMode } from "./agent-harness.js"; import { setAssistantWorkspaceField, getAssistantSession, ensureAssistantStudioSession, setAssistantSessionStudio, captureActiveSessionState, applyAssistantModeDraft } from "./agent-workspace.js"; import { getHomeGuiLaunchToken, persistAgentWorkspaceNow, flushAgentWorkspace, bindWorkspaceMergeGuard } from "./harness-host.js"; import { getLiveTurnCanonical } from "./agent-stream.js"; -import { createHomeClipboardClient } from "/apps/home/home-clipboard-client.js?v=home-20260726a"; +import { fetchModelOffers } from "./agent-live.js"; + +/* UI ≠ authority: a mode control is shown only when the Runtime backs it. + Build is a saved label today (no run reads it), so its segment stays hidden; + Studio appears once an image or video offer is advertised. */ +const MODE_CAPABILITY = { chat: true, build: false }; export async function bindAssistantModes(saved = {}) { const panel = document.querySelector("#assistant-studio"); const notice = document.querySelector("#assistant-workspace-notice"); + const segment = document.querySelector("[data-assistant-mode-segment]"); + const studioRow = document.querySelector('button[data-assistant-mode="studio"]'); const offer = document.querySelector("#studio-offer"); const draft = document.querySelector("#studio-draft"); const status = document.querySelector("#studio-status"); @@ -23,12 +30,23 @@ export async function bindAssistantModes(saved = {}) { status.after(check); const controllers = new Map(); let visibleSessionId = saved.activeSessionId || null; - const clipboard = createHomeClipboardClient({ - targetId: "assistant", homeToken: getHomeGuiLaunchToken(), - homeOrigin: new URL(location.href).searchParams.get("home_origin") || "null", - targetWindow: window.top, sourceWindow: window, - }); - clipboard.start(); + let studioAvailable = false; + + function modeAvailable(mode) { + return mode === "studio" ? studioAvailable : Boolean(MODE_CAPABILITY[mode]); + } + function syncModeControls() { + segment.hidden = !MODE_CAPABILITY.build; + studioRow.hidden = !studioAvailable; + } + async function refreshStudioAvailability() { + try { + studioAvailable = eligibleStudioOffers(await fetchModelOffers()).length > 0; + } catch { + studioAvailable = false; + } + syncModeControls(); + } function render(view) { if (draft.value !== view.studioDraft) draft.value = view.studioDraft; @@ -134,15 +152,30 @@ export async function bindAssistantModes(saved = {}) { applyAssistantModeDraft(mode); document.body.dataset.assistantMode = mode; panel.hidden = mode !== "studio"; - for (const item of document.querySelectorAll("button[data-assistant-mode]")) item.setAttribute("aria-pressed", String(item.dataset.assistantMode === mode)); + for (const item of document.querySelectorAll("button[data-assistant-mode]")) { + const pressed = item.dataset.assistantMode === mode; + item.setAttribute("aria-pressed", String(pressed)); + item.classList.toggle("is-active", pressed); + } setAssistantMode(mode); setAssistantWorkspaceField("activeMode", mode); } + // A session saved in a mode this Home no longer backs opens as chat, never as an unreachable room. + function showAvailableMode(mode) { + const wanted = ["chat", "build", "studio"].includes(mode) ? mode : "chat"; + const hasStudioRun = wanted === "studio" && getAssistantSession()?.studio?.activeRun && !getAssistantSession().studio.activeRun.terminal; + showMode(modeAvailable(wanted) || hasStudioRun ? wanted : "chat"); + } window.addEventListener("assistant:session-selected", event => { visibleSessionId = event.detail?.sessionId ?? getAssistantSession()?.id ?? null; - showMode(event.detail?.mode); + showAvailableMode(event.detail?.mode); + }); + const savedMode = saved.activeMode || saved.sessionMode || "chat"; + showAvailableMode(savedMode); + // Offers arrive after boot; a saved Studio session reopens once its offer is confirmed. + void refreshStudioAvailability().then(() => { + if (savedMode === "studio" && studioAvailable && document.body.dataset.assistantMode === "chat") showMode("studio"); }); - showMode(saved.activeMode || saved.sessionMode || "chat"); for (const button of document.querySelectorAll("button[data-assistant-mode]")) { button.addEventListener("click", () => { const turn = getLiveTurnCanonical(); @@ -150,21 +183,13 @@ export async function bindAssistantModes(saved = {}) { notice.hidden = false; notice.textContent = "Finish or stop the current run before changing mode."; return; } captureActiveSessionState(); - showMode(button.dataset.assistantMode); + // The Studio row is the only visible mode control while the Chat|Build segment is hidden, so it toggles. + const leavingStudio = button.dataset.assistantMode === "studio" && document.body.dataset.assistantMode === "studio"; + showMode(leavingStudio ? "chat" : button.dataset.assistantMode); }); } window.addEventListener("assistant:workspace-save", event => { notice.hidden = !event.detail.error; notice.textContent = event.detail.error || ""; }); notice.addEventListener("click", () => persistAgentWorkspaceNow()); - document.querySelector("#assistant-copy-conversation").addEventListener("click", async () => { - const session = getAssistantSession(); - const conversation = (session?.messages || []).map(message => `## ${message.role}\n\n${message.text ?? message.content ?? ""}`); - for (const run of session?.studio?.studioHistory || []) conversation.push(`## Studio\n\n${run.prompt || ""}\n\n${run.output?.resource_id || run.status || "Outcome unknown"}`); - try { - if (!conversation.length) return; - await clipboard.writeText(conversation.join("\n\n"), {purpose: "transcript.markdown"}); - notice.hidden = false; notice.textContent = "Conversation copied."; - } catch { notice.hidden = false; notice.textContent = "Could not copy the conversation."; } - }); } diff --git a/capsules/assistant/browser/assistant.js b/capsules/assistant/browser/assistant.js index 42beaaaa0..59da6a82e 100644 --- a/capsules/assistant/browser/assistant.js +++ b/capsules/assistant/browser/assistant.js @@ -149,7 +149,7 @@ function eligibleTextOffers(payload) { ); } -function eligibleStudioOffers(payload) { +export function eligibleStudioOffers(payload) { const offers = Array.isArray(payload?.offers) ? payload.offers : Array.isArray(payload?.data?.offers) diff --git a/capsules/assistant/browser/index.html b/capsules/assistant/browser/index.html index 7c70d5fe9..ad6c02fb6 100644 --- a/capsules/assistant/browser/index.html +++ b/capsules/assistant/browser/index.html @@ -64,6 +64,11 @@
+ +
-