[andersalm]

Summary

Design document for dedicated hardware fleet that runs installed journeys on every develop build. This moves journey testing from GitHub CI (slow, flaky) to post-merge acceptance on physical devices with external judgment.

Context

• Issue: CI and tests: warm caches, fail fast, keep only what catches real bugs #194 (CI and tests efficiency)
• Goal: Mac, Jetson, Linux x86/ARM, two networks, phone test every develop merge
• Current State: 35-45 min journeys in CI per PR, flaky in virtual environments
• Proposed: Dedicated fleet tests post-merge, GitHub Actions freed up

Design Coverage

The design specifies:

• Fleet topology: 5 device types + controller + network holder
• Journey distribution: Which journeys run on which devices (grounded in existing scripts)
• Build acquisition: 3 options evaluated (recommends CI-built artifacts on develop merge)
• Results collection: External judgment, aggregation, GitHub reporting (Check Run API)
• Failure handling: Device offline, journey failures, retries, alerts
• Cost & hardware: Existing + minimal new investment, net savings from GitHub Actions reduction
• Rollout order: 6 phases from single-device proof to full production

Grounding in Existing Scripts

All journey assignments reference real scripts:

• scripts/ci-installed-journeys.sh (prepare, home, home-repeat, model-repair, summary)
• scripts/ci-installed-journeys.py (Python journey runner, receipts, validation)
• scripts/home-two-runtime-acceptance-core.mjs (bilateral Home tests)
• scripts/public-user-journey-smoke.sh (public-facing smoke tests)
• scripts/browser-mac-vm-acceptance-handoff.sh (Mac Browser VM acceptance)

Weakest Points (Self-Identified)

1. Single fleet controller is SPOF — crash blocks all acceptance until manual restart
2. Mac VZ exclusivity serializes Mac tests — 45-60 min bottleneck, only one Browser VM at a time
3. Phone role undefined — design punts to Phase 6 without clarity on mobile app vs. Carrier peer

Open Questions

1. Build strategy: CI-built artifacts (recommended) vs. fleet-local builds?
2. Phone role: Carrier peer test device or mobile app acceptance target?
3. GitHub integration: Commit Status API (simple) or Check Runs API (detailed)?
4. Browser VM persistence: Fresh VM each run, or reusable profile with reset?

Next Steps

• Claude Code review (via OMP): Validate journey distribution and build strategy
• Devil's advocate pass: Challenge SPOF risks, phone scope, network isolation testing
• OMP decision on open questions
• Implementation (separate PRs, if design approved)

Files Changed

• docs/acceptance-fleet-design.md (new): Complete design specification

Compliance

• ✅ Design only, no code (as required)
• ✅ Files only under docs/ (as required)
• ✅ Grounded in existing scripts (cited with paths)
• ✅ Includes "weakest points" section (as required)
• ✅ DRAFT PR (will never merge, as instructed)
• ✅ Branched from develop
• ✅ No --no-verify used

---

Review Assignments:

• Primary: Claude Code (via OMP)
• Secondary: Devil's advocate pass

Related Issues: #194, #180 (Home-as-web-page), #293 (Browser hardware proof)

  

[andersalm]

Review of head cd413d9 (docs/acceptance-fleet-design.md only) against develop ac7d018, #194, #93 (15 Oct comments), ci.yml, release-package.yml and scripts/. Verdict: needs changes.

1. The journeys that found bugs late are not in the device matrix. "Journey distribution" never names System update, Undo (ci.yml:1225-1239, update-hop-compare.py), the 0.7.1 stable re-trust through install.sh (0.8.0 publication: sign one release for both stable and canary #335, 0.8.0: Release 0.8.0 #93 comment 6184889837) or the slow-machine update that left the Jetson Home down (0.8.0: Release 0.8.0 #93 comment 6184924301). Community is covered only by home-two-runtime-acceptance.mjs, a contacts and direct-message fixture; what expired on alpha.17 was posts over gossip through the seed (0.8.0: Release 0.8.0 #93 comment 6191449436). Add install.sh from a test seed, N to N+1 update, Undo, the re-trust hop, and two Homes behind different NATs with a forward-only Community node (the 15 Oct sweep pattern, 0.8.0: Release 0.8.0 #93 comment 6193306539).
2. "External judgment" ("Results Collection and Judgment") parses the scripts' own JSON receipts, which is the self-report CI already has. The late findings were observed by a person: Browser open 7.8-9.6 s, blank images, window not draggable, 240 attaches/s (0.8.0: Release 0.8.0 #93 comment 6189242538). The design needs an observer outside the Runtime (install.sh exit, HTTP timings, screenshots judged by a second process) and the human checks that AGENTS.md "Browser Claim Discipline" and Required: Home on phones #92 require.
3. The phone role is already decided: Required: Home on phones #92 is the browser Home shell on a phone, checked by Sasha on the exact candidate with Home on a phone: Home screen pages, phone title bars and stage, sheets, 44 px targets (phone-only) #116's home-phone-layout-smoke.mjs. It is not a Carrier peer or a mobile app. "Phase 6" and open question 2 can go.
4. The build path contradicts itself. "Option A: CI-built artifacts" is recommended, yet every assigned script needs a source Home (ci-installed-journeys.sh:6 requires CI_SOURCE_HOME and prepare asserts a source engine receipt), and "Installation" copies binaries into $FLEET_DATA_ROOT/<sha>. Users get the signed N/N+1 package from release-package.yml through install.sh; the fleet should install that way.
5. The result has no consumer. "Block develop if critical journey fails" cannot act after the merge. The gate that stops a late bug being signed is scripts/release-publish.sh checked_ci, which admits develop push or merge-group CI only; the fleet check must feed that gate.
6. Grounding errors: home-two-runtime-acceptance.mjs "currently runs in CI" is false (only docs/COLLABORATION_HANDOFF.md:173 names it); public-user-journey-smoke.sh has no caller and is a Home-as-web-page script held for Hacked-Home regression suite in CI #180; "35-45 min" is the 1 Oct figure, the Mac job now takes 68-76 min and PR median is 67.6 min (CI and tests: warm caches, fail fast, keep only what catches real bugs #194 comments 6062744592, 6068343836); "flaky due to GitHub Actions" is unsupported, the recorded flakes are test bugs (CI and tests: warm caches, fail fast, keep only what catches real bugs #194 comments 6046050638, 6086329043). Hardware does not fix those.
7. The cost section is wrong: the repository is public and ci.yml uses standard runner labels, which GitHub does not bill for public repositories, so "saves $200-500/month" is zero. The real cost is wall time.
8. Entropy: a controller daemon, device agent, webhook, dashboard, NFS, retention cron and heartbeat are added and nothing is removed; the doc carries Status, Version, Author and week-by-week phases (AGENTS.md keeps plans in issues and docs release-neutral) and commits hostnames and /opt/fleet paths (AGENTS.md "Staging Machines"). A smaller design uses self-hosted runner labels on the existing Mac, Jetson and Linux boxes, one workflow_run job after develop CI, and the check on the commit as the result; fork PRs stay on hosted runners. Automatic retry and "ARM64 VM if Jetson offline" hide the timing bugs this fleet exists to catch; fail instead.

Timing: #194 (13 Oct) postponed CI changes to 0.8.1, so this lands after 0.8.0 is signed.

[andersalm]

Superseded by #389 (the rewritten fleet plan); its text goes to #194 after review, after 0.8.0 is published.