Installed journeys in source-home-linux (ubuntu-24.04) fail intermittently, on develop too

What fails

The installed-journey steps in the source-home-linux (ubuntu-24.04) job fail on some runs and pass on others. The same product code gives both results. Four failures so far:

Run / job	Head	Step	Error
39022710033 / 117404156258	develop `9be009b` (push)	`ci-installed-journeys.sh home`	`RuntimeError: finite installed UI journey failed`
38899842659 / 116995610376	#394 (older head)	same	same; the 4 GiB resource-lifecycle journey: `actions` holds only `engine_running`, peak memory 694 MB of 4 GiB, no OOM events
39019721681 / 117389514977	#415 `4728164`	`ci-installed-journeys.sh model-repair`	`stage=model_interrupted_get_retry code=timeout check=restarted_home` (the Runtime restarted, 2 episodes, but one of the 60 s waits after `subcheck = "restarted_home"` timed out)
38932131083	#67 (older head)	installed journey	`installed Runtime exited before health` about 16 s after launch, peak memory 148 MB of 1 GiB, no OOM

The #415 head's elastos/ and capsules/ trees are byte-identical to #413 at 69e2bae. That head passed the same job (run 39019718818). So these failures come from the journeys or the runner, not from the PRs where they showed up.

Why the cause is unknown

On develop, the finite observer keeps only the code finite_resource_observation_failed and drops the exception text (scripts/ci-installed-journeys.py). The private runtime log is not uploaded. A code-backed guess for the finite case: on engine_stopped, the observer raises "engine group exit is unconfirmed" the first time it sees the engine gone while a member of its process group is still alive, and it does not retry (develop ci-installed-journeys.py around 477-479). Stop kills the guard and llama-server with one SIGKILL to the group (capsules/model-provider/src/local_llama.rs:904). This guess is not proven.

Next step

Draft #413 records the observer's phase, exception type, redacted message and runtime_exited in the receipt. Once it lands, the next failure will name its check. Until then, a red run of these steps on a PR is not evidence against that PR.

Acceptance record

Candidate: see 0.8.0: Release 0.8.0 #93. Train fixes: ff2cb64c9, 34b2d44d8, 088fa8bb8.
Completed source evidence: the restore-ready correction and source review are recorded. Exact historical f674ff63738d0e2c8724ede16bb785bef58511cb passed its configured Linux source gate, exit 0 in 1,743 seconds with Full=False; installed, native-provider, Browser, IPFS and real network journeys remain separate.
Remaining acceptance: OMP owns combined-candidate qualification and signed installed Retry/resume proof. The hidden-Marketplace CI failure remained unreproduced in 110 local trials, so its cause stays an inference. Preserve each failed source/run/attempt and the separate startup-observer case.
Remaining gate: exact-candidate CI qualification on alpha.25, including x86 model-repair, and the signed installed Retry/resume proof above; each failed run and attempt is recorded here with its disposition. The ~17-run estimate is a conditional statistical heuristic and its waiver a proposal; neither is an accepted gate. Earlier Keep capability admission bounded and grant retirement consistent #412/Retire superseded Browser observations and verify lifecycle recovery #420/Keep deferred Browser input with its owning page #421 runs and reviews are on those PRs.

Independent check: develop 9be run 39022710033 failed the finite installed UI journey. Artifact 12103805982 matches its GitHub SHA-256; its 4 GiB receipt records engine_running → engine_stopped → engine_running before failure.
Peak memory was 728,760,320 bytes with zero OOM events; final cleanup records an empty cgroup, reaped Runtime and closed Browser. The receipt omits the observer exception, so the failed check remains unresolved.
The reviewed two-file observer correction is now integrated at cd620ab (tree081a6d009a56ad2f22e98af014bd203f10c9847e; sole parent6ee5fa7f). Independent exact-object review confirms both files match patch SHA-256 c13058ad85c4b600a1770703bef3e57317ba534134e0b316d186be53f765cf3e byte for byte. Existing synthetic proof preserves the20/95s deadlines,50s idle minimum and full retained-group checks; the old CI cause and exact-candidate installed acceptance remain open.

OMP agrees with the source review (comment 6235138982). The cause is still a hypothesis: the retained receipt holds no observer exception, so it cannot show which check failed.

What the run proves: the receipt reached idle after busy, explicit Stop and stop-recovery all passed. The product's own closure waits for the guard and then checks that the group is gone (local_llama.rs:876-932).

What can fail at the idle_stopped stage:

The observer's single check at ci-installed-journeys.py:477-479. It runs while the guard or adapter is still alive after llama-server is killed. Normal cleanup can reach this state (hypothesis above).
"Idle engine closed early": the provider releases 60 s after the last run, and the observer requires at least 50 s after it takes the request. That leaves a 10 s margin.
The 95 s stage deadline.

Both finite failures so far (38899842659 at engine_stopped, 39022710033 at idle) happened in a stage where the engine group is being torn down.

Rule for qualifying the 0.8.0-alpha.23 candidate 6ee5fa7f2, without changing the candidate now:

If develop CI for 6ee5fa7f2 passes every finite stage: qualification proceeds and Installed journeys in source-home-linux (ubuntu-24.04) fail intermittently, on develop too #433 stays open. Confirmation comes after 0.8.0 from a narrow diagnostics change that records the observer's phase, exception type and message, split from ci: installed journeys name the observer failure, and the SmolLM2 proof uses a vendored license #413 so it does not bring the held ZIP9 migration. The fix that follows is bounded observation up to the existing deadline, with every exit check kept.
If it fails a finite stage again: a rerun does not close it. The narrow diagnostics and bounded-observation change goes into the candidate as one reviewed, scripts-only commit, followed by one more qualification cycle.

Owners: Codex prepares the inert diagnostics and observation patch and reviews it. OMP integrates it only under the second branch above, or after 0.8.0. The failed run's evidence is kept: artifact source-home-journeys-ubuntu-24.04 of run 39022710033, copied at /tmp/j433-art.

Amending comment 6235170786: a green develop CI on 6ee5fa7f2 lets alpha.23 preparation continue and adds current journey evidence. It does not classify the earlier failure as a harness defect, and it does not defer #433 until after 0.8.0. #433's release disposition stays open until Codex's inert diagnostics and observation proposal (original fails, patched passes; persistent-group timeout and unrelated-error checks) and the installed results on the exact candidate are reviewed. Every cleanup assertion is kept. No whole #413 merge.

Cross-family review of the inert observer proposal: PASS. Claude reviewer; patch sha256 c13058ad…3cf3e, applied on 6ee5fa7f2 in a /tmp copy.

Exit and cleanup checks: every one remains.

These are unchanged and immediate: census, escape, a live child leaving the census, the Runtime leaving its unit, OOM, a second engine, stage binding, interruption, the crash-stage engine check, the 20 s and 95 s deadlines, and the 50 s idle minimum.
finish() and finite_receipt_valid are unchanged apart from two inserted lines.
The only change in behaviour: when the selected engine has left and its group has not fully exited, the observer waits up to the existing deadline instead of failing on the first sample. It still fails with the same message at the deadline.

Synthetic timing cases: the reviewer ran the real action() from both versions with a fake clock.

Case	Before	After
Group member never exits	fails	fails at 20.1 s
Engine never exits (idle)	fails	fails at 95.1 s
Early idle close (30 s)	fails	fails
Second engine	fails	fails
Production cleanup order, group gone within the deadline	fails	passes

Non-blocking: a guard that exits before its engine, where the engine then exits between 50 and 95 s, now passes. No process is alive when that stage passes.

Receipt privacy: the receipt adds only observer_failure: a phase from a fixed set, one of 14 built-in exception type names, and one of 42 static messages. Anything else becomes redacted. No exception text, repr or traceback is formatted, and the reviewer's 13-case redaction probe kept only fixed values.

Tests:

The proof script reproduces: the original code fails the delayed-group case, 9 of 9 focused tests pass, and all three removal controls fail as intended.
ci-release-policy-test.py: 102 tests OK before the patch, 109 OK after.
Python 3.12 journey classes: 43 OK.
None of this ran on Linux pidfds or cgroups, and it does not prove the cause of the old failure.

Integration: after the running 6ee5fa7f2 push gate finishes, the patch goes in as one commit on release/train-0.8.0-rc8. That head gets one gated push, a musl precheck and one qualification run. a23-qualify.sh was stopped before it moved develop or dispatched anything for 6ee5fa7f2. #433 stays open until the installed results on the exact candidate are reviewed.

test(journeys): finite observer waits for the engine group to exit wi…

Fifth failure: #417 at 5873d2f, run 39040338532, job 117453636062, step ci-installed-journeys.sh model-repair. It failed with stage=model_interrupted_get_retry code=timeout check=restarted_home, the same as #415 (run 39019721681).

So the model-repair restarted_home timeout has now happened on two unrelated PRs: #415 changes only a Clipboard test, #417 only zip and the Library extract guard. After the Runtime restart, one of the 60 s waits under subcheck = "restarted_home" in scripts/ci-installed-home-journey.mjs times out: Home ready, signed authority, #desktop, or the "The download stopped before it finished. Retry." text. The receipt does not record which one, so a first step would be to record the wait that timed out.

Generated by Claude Code

Read-only corroboration of Sasha's new report: GitHub records job117453636062 failed at source5873d2f7c89c258a229a8ffe338712d173338e6d, in installed model Stop/switch/interrupted Get recovery. The reported receipt names model_interrupted_get_retry/restarted_home; this check inspected job metadata and source, rather than the receipt or private Runtime log.
The journey harness blob at5873d2f7 is byte-identical to8ea4aac; four60-second waits at scripts/ci-installed-home-journey.mjs:319-328 share restarted_home. The commit is outside8ea history. The next diagnostic should name the failed wait with a fixed phase label; its cause and behavior on8ea remain open.
The existingcd620 observer correction remains part of8ea and addresses a separate finite-UI observer case. Preserve healthy jobs and the recorded failures; exact-candidate installed model recovery and the full#93 acceptance gate still require proof.

Sixth failure: #397 at 56a8084, run 39049059654, job 117474965169. The failing step is ci-installed-journeys.sh home ("installed Marketplace Get and Assistant reply"), and the error is AssertionError: installed Runtime exited before health (ci-installed-journeys.py:868 wait_healthy). That is the same class as #67's run 38932131083.

In the same run, source-home-linux (ubuntu-24.04-arm) and source-home-macos passed the installed journeys. So far every one of these failures has been on the x86_64 ubuntu-24.04 runner.

Generated by Claude Code

Independent job metadata confirms job 117474965169 failed the Marketplace Get/Assistant step at 56a8084fc82d55f266f71b64a992c9d10fa62888; the overall run concluded cancelled. ARM and Mac jobs succeeded at that SHA, with different journey flags.
The reported startup assertion checks FiniteRuntime.poll(), which reports either an observer error or a Runtime exit. The exception alone leaves that distinction open; the reviewed d495a8058 has later observer changes, while its startup call remains.
Keep separate, redacted observer and child-exit causes for startup readiness, with the existing deadlines and cleanup. This older PR result leaves exact-candidate installed startup and model-recovery acceptance open.

Seventh failure: #435 at a143e14 (an installer-script change only), run 39049411613, job 117479780091. The model-repair step failed with a new variant: stage=model_stop_get_different code=assertion_failed check=stop_real_stream. It again ran on the x86_64 ubuntu-24.04 runner. The model-repair journey has now failed at two different stages (restarted_home and stop_real_stream) on PRs that don't touch the model provider.

Generated by Claude Code

Completed metadata confirms x86 job 117479780091 failed steps 31 and 32 at a143e145a5726141f524a11787dd92f06c7fa204; the run concluded cancelled. CI tests the full inherited source, so attribution to #435’s installer delta remains open.
stop_real_stream covers starting a live model run, text and engine checks, Stop/cancellation, terminal cancelled, whole-engine exit and Runtime continuity, including helper assertions. The reported assertion_failed identifies this phase; the exact assertion and cause remain unverified.
The browser harness and shell wrapper match reviewed d495a8058; its Python observer code differs. Preserve the existing evidence and deadlines. The release candidate still needs its own successful installed model-recovery receipt.

One more: #67 baaf15c, run 39051786892, job 117508216882, model_interrupted_get_retry / restarted_home timeout. That is the third hit, after #415 and #417. In the artifact, home-failure.png shows Home ready with the desktop visible after the runtime restart and Marketplace opening. So the wait that times out is in app("marketplace"), showModelDetail, or the "download stopped… Retry." text (ci-installed-home-journey.mjs ~:326-328), not Home itself.

Generated by Claude Code

One more: #395 12fc78b, run 39049051156, job 117514231178, "finite installed UI journey failed". This is the same pattern as develop 9be009b and #394's old run 38899842659. finite_memory.actions = [engine_running] only, status=failed, with no OOM events (peak 725 MB of 4 GiB), and cgroup_empty, runtime_reaped and whole_process_exit all true. So the observer stopped after engine_running without recording engine_stopped, and its error text is not kept. #413's cause recording would show why.

Generated by Claude Code

Merge origin/develop into fix/ci-journey-failure-causes

4th restarted_home hit: #413 04b621d, run 39088847791, job 117562372696. This branch records observer failures, and it narrows the cause:

model_repair.runtime_episodes shows two clean runtime runs (same exe sha256). The actions are engine_running, engine_stopped, restart_get. Cleanup passed, and there is no control_failure, so the observer and Runtime side were healthy.
home-failure.png is identical to Wallpaper from Library: CORP + shell-token wallpaper path, image thumbnails, Set as Desktop Background #67's: Home is ready, and the dock shows Marketplace running (the dot under its icon), but no Marketplace window is on screen.
So the 60 s wait that runs out is in the browser-side Marketplace step after the restart (ci-installed-home-journey.mjs ~:326-328: app("marketplace") / showModelDetail / the "download stopped… Retry." text). The window is launched but never shown or focused. It looks like a Home window-restore issue after a runtime restart, not the model path.

Generated by Claude Code

Completed train run 39254991789, attempt 1 binds ci.yml workflow_dispatch on release/train-0.8.0-rc9 to source 3beb816, tree 5bc1173ec18ec947c40516f76917891f55301145. Metadata shows 15 successful jobs, one failure and one skipped job.
The Ubuntu 24.04 installed Stop, switch and interrupted Get step failed from 14:04:13 to 14:10:13 UTC. OMP read stage=model_interrupted_get_retry, code=timeout, check=restarted_home; my independent check covers the completed run and step metadata. The cause remains open.
Keep this occurrence with the successful develop push and earlier failed attempts. OMP owns the exact wait, Runtime state and safe diagnostics before the final candidate disposition.

Diagnosis (OMP, 20 Oct): a Home window-restore race, not a model or interrupted-Get bug.

What the journey sees: after the Runtime restart, Runtime is healthy, the partial download is kept, and the Home is ready. The failure screenshot is the same in all four runs examined: the Marketplace dock icon shows its open dot, but no window is on screen.
Cause (about 70% confidence; the hidden window was not reproduced locally):
- After a reload, the Home desktop can report "ready" while saved windows are still reopening. The restore is marked done before it finishes (home-gui-shell.js:137-150), so a second summary marks the desktop ready early.
- The journey then clicks the Marketplace dock icon. A dock click on an active, visible window hides it (shell-windows.js:1718-1728).
- The early "ready" was reproduced with the real desktop code.
What a user sees: a dock click during the restore can hide a just-reopened window; one more click brings it back.
Frequency: 11 failures in 62 runs of this step (17–20 Oct), 10 of them restarted_home (about 16%). The step runs only on ubuntu-24.04, so "x86 only" is a selection effect.
Fix for 0.8.0 (branch fix/433-restore-ready; Codex review next):
- one restore promise that every summary waits for;
- a regression test that fails on the current code;
- restarted_home split into one label per wait.
If the fix misses the freeze: a recorded waiver. Runtime stays healthy and the partial download is kept. On ubuntu-24.04, Retry/resume after a restart goes unproven in about 16% of runs.
Note: one green run proves little at this rate. About 17 green runs are needed for 95% confidence.

Home waits for its saved windows to reopen, while a failed window still permits Home use. Review fix433a passed source 2b8dbbfcc3ed5a6c48a21f96e6adeff1feb44d21 (tree e1555eac8529c9aa5266a0b2a57e0dd65f80c52a, approved base a615769b59efa909efbee081ee2bb897d4a62df7); its receipt SHA-256 is a6287d3cd190fb15b4dd60f2b218594f93a8f607b0705db5208e6fd36bf3948d.
OMP integrated the three normal commits into local train 088fa8bb869bc5738153428e8892f0895c6edf0f (tree 5a5dd44433f80ad9e674b0c460ccba2ac4e096eb). Independent comparison found all three full patches and messages byte-identical. The source candidate's completed gate passed exit 0 in 1,897 seconds, with 4,549 Rust passes and 26 ignored; log SHA-256: ee202c373c84a51cead28f106a66d9abee9c5fadf815e4d796b8fc95d9957ca7.
The combined train still needs its own checks and signed installed Retry/resume proof. The hidden Marketplace CI failure stayed unreproduced in 110 local trials, so that cause remains an inference; the existing acceptance requirements remain in force.

Status (20 Oct): source fixed and reviewed: ff2cb64c9, 34b2d44d8 and 088fa8bb8 are on the alpha.25 train (release/train-0.8.0-rc10, current head f674ff637); Codex review fix433a PASS. Remaining: the CI x86 model-repair runs and the installed check on signed alpha.25. Stays open until then.

New failure signature in the same job, 29 Oct. Develop push CI 40889720032 on 57627f88c failed source-home-linux (ubuntu-24.04) at "installed Marketplace Get and Assistant reply" with finite installed UI journey failed (census: RuntimeError: finite journey live child left its unit census) (scripts/ci-installed-journeys.py lines 589-590). The train CI run 40889512306 on the same SHA passed this job, so the product code gives both results.

Likely cause (inference, not reproduced): the observer lists the unit's processes from cgroup.procs, and Linux drops a process from that list when its last thread starts to exit. Its pidfd becomes readable only later, at exit notification. A child that exits between the two reads then looks like a live process outside the unit. A bounded wait on the pidfd before the check would close this window. The log keeps only the message, without the PID or role.

For 0.8.0-alpha.26, alpha-qualify.sh reruns a failed source-home-linux job once, which covers this case. The harness change waits until after the release.

Diagnosis of the census failures in source-home-linux (ubuntu-24.04): the evidence supports the CI harness, not the product, as the cause. CI 40915572998 on 8a74b669c ("live child identity is unavailable") and push CI 40889720032 on 57627f88c ("live child left its unit census") both record observer_failure with runtime_exited: false in installed-journeys.json; memory peaks 146 MB and 635 MB of 4 GiB, no OOM. The finite-journey observer in scripts/ci-installed-journeys.py checks a child's pidfd once; a multi-threaded child that is exiting (most likely the short-lived kubo init) has already lost /proc/PID/exe and left cgroup.procs before its pidfd becomes readable, so the census raises and wait_healthy (line 1065) reports it as "installed Runtime exited". The same message appears in runs 39102682843 (18 Oct) and 39240606353 (19 Oct), before the Kubo-offline and seed-holder changes; the observer runs only on ubuntu-24.04 (ci.yml:920-921, 934-935). The same commit passed on push (40916013126). Limit: the log does not record the exiting child, so the child named above is inferred and an abnormal capsule exit is not fully excluded; nothing in either receipt points to one. Both failed runs stay on record as evidence. Fix (CI script only, 0.8.1): a bounded pidfd wait (2 s) at the census checks, a correct failure label when runtime_exited is false, and two cases in ci-release-policy-test.py.

Seen again on 3 Nov (develop) and 4 Nov (PR #526, run 41754568343, source-home-linux ubuntu-24.04: "finite journey live child left its unit census"); the arm64 job passed on the same commit. Each occurrence costs a 30 to 60 minute rerun and cancels the Mac and update-hop jobs. Moved to 2.1.0 with the pipeline work in #527.

Merge pull request #533 from elastos/fix/433-census-flake

#533 merged (27ea103): the census now records the missing child's pid, start tick, state, role, executable hash, pidfd state and current cgroup (systemd unit names kept), plus its parent and a re-read of the expected unit, in the uploaded receipt. Same family seen again on PR #531 (5 Nov 02:32Z, stage_idle_stopped: "finite journey live child identity is unavailable") before #533 was on that branch. The next occurrence on a branch with develop merged gives the evidence.

The census diagnostics from #533 caught the next occurrence and name the cause.

PR #547, CI run 42188746234, source-home-linux (ubuntu-24.04), "installed Marketplace Get and Assistant reply": finite journey live child left its unit census. installed-journeys.json finite_memory.census_gap:

retained: pid 60006 llama-server, ppid 60005, state S, cgroup /system.slice/elastos-ci-model-59715-bdefb9fa3e3baeec.service, start_ticks 152912;
observed 109 ms later: same pid, same start_ticks, same cgroup, state Z, empty cmdline;
expected_cgroup_procs (31 pids) no longer lists 60006; parent 60005 still alive and in the unit (pidfd_exited: false).

So the process did not leave the unit: llama-server had exited and was a zombie waiting for its parent to reap it. Linux removes an exited task from cgroup.procs before the parent reaps it, while /proc/<pid> still shows it as Z. The census counts that zombie as a live child outside the unit.

Fix: the census treats state Z as exited (not escaped), and keeps the existing whole-unit exit check that proves it is reaped when the unit ends. Briefed on a separate branch.

Merge pull request #549 from elastos/fix/433-census-zombie

Navigation Menu

Repository navigation

Installed journeys in source-home-linux (ubuntu-24.04) fail intermittently, on develop too #433

What fails

Why the cause is unknown

Next step

Acceptance record

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Installed journeys in source-home-linux (ubuntu-24.04) fail intermittently, on develop too #433

Description

What fails

Why the cause is unknown

Next step

Acceptance record

Activity

andersalm commented on Oct 17, 2026

andersalm commented on Oct 17, 2026

andersalm commented on Oct 17, 2026

andersalm commented on Oct 17, 2026

SashaMIT commented on Oct 17, 2026

andersalm commented on Oct 17, 2026

SashaMIT commented on Oct 17, 2026

andersalm commented on Oct 17, 2026

SashaMIT commented on Oct 17, 2026

andersalm commented on Oct 17, 2026

SashaMIT commented on Oct 17, 2026

SashaMIT commented on Oct 17, 2026

SashaMIT commented on Oct 18, 2026

7 remaining items

andersalm commented on Oct 19, 2026

andersalm commented on Oct 20, 2026

andersalm commented on Oct 20, 2026

andersalm commented on Oct 21, 2026

andersalm commented on Oct 29, 2026

andersalm commented on Oct 29, 2026

andersalm commented on Nov 4, 2026

andersalm commented on Nov 5, 2026

andersalm commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions