Skip to content

A stalled job is named while it stalls, ends libc_children, and a suite the driver ends leaves no scratch - #1229

Merged
Japabu merged 2 commits into
mainfrom
stalled-program-named
Nov 11, 2026
Merged

Japabu merged 2 commits into
mainfrom
stalled-program-named

Conversation

@Japabu

@Japabu Japabu commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

libc_children runs 23 programs in one boot, one run_test each. When a program hung, nothing named it: the row printed failures only after its last program, each later program spent its own whole guard in silence because the guest's runner was still waiting for the hung job, and the driver's 900 s silence guard ended the suite first. The killed processes' scratch then went red as a second step and stayed in /tmp on the runner (guest-rows run 42949792797). Recorded in issues/a-program-that-stalls-in-libc-children-is-not-named-and-the-driver-ends-the-suite.md, which this closes.

Resulting behaviour

  1. A job is named while it is waited for. QemuInstance::run_test says once a minute which job it waits for, how far into its guard it is and when the guest was last heard. Every QEMU row waits for its jobs there, so this holds for all of them, and the driver hears a harness at a bounded wait: its silence guard no longer outruns a job's host-scaled ceiling (a 300 s guard is up to 2400 s at the scale's cap).
  2. libc_children says each program's verdict as it ends, and ends at the first program with no exit code, naming it and the programs not run. A stalled program costs one guard.
  3. The driver removes what a group it ended left under the job's $TMPDIR and /tmp before it judges what is left (toyos_tmpdir::reclaim_gone, by the sweep's own protocol: moved under the lock into a root of the driver's there, removed outside it). Whether the steps wrote to $TMPDIR at all is read before that removal, which makes the lock it is read from. The hung step is the one red. TempDir::adopt, which did this for one pid and had no caller, is deleted.

Decisions

  • The record's proposed exit, judged. Stopping at the first program with no exit code is right, and kept: that boot runs nothing more. Alone it names the program only when its guard expires (300 s times the host scale), and only if the driver's 900 s has not fired first. Saying each verdict as it ends has the same limit. The report in run_test is what names it when it happens.
  • Other rows. Every other run_test caller in tests/toyos.rs and tests/common/ returns at its first failing job (crypto jobs, job_window sequences, the designation pair), so none goes on after a stalled one. They share only the silence, which (1) covers. No separate record.
  • reclaim_gone goes through a root of the caller's rather than deleting under the lock: /tmp is shared by every process of the user, and the lock is held for renames only.
  • A driver that ended one group reclaims every gone root under the job's own $TMPDIR, including one of a process that died by itself in another step of the same job; that step is red on its own account, and the step's line says the roots were of gone processes.
  • Any program with no exit code ends libc_children, a spawn the runner refused (error=) among them, as a failing job ends every other row; judge's header now says so rather than giving only the hung job's reason.

Checks

  • cargo test --test toyos-checks a_stalled_program: exit 0. checks::a_stalled_program_ends_its_row_and_is_named feeds the row's judge a passing program, a failing one and the result a guest silent past its guard gives (ceiling_verdict's own sentence in a WaitVerdict), and asserts the later programs were never asked for, the red names the program and those not run, and the suite files it as a blown guard.
  • Negative control: without the break, that test fails on the programs asked for (exit 101).
  • cargo test -p toyos-tmpdir at 8841591: exit 0, 13 passed, adopt's test gone (h-tmpdir.log); new a_gone_root_is_reclaimed_when_asked_and_a_live_one_never (a SIGKILLed holder's root goes, a live one, an unasked one and the lock stay).
  • cargo test --lib ci::tests at 8841591: exit 0, 21 passed (h-ci.log). what_an_ended_group_left_is_removed_and_an_earlier_root_stays writes the private directory's lock first, as its siblings do, and holds the refusal before the reclaim to naming the root under $TMPDIR in the leftovers' clause and the one under /tmp in the dead-process clause; after the reclaim both are gone and the root dead before the steps stays.
    • Its two mutations each fail it on the clause they blind (exit 101): left_behind ignoring the private directory's roots (h-blind.log, patch), and ignoring the dead roots under /tmp (h-blind-short.log, patch). Before this round the test passed with the private directory's roots ignored: its refusal was the missing lock.
  • cargo test --test toyos-checks a_stalled_program at 8841591: exit 0 (h-stalled.log).
  • cargo clippy -p toyos-tmpdir -p toyos-build --all-targets with the adopted lints at 8841591: exit 0 (h-clippy.log).
  • Guest, x86-64 QEMU on the guest runner (the row's declared target; it needs a guest because the programs are libc's children under the ToyOS kernel):
    • libc_children at 96514f2: green, run 43022077634, 8 s of testing.
    • Control, branch stalled-program-named-control (libc_read_restarts hangs in silence, bound 50 s): run 43022082892, red as intended in 86 s. At 60 s [qemu] test_rs_libc_read_restarts has run 60s of its 78s guard, the guest last heard 30s ago; at the guard libc_read_restarts: FAILED, it ended with no exit code; 1 program(s) after it were not run: tty_interrupt; STALL libc_children; nothing left in $TMPDIR or /tmp green; one red step. Bounded before dispatch: the host scale is at most 8, so the guard expires by 50 * 8 + 15 = 415 s, and with both the report and the stop broken the two waits are 830 s, inside the driver's 900 s.
  • cargo run -- --ci host at 96514f2, before this round's test, comment and roughly 20-line ci.rs change, which the checks above cover (macOS arm64, harness-ci-host-interrupted.log), the change touching the harness and the build system's ci.rs:
    • the build system's tests: red only on llvm::tests (385 passed, 1 failed), the known macOS defect issues/the-llvm-environment-test-breaks-git-on-macos.md; the new ci::tests pass;
    • the harness's own checks, the workspace's host members (toyos-tmpdir among them), toyos-crypto in its three forms, the kernel's library and the licences: green;
    • clippy, warnings denied: clean;
    • the kernel's models were voided when the shared target/ was removed under them, and the userland step was not reached. Neither covers this change, which touches no kernel code and no program under system/, apps/ or tests/guest/; by the owner's testing policy they are not owed, and the merge queue runs the full host check.
  • No aarch64 row: the change touches no AArch64 code or shared code beneath it.

Evidence: ToyOSOrg/evidence pr-1229; this round at 0a1be3a9 (linked above), the first at a2af643c: the guest runs (guest-green-43022077634.log, guest-control-43022082892.log) and the control's diff (guest-control.patch); the host tests and their control (checks-test-1.log, ci-test-1.log, tmpdir-test.log, harness-control.log, harness-control.patch); the full host check above; and the run that named the defect (run-42949792797.log).

Risks and what is not covered

  • The driver's reclaim on its kill path is held by host tests of its parts; the flag that joins them (ENDED_A_GROUP, set where the group is signalled) is read in review. No guest control exercised it: one would hold the runner for 900 s.
  • Harness waits outside run_test (await_guest, drain_until, await_exit) still say nothing while they wait. Their bounds are 120 s or less before scaling, so only a host at the scale's cap brings one near the driver's guard; not observed.

🤖 Generated with Claude Code

…te the driver ends leaves no scratch

`libc_children` runs 23 programs in one boot. A program that hung was named
nowhere: the row printed failures only after its last program, each later
program then spent its own whole guard in silence because the guest's runner
was still waiting for the hung job, and the driver's 900 s silence guard
ended the suite first, whose killed processes left their scratch for a second
red (guest-rows run 42949792797).

- `run_test` says once a minute which job it waits for, how far into its
  guard it is and when the guest was last heard. Every row waits for its jobs
  there, so a stalled job is named while it stalls in any of them, and the
  driver hears a harness at a bounded wait rather than a hung one: its guard
  no longer outruns a job's scaled ceiling.
- `libc_children` says each program's verdict as it ends, failures included,
  and ends at the first program with no exit code, naming it and the programs
  not run. It was the one row that went on after such a job; the others
  return at their first failing job.
- The driver removes what the processes of a group it ended left under the
  job's `$TMPDIR` and `/tmp` before it judges what is left
  (`toyos_tmpdir::reclaim_gone`, by the sweep's own protocol). The hung step
  is the red; its scratch no longer stays on the host until a later process
  sweeps, nor reds a second step.

Closes issues/a-program-that-stalls-in-libc-children-is-not-named-and-the-driver-ends-the-suite.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #1229 at 96514f2

Architecture

The change has three parts, and each sits with the right owner:

  • Waiting report. The once-a-minute line is in QemuInstance::run_test, the one place every QEMU row waits.
  • The row's judgement. judge is now a pure function over an injected runner, so it can be tested on the host.
  • Reclaim. toyos_tmpdir owns the reclaim protocol: rename under GLOBAL into a root of the caller's, then remove outside the lock. The driver only decides when to reclaim, through a flag set where it ends a group.

The change adds about 75 production lines and 80 test lines. It leaves one duplicate behind (B2).

Findings

BLOCKER B1 — build/ci.rs:1632: the test's "red while they stand" precondition proves nothing.

  • At that point nothing has made toyos-tmp.lock under private(&tmp). The test makes none there, and gone_roots(&short) makes only short's.
  • So left_behind(.., owed = true) refuses at ci.rs:1201 because the lock is missing, before it looks at any root.
  • The expect after the reclaim passes only because reclaim_gone's Root::make happens to create that lock.
  • So the PR's claim that "an ended group's roots under both bases are a red before the reclaim" is not established.
  • Fix: write the lock first, as the sibling tests do (ci.rs:1591, 1605), and assert that the refusal names both toyos-tmp-2-0 roots.

BLOCKER B2 — lib/tmpdir/src/lib.rs:122-133, lib/tmpdir/tests/reclaim.rs:289-330: TempDir::adopt is superseded.

  • adopt has no caller; the PR notes this.
  • Its documented purpose is "for a caller that SIGKILLed pid after this process's one sweep". That is exactly what reclaim_gone now does, and reclaim_gone does it more generally: a predicate over gone roots, removed now.
  • That leaves two public mechanisms for one job, one of them dead. Delete adopt and an_adopted_root_goes_with_its_adopter here.
  • The liveness rule that test also covered (the lock decides, not the pid) is gone()'s, and the new test's live holder exercises it.

SUGGESTION (optional):

  • tests/common/libcchildren.rs:58-61, 75-82: the row also stops at a job the runner reported as error=, meaning a spawn or a duplicate was refused (tests/guest/test-runner/src/main.rs:281, 299, 332, 338).
    • That job did end, and the runner then reads the next run. So the doc's reason ("a boot whose job never ended runs nothing after it") does not hold for it, yet the red lists the later programs as "not run".
    • Either break only when run_test's own wait ended (the ceiling, or QEMU gone), or have the doc say that any job without an exit code ends the row, as every other row already does.
  • build/ci.rs:1171-1172: reclaim_ended runs before left_behind.
    • reclaim_gone creates the lock that left_behind's owed check takes as proof that the steps wrote to this $TMPDIR. So on a run where the driver ended a group, that misdirection check no longer bites. Read the lock before reclaiming.
    • The success text says the roots were "of the processes this driver ended", but |_| true takes any gone root under the job's $TMPDIR.

Correctness (verified)

  • judge's pass arm. Some(0) is equivalent to the old (None, Some(0)): run_test never returns an exit code together with an error (tests/common/qemu.rs:2014-2030).
  • Other rows. They return at their first failing job (tests/toyos.rs:5988-5996, job_window at 5305-5314), as the PR says.
  • The report reaches the driver.
    • The report comes from run_test's loop, which runs at least every VERDICT_POLL.
    • It goes through the harness's stderr printer; toyos-build is harness = false, so libtest does not capture it.
    • heard reads that stream, and 60 s is far inside the 900 s QUIET.
  • reclaim_gone liveness.
    • Liveness is the owner lock.
    • flock conflicts across open file descriptions even within one process, so the driver's own live roots are not taken.
    • The function removes its own root afterwards, and the tmpdir test confirms this with a real SIGKILLed holder.

Is the claim held by a test?

  • Naming and ending the row: yes. a_stalled_program_ends_its_row_and_is_named checks it, and its control (the break removed) fails with exit 101.
  • The guest control (run 43022082892) shows the once-a-minute line, the naming, the STALL classification and a single red step.

Not exercised end to end

  1. The driver's kill path. Nothing runs the whole chain: heard ending a group after 900 s of silence, ENDED_A_GROUP set, then nothing_left reclaiming a really killed test binary's roots under the job's $TMPDIR and /tmp.
    • Its parts are tested: the tmpdir test uses a real SIGKILL; the ci test uses fabricated roots with no owner. The flag joining them is read in review.
    • In the guest control the row ended itself, so its green "nothing left" step came from normal unwinding, not from the reclaim.
  2. The driver's guard through a long wait. Nothing shows the guard staying quiet through a run_test wait longer than 900 s; the control ran 86 s. This is arithmetic.
  3. Several programs after a stall. On a guest the control had one program after it; the host test has two.
  4. A runner-reported error= under the new break.
  5. Waits outside run_test, as the PR states.

None of these needs a new guest run.

SEND BACK (B1, B2). Both fixes are host-only: cargo test --lib ci::tests::what_an_ended and cargo test -p toyos-tmpdir.

…adopt goes with the job reclaim_gone took

The ended-group test's "red while they stand" passed on the private
directory's missing lock, before `left_behind` looked at a root. It now
writes the lock as its siblings do and holds the refusal to naming the root
under each base, each in its own clause; a `left_behind` blind to either
base fails it.

`nothing_left` reads whether the steps wrote to `$TMPDIR` before the
reclaim, which makes that lock itself, so a driver that ended a group still
refuses a `$TMPDIR` the steps never saw. Its line says the roots it removed
were of gone processes, which is what it takes.

`TempDir::adopt` had no caller and did for one pid what `reclaim_gone` does
for every gone root; it and its test are deleted. `judge`'s header says
that any program with no exit code ends the row, a refused spawn among
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit to ToyOSOrg/evidence that referenced this pull request Nov 11, 2026
…ts named before the reclaim, two mutations, toyos-tmpdir without adopt
@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Round 2 at 8841591 (one commit on 96514f2):

  • B1 — the test writes the private directory's lock first and holds the refusal before the reclaim to naming the $TMPDIR root in the leftovers' clause and the /tmp root in the dead-process clause (a plain contains was not enough: the /tmp root's full path ends in the same toyos-<uid>/toyos-tmp-2-0). Two mutations, left_behind blind to either base, each fail it (exit 101); at 96514f2 the test passed with the private directory's roots ignored.
  • B2 — TempDir::adopt and an_adopted_root_goes_with_its_adopter deleted.
  • Suggestions taken: the lock is read before the reclaim that creates it (never_written, shared with left_behind); the step's line says the roots were of gone processes; judge's header says any job without an exit code ends the row.

Checks: cargo test --lib ci::tests (21 passed), cargo test -p toyos-tmpdir (13 passed), cargo test --test toyos-checks a_stalled_program, clippy on toyos-tmpdir and toyos-build; logs at evidence pr-1229 @ 0a1be3a9.

@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review of #1229 at 8841591 (round 2)

Architecture

The architecture is the same as in round 1. This round deletes the duplicate mechanism and makes the test meaningful.

  • Net −37 lines. adopt and its 46-line test are deleted.
  • never_written is the old inline lock check, moved into a function so that nothing_left can read the lock before the reclaim creates it.

Previous blockers

  • B1: resolved.
    • build/ci.rs:1638 writes the private directory's lock first.
    • 1647-1653 require two clauses in the refusal: the $TMPDIR root (<private>/toyos-tmp-2-0) in the leftovers clause, and the /tmp root's full path in the dead-process clause.
    • Splitting on "; " is sound. left_behind joins its clauses with exactly that (ci.rs:1237), and neither clause's fixed text contains it.
    • Both mutations make the split fail, exit 101, at 8841591 plus their patches. h-blind.patch drops the private roots from left; h-blind-short.patch drops the dead roots.
    • After the reclaim, both roots are gone and the earlier root stays.
  • B2: resolved.
    • TempDir::adopt and an_adopted_root_goes_with_its_adopter are deleted, and nothing references adopt any more.
    • reap_into still serves sweep and reclaim_gone.
    • reclaim.rs still uses OpenOptions and OWNER in its other tests.

Suggestions taken (checked)

  • nothing_left (ci.rs:1173) refuses a $TMPDIR the steps never wrote to before reclaim_ended runs. reclaim_ended → Root::make → global creates the lock (lib/tmpdir/src/lib.rs:256, 436), so the order matters, and the doc sentence says why.
  • The step's line now says "root(s) of gone processes", which is what |_| true takes.
  • judge's header matches its None arm, which breaks for any job without an exit code.

Evidence

  • h-ci.log (21 passed), h-tmpdir.log (13), h-stalled.log and h-clippy.log (exit 0) all ran at 8841591 with an empty worktree diff (da39a3ee5e6b is the empty diff's hash).
  • That is sufficient for a host-only change. The round touches nothing a guest row runs: libcchildren.rs changed only a doc comment.

PR text (for the orchestrator)

The full-host-check bullet calls this round's ci.rs change "three-line". The production change is about 20 lines: never_written moved out and the check before the reclaim added. The targeted checks above cover it, so nothing more is owed.

Findings

None.

LAND

@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator: follow-up review r2 LAND at 8841591 (B1 and B2 resolved; host-only change, evidence at a clean tree). I corrected the body's size of this round's ci.rs change. Queueing.

@Japabu
Japabu marked this pull request as ready for review November 11, 2026 17:37
@Japabu
Japabu added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 45935f4 Nov 11, 2026
10 checks passed
@Japabu
Japabu deleted the stalled-program-named branch November 11, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant