Skip to content

The fatal path waits a second on the clock for the console registers, not 10^8 pauses - #946

Merged
Japabu merged 2 commits into
mainfrom
wt/toyos-panic-spin-time
Oct 15, 2026
Merged

Japabu merged 2 commits into
mainfrom
wt/toyos-panic-spin-time

Conversation

@Japabu

@Japabu Japabu commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

What this changes

serial::panic_registers now waits for another holder of the console registers for PANIC_LOCK_WAIT, a one-second Budget on the clock, polled through clock::settles, instead of 100,000,000 try_lock + pause iterations that the old comment called "~1s". serial_lock loses its count-bounded within and Seized::Expired; BackendLock::try_seize asks once and the caller owns the wait. Before calibration settles asks once.

Why

The count was not time: the wait lasted however long 10^8 pauses took on the CPU running them. clock::settles is the kernel's existing bounded poll. It compares the CPU's counter against a deadline converted once, without calling the nanosecond clock's divide, so it is as usable from an NMI as the count was.

Uncalibrated, one ask is correct: kernel_main runs arch::boot::clock (which sets the counter period) before arch::boot::start_other_cpus on both architectures. A fatal path before calibration therefore runs on the only CPU, any holder is beneath its own frame, and no wait could end with the registers released. panic_console::seize uses the same reasoning.

No host test with an injected clock was added. serial_lock no longer contains timing logic, and the wait is clock::settles, which is shared with every other bounded poll in the kernel. Injecting a clock into clock.rs only for this would be machinery for its own sake. The loom models now drive try_seize with their own bounded asks. The model that checked within's exact count is deleted along with the count.

Evidence

All run on the tree of 06a3a53a8 (its commit adds only its message over the tested tree), macOS host. Logs, home and temp paths masked, are at evidence commit 65c149b14d63238ea74fc1790454ed743e459b12 on pr-946. The later head commit rewraps two issue paragraphs only, so no code, test or build input differs from the tested tree.

  • cargo test --manifest-path kernel/loom/Cargo.toml --test serial_lock: EXIT=0, 8 passed (log).
  • cargo run -- --ci host: EXIT=0, "Host: 86 step(s), all green" (includes clippy -D warnings for both kernel arches and the serial-try-lock-then-some negative control, which still reds serial_lock) (log).
  • cargo run -- --build-only: EXIT=0 (log).
  • cargo test --test toyos-build -- nested_nmi_is_loud: EXIT=0. x86-64 QEMU metal profile. A fatal path nested on a fatal path, with no "held through the bound" / "drained raw" line (faults::UNCLEAN) (log).
  • cargo test --test toyos-build -- machine_shutdown_wire_kept virt_early_panic virt_fatal_halts_the_others_first: EXIT=0, 3 passed. virt_early_panic panics before arch::boot::clock, so it runs panic_registers through the uncalibrated one-ask path; virt_fatal_halts_the_others_first is the multi-CPU AArch64 fatal path (log).

No guest test holds the registers through the whole bound on another CPU. The one-second expiry itself is settles' contract and is not exercised by a run.

What is unsure

  • The bound value (1 s) keeps the old comment's stated intent; it was not re-derived. A burst holds the registers for microseconds. Another CPU's fatal path can hold them for its whole report, as before.

Anything a reader of main must not miss

Closes issues/the-panic-lock-spin-limit-is-a-count-its-comment-calls-a-second.md. Updates the citation in issues/a-syscall-panic-reset-the-guest-announced-and-qemu-never-saw.md and the PANIC_LOCK_SPIN_LIMIT/seize names in issues/a-fatal-path-inside-its-own-cpus-console-burst-waits-out-the-bound.md. Deletes the loom model within_asks_a_silent_attempt_exactly_its_tries along with the function it tested.

… not 10^8 pauses

`PANIC_LOCK_SPIN_LIMIT` bounded `panic_registers`' wait for another holder
of `BackendGuard` at 100,000,000 `try_lock`s and `pause`s, and its comment
called that "~1s of spin". Nothing converted it to time, so the wait lasted
whatever that many pauses cost on the CPU running them.

The wait is now `PANIC_LOCK_WAIT`, a one-second `Budget`, polled through
`clock::settles`, the kernel's existing bounded poll: it compares the CPU's
free-running counter against a deadline converted once, never the
nanosecond clock's divide, so it is as safe from an NMI as the count was.
`serial_lock` loses its count-bounded `within` and its `Seized::Expired`;
`BackendLock::try_seize` asks once, and the waiting is the caller's.

Before the clock is calibrated `settles` asks once. That is the right
answer here: on both architectures `kernel_main` runs `arch::boot::clock`,
which sets the counter's period, before `arch::boot::start_other_cpus`, so an uncalibrated fatal path runs on
the only CPU, any holder is beneath its frame, and no wait could end with
the registers let go. `panic_console::seize` reasons the same way.

The loom models drive `try_seize` and their own bounded asks; the model
that checked `within` asked exactly its count is deleted with the count.

Closes issues/the-panic-lock-spin-limit-is-a-count-its-comment-calls-a-second.md;
the two issues that named it or `PANIC_LOCK_SPIN_LIMIT` are updated.
@Japabu

Japabu commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #946 at 06a3a53 (base origin/main 979d4c5)

Assessment. This is the right change, made in the right place. The lock (serial_lock) now answers one question per call: try_seize returns taken, reentered or refused. The one caller that has a policy, serial::panic_registers, owns the wait and bounds it with the kernel's shared bounded poll, clock::settles, against a named Budget. That removes a parameter, an enum variant and a helper function, and nothing is added in their place. Net: +59/−92, including the closed issue's −18.

NMI and fatal-path safety of clock::settles. Per call, it does three things:

  • one relaxed load (calibrated)
  • one conversion: counter_ticks, a single u128 divide, which is a pure, reentrant compiler_builtins call
  • then a loop of cpu::counter() reads and ready() calls

It takes no lock, does no logging, does no MMIO beyond what try_seize already did, and allocates nothing. The deadline is taken and compared on one CPU under the IrqGuard panic_registers already holds, so cross-CPU counter skew cannot enter the comparison. The closure's Cell<Option<Seized>> only ever overwrites a None: the loop stops on the first Some. So no Held is dropped early, and the None left after expiry is exactly the refusal. Budget::of is const, as other Budget constants in the kernel are.

The uncalibrated argument holds on both architectures. kernel_main runs arch::boot::clock at main.rs:429, and start_other_cpus at :463. The two set_counter sites are x86 hpet.rs:70 (the HPET calibration, which asserts rather than returning uncalibrated on a stall) and AArch64 boot.rs:352. Neither can store a zero period on a real part. So !calibrated() implies one CPU, and with interrupts closed any holder lies beneath this frame: one ask is the full answer. The early-panic branch (main.rs:138-150) does reach panic_flush → panic_registers, so the claim that virt_early_panic runs the one-ask path is correct. The reasoning matches panic_console::seize.

Loom. within_asks_a_silent_attempt_exactly_its_tries tested only the deleted count function. Its deletion is correct, and nothing it established survives to need cover. The other six models keep their claims and are re-expressed over try_seize:

  • release on another CPU becomes takeable
  • a kept hold is refused
  • reentry
  • mutual exclusion against a burst
  • a burst refused beneath its own fatal path
  • a burst waiting out another CPU's

In a_seize_takes_a_burst_let_go_on_another_cpu the retry loop is now the model's own, not the kernel's. What it still establishes is the cross-thread release→try_seize edge, and that is the property loom can check. The serial-try-lock-then-some red row in build/ci.rs names only try_lock tests, which are unchanged.

Issue edits. Both are legitimate.

  • a-syscall-panic-…-never-saw.md pointed at the file this PR deletes. The edit removes the dangling path and keeps the measurement it recorded, which is what issues/README.md's "a pointer is a claim something can check" asks for.
  • a-fatal-path-inside-its-own-cpus-console-burst-waits-out-the-bound.md describes the mechanism this PR changes, in the present tense. Its identifiers would otherwise name deleted items. The new sentence is accurate, and the defect still holds once the clock is calibrated.

Findings

  • BLOCKER: PR body, "Evidence". All five checks (loom serial_lock, --ci host, --build-only, nested_nmi_is_loud, and the three-test virt_* / machine_shutdown_wire_kept run) are stated with "logs kept in the implementer's scratch directory". The evidence repo has no pr-946 branch (gh api repos/ToyOSOrg/evidence/branches lists pr-944 and pr-945 only). implementer.md requires evidence a PR relies on to survive session cleanup, published on pr-<N> and linked at a full commit hash. These are the PR's only behavioural evidence for a fatal-path change: the uncalibrated path and the nested-NMI path. Unpublished, they are unverifiable claims. Publish the existing logs and link them. No rerun is needed: the head commit's tree is the tested tree.
  • SUGGESTION: kernel/src/drivers/serial.rs:151 and WRITTEN_OVER (:162). Before calibration, Hold::Expired's doc ("kept them through the whole bound") and the wire text ("stayed held through the bound") now describe a single ask. Both remain true if one reads "the bound" as settles' bound, so changing them is optional.
  • SUGGESTION: issues/a-fatal-path-…-waits-out-the-bound.md:14. The rewrap left one line about 100 columns wide among 78-column lines.

SEND BACK

@Japabu

Japabu commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Round 1 resolution, head c59df381c9db3f41305caa31880df67f4ceb798e:

  • BLOCKER (evidence unpublished): fixed. The five logs are published on pr-946 at https://github.com/ToyOSOrg/evidence/tree/65c149b14d63238ea74fc1790454ed743e459b12/06a3a53a8. Home and temp paths are masked, and a scan found no MAC, IPv4, IPv6, serial or hostname shapes. The PR body links each log at that hash. These are the logs from the original runs; nothing was rerun, because the tested tree is 06a3a53a8's.
  • SUGGESTION issues/…waits-out-the-bound.md:14 width: taken. I also rewrapped the lines this branch had made too long in a-syscall-panic-…-never-saw.md. The new commit c59df381c changes only those two issue files' prose, so no code, test or build input differs from the tested tree, and the evidence still applies.
  • SUGGESTION serial.rs:151/:162 "through the bound": declined. Both read correctly as settles' bound, which before calibration is a single ask. WRITTEN_OVER's text is also matched by tests/common/faults.rs UNCLEAN, so rewording it would mean another coordinated change and a guest rerun, for no change in what the text claims.

@Japabu

Japabu commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Review round 2: #946 at c59df381c (base origin/main 979d4c539)

Round-1 blocker (evidence unpublished): resolved.

  • pr-946 exists on ToyOSOrg/evidence. Its tip is 65c149b14d63238ea74fc1790454ed743e459b12; compare reports identical. The PR body links every log at that full hash, and each link resolves.
  • Each log supports its claim:
    • loom.log: kernel/loom serial_lock, 8 passed, EXIT=0. The six re-expressed models are present, including a_burst_beneath_its_own_cpus_fatal_path_is_refused - should panic. within_asks_… is gone.
    • ci-host.log: "Host: 86 step(s), all green", EXIT=0.
    • build-only.log: "Build finished.", EXIT=0.
    • guest-nested-nmi.log: nested_nmi_is_loud PASS on x86_64, EXIT=0.
    • guest-panic.log: virt_early_panic (aarch64, the uncalibrated one-ask path), virt_fatal_halts_the_others_first (aarch64, multi-CPU) and machine_shutdown_wire_kept (x86_64) each PASS, 3 passed, EXIT=0.
  • Home and work paths are masked (<work>/…). I saw no personal or machine identifiers. The build log's signing line names a throwaway checkout key, not an identity.
  • Revision. The logs do not print the revision themselves. The tested tree is identified as 06a3a53a8's, by the evidence directory name and the PR body. I accept that: c59df381c changes only two issues/ files' prose (+9/−9), so nothing that compiles or runs differs.

New commit c59df381c (issue rewraps). It is prose only, and the wording is unchanged apart from the wrapping. Declining the serial.rs:151/:162 suggestion is reasonable: WRITTEN_OVER is matched by faults::UNCLEAN, and both texts read correctly as settles' bound.

BLOCKERS

None.

SUGGESTIONS

  1. issues/a-syscall-panic-reset-the-guest-announced-and-qemu-never-saw.md:35: the rewrap moved the over-long line rather than removing it. "~95 s this red ran before giving up, and it still reset inside budget. A held console wire is therefore not what happened" is about 120 columns among 78-column lines. Line 23 is a quoted log line and was already that long on main. This is cosmetic.
  2. In future evidence, printing git rev-parse HEAD at the top of each log would make the tested revision self-evident, without relying on the directory name.

LAND

@Japabu
Japabu marked this pull request as ready for review October 15, 2026 10:01
@Japabu
Japabu added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit 3bd5ed4 Oct 15, 2026
6 checks passed
@Japabu
Japabu deleted the wt/toyos-panic-spin-time branch October 15, 2026 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant