Skip to content

osbench: a row says where its processes ran - #1126

Merged
Japabu merged 2 commits into
mainfrom
osbench-ran
Nov 1, 2026
Merged

Japabu merged 2 commits into
mainfrom
osbench-ran

Conversation

@Japabu

@Japabu Japabu commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A request/reply row that reads high on a machine with two LLCs may only have run its two sides in different caches. On the AMD bench, ipc.reqrep.8 and the fs.disk rows read about a quarter slower in some boots than in others. Nothing in a row said where either side had run, so a scheduler A/B could not tell placement from the change.

Behaviour

  • Each row that measure::each times, and each of ipc's request/reply rows, now ends with ran=name.pid@cpuN/llcL:share%,....
    • It lists up to four processes the kernel's diary saw picked during the row's warmup and timed samples, most picked first.
    • Each comes with the CPU it was picked on most, that CPU's LLC and the share of its picks there.
    • ran_lost counts records the rings overwrote.
  • The diary is drained before the warmup and read after the samples, so neither read is timed. The opening drain reads every ring, several MiB on the AMD bench, so it comes before the warmup, which then rebuilds the working set the samples run in. A drain that fails keeps no diary handle, so the field never reports records from before the span. A boot without the trace capability adds nothing.
  • Linux adds nothing. This field reads the ToyOS kernel's own diary. Linux has no equivalent a process may read. The field reads nothing inside the timed span, so both systems' rows stay comparable. No A/B compares a row with and without the field; its only cost before the samples is now the warmup's.
  • The LLC comes from CPUID's deterministic cache parameters at their highest level: leaf 04H on Intel (SDM Vol. 2A) and 8000_001DH on AMD (APM Vol. 3). The APIC IDs sharing a cache differ only below the sharing count rounded up to a power of two.

Checks

Evidence: pr-1126/r2 at 3ccd1580; round 1 pr-1126/r1.

check revision result
cargo run -- --ci host fc90d1018 (review r1, on main 24a538c88), clean tree 107 steps, all green, exit 0
cargo run -- --ci host 6adab470b 107 steps, all green, exit 0
osbench_ipc and osbench_files on 4 TCG CPUs (a run-time edit of osbench_capability_groups_exit_0, never committed) 6adab470b exit 0; every ipc and fs row carries ran=. TCG's CPU describes no cache, so its LLC reads ?
metal: 9 AMD boots and 2 T14 boots of this commit on main 69512b82b, alone and under the scheduler stack d51d28819 and its stack arms osbench passed on all 11. On the AMD, the LLC column matches the kernel's own topo: lines (cpu0–7 LLC 0, cpu8–15 LLC 1) in every row. The T14 reads LLC 0 throughout, as its single L3 is.

The metal boots ran the field read after the warmup, before review r1 moved it; the move changes when the diary is read, not what it holds. They are the scheduler AMD diagnostic: pr-1111/r2. Its placement.txt is what this field was written for. Each ipc.reqrep.8 and fs.disk.read_4k_at value there follows where its pair ran: 8.4–8.5 µs split across the LLCs against 6.7–6.9 µs within one.

Risks

  • The field reads only what the diary keeps. A row that overwrites its rings reports its latest picks, with ran_lost saying how many records went. Each CPU has its own ring, so a busy CPU's share is then counted short against a quiet one's.
  • The field now spans the warmup too, so a pair that moves during the warmup shows a share under 100%.
  • The aggregation (most-picked CPU, share, top four) has no host test.
  • A process whose picks are split across CPUs shows its mode and its share. A share well under 100% says the side moved.

🤖 Generated with Claude Code

A request/reply row that reads high on a machine of two LLCs may only
have run its two sides in different caches. On the AMD bench the ipc
reqrep and fs.disk rows read about a quarter slower in some boots than
in others, and nothing in the row said where either side had run.

Each row that measure::each times, and each of ipc's request/reply rows,
now ends with ran=name.pid@cpuN/llcL:share%. It lists up to four
processes the kernel's diary saw picked during the timed samples, most
picked first, each with the CPU it was picked on most, that CPU's LLC and
the share of its picks there; ran_lost counts records the rings
overwrote. The LLC comes from CPUID's deterministic cache parameters
(leaf 04H on Intel, 8000_001DH on AMD), at their highest level. The diary
is drained before the samples and read after them, so neither read is
timed. Linux, and a boot without the trace capability, add nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ran::start reads the whole diary, 16 rings of 8,192 records on the AMD
bench, into a vector of several MiB, and it ran after the warmup and the
clock's settle, immediately before the first timed sample. So it evicted
the working set the warmup had just built, and nothing compared a row
with and without it. Each row now opens the span before its warmup, which
rebuilds what the samples run in; the field covers warmup and samples,
which the header says.

A drain that failed partway was ignored, so the field could report records
from before the span. start now keeps no diary handle when its read
fails. The header also notes that ran_lost biases the share against a busy
CPU, whose ring overwrites first, and drops the rationale sentence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Review r2 of #1126 at fc90d1018

Round-1 required finding: fixed. ran::start() now opens before the warmup at all three call sites: measure.rs:24, ipc.rs:78 (after control, which sets up the peer) and ipc.rs:127. So the diary drain's cache damage is rebuilt by the warmup and is not timed.

Also checked:

  • A failed opening drain now keeps no handle (ran.rs start), so fields can never report pre-span records.
  • The module header now states the contract (the span covers the warmup) and the ran_lost bias, and drops the rationale.
  • The body states that the metal boots predate the move, and why that leaves the LLC check valid.
  • Host CI at fc90d1018: 107 steps green, exit 0 (r16/ci-host-1126.log, rev file matches).

No blocker. LAND.

@Japabu
Japabu marked this pull request as ready for review November 1, 2026 12:00
@Japabu
Japabu added this pull request to the merge queue Nov 1, 2026
Merged via the queue into main with commit adf9e53 Nov 1, 2026
10 checks passed
@Japabu
Japabu deleted the osbench-ran branch November 1, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant