Skip to content

A reserved write's bytes go to the device at once, at the session's depth - #1216

Merged
Japabu merged 7 commits into
mainfrom
seqwrite
Nov 11, 2026
Merged

Japabu merged 7 commits into
mainfrom
seqwrite

Conversation

@Japabu

@Japabu Japabu commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Status: draft. Review r2 (on d3d67f8) is LAND. Both metal benches are down with no date for their return, so this PR lands on its reviews, its host checks and its guest row nvme_disk_boots at d3d67f8 (run 43070793437); the head, 7ad9bca, changes only the issue record and is code-identical. The speed-up of the code that lands is not measured on metal. Every gain figure below is cc718f6's, from the first A/B, before review r1's fix; the second A/B that was to measure the landing code was staged and never booted. What is owed on metal is listed below and in the record. This PR is a step: it does not reach Linux's figure.

Problem

persist.seq_write_fsync (8 MiB in 1 MiB calls, then sync_all) reads about half Linux's on both benches: 243 to 274 MB/s against 493 to 520 on the AMD bench, 138 to 259 against 532 to 593 on the T14.

Where a round's time went on main (diagnostic round, main with stage timers, 2 boots a bench)

AMD, ms a round, ToyOS against Linux on the same drive in the same session:

ToyOS main Linux
create (truncates the last round's 8 MiB) 4.1 2.4
eight writes into the file's pages 4.5 5.5
fsync 22.8 8.0

Of main's fsync: 128 device round trips, one at a time, 9.4 ms (the drive itself answered in 6.1); the extents' transactions, checksums and buffers 6.1 ms; the commit 5.1 ms, of which five Flushes 4.5; reading the pages out 0.9 ms. Linux sends the round as 66 writes of at most 128 KiB and one Flush.

T14: every ToyOS boot of that round ran in the drive's slow mode. Main's fsync was 35 to 47 ms: round trips 15, commit 12 to 23 (five Flushes of 2.3 ms and up).

Change

  1. A reserved write's bytes go to the device at once. lib/bcachefs cuts and seals a reserved write's extents first, sends each stretch of contiguous sectors as one device write, then makes the pointers. system/diskserver's served disk asks every request of a write at once. A round is 4 device writes of 2 MiB, each 16 requests of 128 KiB in flight together, instead of 128 awaited one by one.
  2. Up to eight extents a transaction. A reserved write points up to eight fresh extents of one bucket in one transaction, and a truncation removes up to eight whole extents of one bucket in one. The group changes its bucket once: one alloc change, one move of its fragmentation-LRU entry from the fill before to the fill after, one accounting change, and the inode once. A round is 32 transactions instead of 256, and its commit one journal entry between two Flushes instead of three between five.

Eight comes from the transaction's own bounds. Each extent past the first adds only its extent key and its backpointer, in at most one node more of each of those trees; the bucket's LRU entry touches at most two LRU leaves for the whole group. Measured, largest transaction (journal u64s of 512, nodes of 28): a streaming write 212 and 7 (encrypted 228 and 7), its truncation 202 and 8, and a full group whose bucket's LRU entry moves from 1/16 to 9/16 full 203 and 6 with the LRU in one leaf, 203 and 8 with it spread over many leaves. Tests hold a quarter of the journal bound and half the footprint to spare on both volumes; with 64 a transaction the engine's own assertion stops it. Before review r1 each piece changed its bucket on its own: 247 and 263 u64s streaming, and 11 nodes on the spread LRU.

No interface changes and no manual page changes: the block layer's and the format's decisions stand.

The first A/B, of cc718f6 (before review r1's fix)

M is main ad5db5a with the osbench_disk row only, P this PR's code at cc718f6 with the same row only; no timers in either. Four boots an arm a bench, interleaved, and Linux's osbench persist after rounds 1 to 3. A T14 boot is in its drive's slow mode when its persist.write_4k_fsync p50 is over 3 ms; the T14 is compared mode by mode as well as over all boots.

persist.seq_write_fsync and fs.disk.seq_write.1048576, MB/s, each boot:

bench main (M) this PR (P) P against M, medians Linux
AMD 242, 241, 262, 239; 452, 454, 460, 455 375, 404, 409, 403; 668, 672, 676, 681 +67%; +48%, both resolved 491, 495, 468; 806, 805, 800
T14 slow 156, 169, 182; 455, 485, 466 228, 217, 202; 563, 573, 569 +28%; +22%, both resolved no slow run this session
T14 fast 245; 368 (one boot) 349; 498 (one boot) not established 579, 603, 609; 347, 349, 344
  • No regression (A1): holds on both benches. No fs.disk.*, nvme.* or persist.* row is more than 5% worse in P and resolved, over all boots on either bench or in the T14's slow mode alone. The rows P reads worse at the median are unresolved and outside this change's path: on the AMD the file-server requests (fstat, stat, open_close, direct.*, 7 to 12%) follow where the client and the server ran in each boot, slower across LLCs or on what look like two threads of one core, and the memory file system's rows, which this change does not reach, split the same way in both arms; on the T14 fs.disk.seek and fs.disk.read_4k, which answer in under a microsecond without a request to the server, in code this PR does not change.
  • Correct (A2): holds. The row judges pass every boot, run offline over the readbacks (each M, P and timer boot under osbench_disk, the round trip under bench_data_survives_reboot; a local, uncommitted patch only pointed them at a copy of the benches' records). Every boot mounted DATA once and formatted nothing, found its directories, and logged no DATA error, failed or reissued request, Lost flush, stopped mount or panic; every test exited 0. The reboot round trip at the head (256 KiB written and synced, read back after a reboot) passed both boots on the T14.
  • Gain (A3): the table. The T14's fast mode is not established with one boot an arm.
  • Against Linux (A4, not a gate of this PR): does not hold. On the AMD P is 18% short on the durable row and 16% on the 1 MiB row; on the T14, in the fast mode Linux's runs were all in, P's one fast boot is above Linux on the 1 MiB row without a sync (498 against 344 to 349) and 40 to 43% short on the durable row (349 against 579 to 609).

Where the round goes at cc718f6 (one boot a bench with stage timers)

  • AMD: fsync about 14 ms against Linux's 9 to 10 this session. Writing the 8 MiB back is 9.7 ms: 5.3 ms in the device's call, with up to 15 commands held and 1.2 ms of it copying the blocks for the session, and 4 ms of the extents' lookups, building and checksums before it and their pointer transactions after it, none of it beside the device's time. The commit is 2.2 to 2.6 ms, two Flushes. Reading the file's pages out is 0.8 ms; about 1.3 ms is not split. Expected before the boots: 12 to 14 ms and about 400 MB/s.
  • T14, slow mode: fsync 29 to 31 ms. The device took 11 to 13 ms for the 8 MiB and the commit's two Flushes 8.5 to 11.4 ms. Expected before the boots was 22 to 26 ms; the drive's slow mode decides it.

Durability

Unchanged. A pointer is made only after its bytes completed, and the flush before a journal entry still orders them first; a commit still ends with a Flush after its last entry. Grouping changes how many keys one transaction carries, not when anything becomes durable, and a group is one transaction, appended whole to one journal entry. Evidence: a crash-model test of a three-extent reserved write whose first extent replaces an old one and whose other two are fresh in one bucket, each transaction committed as it applies (a test-only switch) and the device cut at every request, torn and reordered: every crash state holds none, the first or all three extents, never two, and checks clean, and the crashes do cut between the lone extent and the group (red with grouping off: two of three at request 16). Also the existing crash suites, and a control in which pointers are made before the bytes (2 tests red). On metal, 256 KiB written, synced and verified after a reboot passed on the T14 at 5a90355 and at cc718f6. One behavioural difference: a stopped mount refuses a reserved write before sending bytes.

Checks

  • Review r1's tests: the crash-model test above; the bounds test now asserts one pointer transaction for each eight extents (one more on the encrypted volume, its version mark); a group moving its LRU entry across leaves; and the bench's round six times on 512 KiB buckets, remounted between rounds so emptied buckets are refilled in groups, fsck clean after each round and the empty-bucket count back where it started once the file is gone.
  • Negative controls, each red: pointers before bytes, a device write per extent, requests one at a time, 64 extents a transaction; and after review r1, grouping off (crash test two of three; 80 transactions for 80 extents) and a bucket drained one sector short (the reuse test's fsck). The code before the fix, a bucket change a piece, is no red: its footprint is within the margins too, by the figures above.
  • On the tree committed as da1c40c, but for one comment, for the crates it touches: engine suite exit 0 (204 passed), file server and served disk suites exit 0, cargo clippy -p bcachefs -p diskserver --all-targets with the workspace's adopted lints and -D warnings exit 0. The record commit d3d67f8: cargo test --lib sourcegate exit 0. The full cargo run -- --ci host was not rerun (the owner's testing policy: touched crates only; the merge queue runs it).
  • Guest row nvme_disk_boots (DATA served off the NVMe driver's real session; QEMU x86-64): pass at 5a90355 and cc718f6, runs 42986107985 and 43018266273; at d3d67f8, the code that lands, run 43070793437, queued. It is the landing condition for the code; QEMU's clock gives no timing claim.
  • cargo run -- --ci host at cc718f6: exit 1, 109 of 110 steps green; the red is the macOS-only llvm environment test (issues/the-llvm-environment-test-breaks-git-on-macos.md), which this branch does not touch.
  • Metal: the first A/B above and its reboot round trip, of cc718f6, 20 ToyOS boots and 6 Linux runs over both benches, every boot's verdict passed and its row judge 1 passed, 0 failed. Nothing of the code that lands has run on metal: the second A/B (the reviewed head against main, ABBA on each bench; rules fixed before staging and amended before any boot) never booted, because both benches went down. Its images are deleted; what it was to establish is owed, below.
  • The record commit 7ad9bca: cargo test --lib sourcegate exit 0. No guest run for it: it changes only the record.
  • Evidence: https://github.com/ToyOSOrg/evidence/blob/5358ac0cc6804c914b7a465e95235a4324c79a4d/README.md (r1: host model, suites, controls, first guest row; r2: the diagnostic metal round, the second commit's checks, the first A/B's rules; r3: the first A/B judged; r4: review r1's fixes, footprint, controls, the second A/B staged; r5: its rules amended before its boots; r6: the benches down, what is owed)

Record

issues/a-sequential-durable-write-on-data-runs-at-about-half-linuxs.md is new, carries the first A/B's figures and split and what is owed on metal, and stays open after this PR.

Owed on metal (in place of the metal landing conditions)

When a bench returns, main after this PR owes, as the record states:

  • The A/B of main from before this PR against main after it, row osbench_disk (files, nvme, persist on the bench's NVMe), on the T14 and the AMD bench, ordered before, after, after, before on each bench, with DATA carried from boot to boot and every mount's bytes free recorded. Main after must show every boot passing its row judge; DATA mounted once, nothing formatted and no refusal from the storage path; no fs.disk.*, nvme.* or persist.* row more than 5% worse and resolved; persist.seq_write_fsync and fs.disk.seq_write.1048576 above before's; and no drop of the free figure at the mount after one of its boots. The T14's fast mode needs two boots a side or is reported as not established.
  • The durable rows against Linux: osbench persist under each bench's Linux on the same NVMe, in the same session, on both benches; the record closes when main is at or above Linux's.
  • bench_data_survives_reboot on the T14: 256 KiB written and synced, read back whole after a reboot (passed at cc718f6).

Not established, and risks

  • The speed-up of the code that lands is not measured on metal (owed above). The T14's fast mode: one boot an arm in the first A/B; not established.
  • The T14's DATA free figure fell by 512 KiB: one more occupied bucket or one more btree node. It fell after r4-P (cc718f6) and after no other boot of either bench; nothing in r4-P's log refuses anything. On the host the reuse-cycle test refills emptied buckets in groups six times and the empty-bucket count comes back, with fsck clean. The owed A/B records every mount's figure: back up means transient metadata, steady means a one-time change, a further drop after a boot of main after this PR means a leak. The record carries it until then.
  • What is left on the AMD, by the split: about 4 ms of the engine's work done before and after the device's time rather than beside it, which the engine can overlap, and about 2 ms of copies, the pages read out and the blocks copied for the session, which belong with the move of file data to paged objects and are not in this PR. On the T14 in its slow mode the drive decides the round.
  • Main has moved since the base (ad5db5a); the branch is not rebased, so both A/B arms share one base.

Review history

  • r1, the review of e64953f: SEND BACK. (1) POINTED's derivation was false for the write side: each piece changed its bucket, so its LRU entry, on its own. Fixed as preferred: one bucket map a group, measured on a fresh and a spread LRU. (2) The crash-model test never grouped. Fixed: the old file lies under the first extent only, transactions are committed as they apply, and only 0, 1 or 3 extents pointed are accepted. Taken: suggestion 9 (the reuse cycle on 512 KiB buckets), 10 (a grouping count), 3, 4, 5, 6, 7, 8 and part of 2 (a fresh piece is no longer looked up again inside its transaction; an overwriting piece still is, because an earlier piece may have left a slice of its old extent). Not taken now: 1 (pointing each stretch as its bytes complete), the step toward overlapping the engine's work with the device, which changes the order of the write and is the next change rather than this one. The free-figure wording is the review's.
  • r2, the review of d3d67f8: LAND, pending the guest row and the second A/B. Its three notes on the second A/B's rules were settled by an amendment before any boot; its wording for the checks is taken. With both benches down, the orchestrator ruled that metal is no longer a landing condition: the second A/B is owed on metal instead.

🤖 Generated with Claude Code

Japabu and others added 3 commits November 11, 2026 10:06
…ssion's depth

A write-back of DATA sent each 64 KiB extent to the disk by itself and
waited for its answer before the next: an 8 MiB file was 128 writes, one
on the device at a time, each a round trip from the file server through
the NVMe driver and back, though 127 of them followed the one before on
the device.

Now the engine cuts and seals a reserved write's extents first, sends
each stretch of contiguous sectors as one device write, and makes the
extents' pointers once the bytes are there, a transaction each as before.
The served disk asks every request of a write at once instead of one
after another's answer, so the device holds as many as the session
carries. The 8 MiB file is four device writes of 2 MiB, each sixteen
requests in flight together.

What fsync promises is unchanged: a pointer still exists only after its
bytes completed, and the journal's flush before an entry still orders
them before it. A write the device refuses leaves every extent as it was
and stops the mount, as before; a stopped mount now refuses a reserved
write before sending its bytes.

Tests: one device write per stretch and no pointer at refused bytes; a
reserved write of three extents cut at every request under torn and
reordered writes; a write's requests all on the wire before the first
answer, and each asked again across a session end.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The record of the gap #1216 works on: both benches' figures against
Linux's in one session, what the device was asked as counted on the host
model, what #1216 changes and what no bench has yet measured.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tion, and a truncation removes as many

Every 64 KiB extent of a write-back was a transaction of its own, and so
was every extent a truncation removed: an 8 MiB file written over the
last one was 256 transactions, each journaling the inode, the bucket and
the accounting again beside its own extent and backpointer. On the AMD
bench those transactions, with the checksums and buffers, were 6.1 ms of
the round's 22.8 ms fsync, and their 188 KiB of journal made the commit
three entries between five Flushes, 5.1 ms more; on the T14 in its
drive's slow mode the Flushes alone were 12 ms and up.

Now a reserved write points up to eight fresh extents of one bucket in a
transaction, and a truncation removes up to eight whole extents of one
bucket in one. The inode, the bucket and the accounting are journaled
once for the eight. An extent that replaces old ones, or straddles the
truncation's end, still goes alone. Counted on the host model, the row's
round is 32 transactions and one journal entry of 14 blocks between two
Flushes.

Eight, by the transaction's own bounds: the largest such transaction is
263 u64s of journal against the bound of 512 and 8 nodes against the
footprint of 28, on an encrypted volume, whose extent values are the
larger; each extent more is an extent and a backpointer, in at most one
node more of each tree. With 64 a transaction the engine's own assertion
stops the new test at 746 u64s.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The A/B of #1216's code against main on both benches is judged: four
boots an arm a bench, Linux in the same session. The record's "not yet
established" named what the bench would read with the transactions
grouped; it now carries those figures, where a round's time goes with
the change, and what is still not established, the T14's fast mode with
one boot an arm. The record stays open: neither bench reaches Linux's
durable row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Review: #1216 at e64953f (code cc718f6, base ad5db5a)

Architecture

The responsibilities are split correctly. The engine owns the ordering: a reserved write's bytes complete, then its pointer transactions apply, then the journal's preflush puts them on the medium in that order. The engine also owns the grouping. The served disk owns request concurrency. It hands anything past its depth to the session's client, which already holds back requests that the ring or arena cannot take. Nothing changes in the format, the ABI or BlockDev. The storage manual states no rule that this change touches, so the manual needs no edit. The reserved path no longer goes through write_run's Option<(id, sector)> branch, which makes the code simpler. Net size: production +161 lines (data.rs +234/−93, disk.rs +29/−9, mod.rs +9, all test-only), tests +194, record +97.

Crash consistency holds, by inspection.

  • Every stretch is awaited, through reissue, before the first transact. A pointer therefore exists only in a transaction applied after all of the write's bytes completed.
  • write_entries flushes before and after each entry. No entry that carries a pointer can reach the medium before those bytes, whatever the crash point.
  • A group is one transaction, and apply appends it whole to a single entry. After a crash, the applied groups therefore form a prefix.
  • write_shared returns only after every request has been answered. The flush that follows covers all of them, and a reissue never races a request still in flight.
  • Grouped truncation stays under the logged truncate, which resume_truncates finishes.
  • The new check_writable, before any byte is sent, is correct.

The bound of 8 is safe, but the stated reason is wrong (finding 1). The journal measures 263 of 512 u64s. For nodes, my worst-case estimate with a populated LRU is about 20 of the 28 in FOOTPRINT.

A leak is not plausible from the changed paths.

  • The allocator is rebuilt at every mount from the device accounting (FREE + NEED_DISCARD) and the trees (allocator.rs:385-400, metadata_budget). Pins, holes and taken cannot carry a loss across a reboot. Only on-disk state can.
  • In a pointing group, each piece's bucket() reads the bucket through the transaction (get_in), so the alloc key, accounting, LRU and take_free / need_discard changes build on each other and add up correctly. bucket_states is replayed in order, so the last state wins.
  • Grouped truncation uses a single bucket map for the group.
  • A USER bucket with no extents would be an fsck finding. Live data left behind would be a file. Neither comes from a difference in how keys are grouped.
  • The figure on the mount line also cannot distinguish an extra bucket from an extra btree node (see the description section below).

Controls

I checked each one in its log. Each turned red as claimed:

  • pointers before bytes: the two refused-write tests;
  • a write per extent: the stretch test;
  • requests one at a time: both served-disk tests;
  • 64 extents a transaction: apply's assertion at 746 u64s.

The first three were run against 5a90355. Their tests survive unchanged at cc718f6, and the bytes-then-pointers structure is the same there, so reusing them is sound. Two limits:

  • The crash-model test stayed green under "pointers before bytes" (finding 2).
  • No control exercises the node footprint.

Required

  1. lib/bcachefs/src/engine/mount/data.rs:41-44: the derivation of POINTED is false for the write side.

    • The comment says each piece past the first adds only an extent and a backpointer. It does not. In a pointing group, each piece calls point_extent → replace_data_keys. That builds its own bucket map and calls data_buckets → bucket_changed once per piece (data.rs:343-347, 733-747, 778-792).
    • Each of those calls moves the bucket's fragmentation-LRU entry from the previous fill level to the next (reclaim.rs:852-860, bucket.rs:86-93). So every additional piece also journals an LRU key. That key's position is set by the fill fraction, which spreads it across the whole LRU key space, so it can fall in a different LRU leaf each time.
    • The measurements agree. The 64-extent control capped at 32 a bucket: (746 − 263) / 24 ≈ 20 u64s per extra piece. That fits an extent, a backpointer and an LRU key, not an extent and a backpointer alone.
    • The 8-node figure was measured on a fresh volume whose LRU is one leaf.
    • Consequence: anyone who raises the bound using this comment's arithmetic can exceed FOOTPRINT on a fragmented volume. finish's assertion would then kill the file server.
    • Preferred fix, which is also simpler: use one bucket map per group, as trim_extents already does (data.rs:873-889). For fresh pieces old is empty and all share one bucket and one generation. The group can build its keys and call replace_data_keys once. The comment then becomes true, and the transient LRU whiteouts go away.
    • Alternative fix: correct the comment and the margin claim so they include the LRU key and its leaf.
  2. Durability evidence for grouping: lib/bcachefs/src/engine/tests/data.rs:820-874.

    • The PR's Durability section cites this crash-model test as evidence for grouping, but the test never groups. old covers all three extents (:827), so every piece replaces an extent and fresh is false (data.rs:315). Each piece is therefore its own transaction (data.rs:336).
    • The test also stayed green under the pointers-before-bytes control (5 passed; the 2 reds are the refused-write tests). That is expected: with one commit at the end, nothing is journaled between pointing and the bytes.
    • Result: no test cuts a grouped transaction. No test fscks, at every crash point, a group pointing into a reused (need_discard) bucket, which is the path the benches run.
    • Cheapest sufficient fix: let old cover only the first extent. The test then cuts a lone overwrite followed by one group at every request. Accept only the prefixes the groups allow: 0, 1 or 3 extents pointed, never 2.
    • Otherwise, the claim must be narrowed to: grouping is atomic by inspection, plus fsck after the non-crash bounds test.

Description and record (orchestrator's)

  • The free-space sentence overstates the measurement. "r4-P left one bucket more in use than it found" is not what the figure shows.
    • The mount line prints space().1, which is min(free bytes, logical room). On the T14 the logical term is the one printed: the figure is a half-bucket multiple (345917.5 buckets).
    • In 256 KiB node units, logical room is 2·(empty + btree buckets) − (2·metadata nodes + 2) − transient (data.rs:461-472, allocator.rs:153-174).
    • So one more non-root btree node lowers the figure by exactly 512 KiB, the same as one more occupied bucket.
    • The figure is known to move with metadata: R2-1-write's 256 KiB file lowered it by 2 MiB, more than that file's buckets could account for.
    • Suggested wording: "the free figure fell by 512 KiB: one more occupied bucket or one more btree node."
  • No silent debt. If r5/r6 leave the drop unexplained, it belongs in issues/, either in the sequential-write record or in its own record, with what would settle it.
  • AMD retries (no change needed). AMD r3-M and r3-P needed retries with no verdict. The queue log shows both were bench ssh failures before anything was flashed, unrelated to ToyOS. "Every boot's verdict passed" stands.

What the T14's r5/r6 boots must show (LAND pending metal once 1 and 2 are resolved)

  • Fast mode: at least two fast boots an arm, counting r4-M and r1-P. A1 holds in fast mode, and A2 holds on every boot (verdict passed, row judge passing, one mount, no format, no refusal lines).
  • Free figure: record every mount's bytes free.
    • r5-M reads what C2 (P with stage timers) left.
    • A boot's effect is read only by the mount that follows it. Do not use the request's early stop after r5-P. Run through r6-M, and add one boot after r6-P (any image), or record the next T14 boot's mount line.
    • Reading the results:
      • Back at 181360394240 at any mount: the drop was transient metadata, not a leak. A bucket holding no extents is never drained again.
      • Steady at 181359869952 across every transition: there is no recurring loss. Record the one-time drop as described above.
      • Lowered again by any P or C2 boot: treat it as a leak and SEND BACK. The first step is suggestion 9.

Suggestions (optional)

  1. data.rs:285-332 seals the whole write into buf before sending any of it. The file server already holds the pages, and Served::write copies them again into blocks the session keeps until a flush. Pointing each stretch's groups as soon as its bytes complete would bound the buffer to one stretch. It is also the natural step toward overlapping the 4 ms of engine work the timings split out.
  2. data.rs:344-345: inside the transaction, span walks the extents again for every piece just to assert what the data slot already guarantees. Carrying old in Piece removes one lookup per piece from that 4 ms.
  3. data.rs:288, 301-306: ino belongs to the reservation and never changes. Read it, and check regular_mode, once before the loop instead of using the 0 sentinel and pieces.is_empty().
  4. data.rs:318-331: pieces.chunk_by(..) expresses the stretches directly.
  5. POINTED also bounds removal (extents_past), but its name and doc mention only pointing.
  6. system/diskserver/src/disk.rs:180-181, "the first refusal": in this module a refusal means Outcome::Refused, a session end that is asked again and never surfaces here. Say "the first failure".
  7. tests/data.rs:893: drop the eprintln!. The assertions already hold the margins.
  8. data.rs:4-6: the new module-header paragraph describes one function's steps. It belongs in write_reserved's doc.
  9. Add a host test of the reuse cycle on a volume with 512 KiB buckets: a few rounds of create-truncate, eight 1 MiB writes and fsync, remounting between rounds so need_discard buckets are refilled in groups. Assert that the empty-bucket count returns after the unlink and that fsck is clean. No host test currently refills a reused bucket in a group. This is the cheapest check of the path the benches run, and it becomes required if r5/r6 show a further drop.
  10. No test fails if grouping stops (for example, if fresh were always false); only metal would show it. Asserting that the bounds test's write took at most ⌈extents/8⌉ pointer transactions would hold grouping cheaply.

SEND BACK. Findings 1 and 2 must be resolved first. After that, LAND pending the r5/r6 boots under the criteria above.

Japabu and others added 2 commits November 11, 2026 19:42
Review of #1216 found the derivation of POINTED false for the write
side: each fresh piece of a pointing group went through point_extent and
replace_data_keys with a bucket map of its own, so the group's bucket
changed once a piece, and its fragmentation-LRU entry moved through every
fill on the way. Each extra piece journaled an LRU key as well as its
extent and backpointer, and on a volume whose LRU spans many leaves the
group could touch a leaf a piece.

A group of fresh pieces now builds its extent keys and calls
replace_data_keys once, as trim_extents already does for removal: one
bucket map, one alloc change, one LRU move from the fill before to the
fill after. An overwriting piece is still pointed alone, its old extents
read again inside its transaction, because an earlier piece may have left
a slice of one. A fresh piece needs no such read: an old extent reaching
its blocks would have made it not fresh. POINTED's comment now says what
it bounds on both sides.

Measured on the host (largest transaction, journal u64s and nodes; before
the fix in parentheses):
- streaming 5 MiB over 2 MiB buckets: plain 212 (247), encrypted 228
  (263), 7 nodes either way; removal 202 and 8.
- a full group moving its bucket's LRU entry from 1/16 to 9/16 full, on
  1 MiB buckets of 16 KiB nodes: 203 (238) u64s; 6 nodes with the LRU in
  one leaf, 8 (11) with it spread over many leaves by 1500 buckets left at
  every fill.
The margins the tests hold, a quarter of K_TX and half of FOOTPRINT, now
include the LRU leaves on both volumes.

Tests:
- The crash-model test now groups: the old file lies under the first of
  three extents only, so the first is pointed alone and the other two
  together, and each transaction is committed as it applies (a test-only
  switch), so a crash can fall between any two. Every crash state holds
  none, the first or all three pointed, never two, and checks clean; the
  test also asserts that the crashes cut between the lone extent and the
  group.
- The bounds test asserts the write took one pointer transaction for each
  POINTED extents (one more on the encrypted volume, its version mark),
  and a new test measures a group across LRU leaves.
- A new test runs the bench's sequential round on 512 KiB buckets six
  times, remounting between rounds so emptied buckets are refilled in
  groups; fsck is clean after each round, and once the file is gone the
  empty-bucket count is what it was before the first round.

Negative controls, each applied, run and reverted:
- grouping off (no piece fresh): the crash test sees two of three pointed,
  the bounds tests count 80 and 8 transactions; red.
- a refilled bucket drained one sector short: the reuse test's fsck finds
  alloc keys that differ from what the extents derive; red.
- a bucket change a piece (the code before this commit) stays green: its
  footprint is within the margins too, by the figures above.

Also from the review: write_reserved reads its inode once before cutting,
finds stretches and groups with chunk_by, carries its own doc instead of
the module header, and the bounds test prints nothing; the served disk's
write says "failure" where it meant one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1216's code changed in review, so its bench figures are of cc718f6
and the reviewed head is still to be measured. The T14's DATA free
figure fell by 512 KiB after one boot of that code and after no other:
one more occupied bucket or one more btree node, with what the next
boots' mount lines would show for each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Review r2: #1216 at d3d67f8 (code da1c40c, base ad5db5a)

This review covers r1's two required findings, the code changed since e64953f, suggestions 9 and 10, and the second A/B's rules. I built and ran nothing. The evidence was read at evidence ef5b289.

Architecture

The architecture is unchanged and still right. The engine owns the ordering and the grouping. A group of fresh extents now goes through one bucket map, the shape trim_extents already used, so pointing and removal share one rule and POINTED's comment describes both. Production grows by about a dozen lines net: data.rs +56/−43, most of it the new_extent split and the doc moved out of the module header, and drop(slot) in mod.rs. Tests grow by 126 lines and the record by 10.

Finding 1: resolved

  • The grouping (data.rs:331-355). chunk_by joins neighbouring pieces only when both are fresh and in one bucket; chunks(POINTED) then splits each run.
    • A piece that is not fresh never joins a group, so it is always alone and takes the [piece] if !piece.fresh arm.
    • Every other group is fresh and lies in one bucket. It calls replace_data_keys once with old empty. That builds one bucket map in data_counted, so bucket_changed runs once: one alloc key, one LRU delete and insert, and one freespace or need_discard change.
    • account (mod.rs:879) merges into one key per record per transaction, so each extent's REPLICAS and INUM calls add counters, not keys.
    • Reading gen once from group[0]'s bucket is correct: the whole group is in that one bucket, and the read comes before any change in this transaction, as data_counted's before does.
  • Pointing an overwriting piece alone is necessary. span reads through extents_in, which walks the applied tree and not txn.updates. When two overwriting pieces share one old extent, the first leaves that extent's remainder as a new key in its own transaction. The second piece reads the remainder correctly only after that transaction has applied, which pointing it alone guarantees.
  • Grouping fresh pieces is sufficient.
    • The pieces' block ranges are disjoint. Every piece after the first starts at a run boundary or at span's cut, and both are block-aligned.
    • A slice that an earlier piece leaves behind lies inside that piece's old extent. If that extent reached a later piece's blocks, the later piece's span at planning time would have found it, and the piece would not be fresh.
    • The data slot is held from planning to the last pointer.
    • So a fresh piece needs no re-read, and dropping its in-transaction assertion loses nothing.
  • POINTED's derivation (data.rs:37-43) is now true on both sides.
    • Pointing: as above.
    • Removal: extents_past joins only whole extents of one bucket, and a partial head stays alone under one bucket map.
    • Beyond the shared bucket, inode and accounting changes, each extent adds only its extent key and its backpointer.
  • Footprints (r4/logs/verify-measure.log; each variant built fresh with its data.rs hash printed, in the order fix, per-piece, fix):
    • Measured, largest transaction:

      case u64s nodes
      full group, LRU in one leaf 203 6
      full group, LRU spread (LRU root level > 0 is asserted) 203 8
      streaming write 212 (228 encrypted) 7
      its truncation 202 8
    • All of them are within within_margins: at most 384 of 512 u64s and at most 14 of 28 nodes. The largest are 228 u64s (45% of K_TX) and 8 nodes (29% of FOOTPRINT).

    • The per-piece variant measures 238 u64s and 11 nodes on the spread LRU, which confirms r1's diagnosis.

    • The margins do not tell the fix from the per-piece code. The single replace_data_keys call is what holds the fix, and inspection is enough to establish that.

Finding 2: resolved

  • The test (tests/data.rs:822-883).
    • Setup: the old file is 64 KiB, lying under extent 1 only. Piece 0 overwrites it and is pointed alone; pieces 1 and 2 are fresh and in one bucket.

    • The oracle works because the inode size is set in the same transaction as the pointers. Each crash state reads as one of:

      state what the file reads
      0 pointed 64 KiB of old data
      1 pointed 64 KiB of new data
      3 pointed 192 KiB of new data
      a split group 128 KiB, refused
      the group without the lone extent 192 KiB that does not match new, refused
    • seen == {0, 1, 3} asserts that the crash points fall on both sides of the lone extent's commit.

    • It is deterministic: the model's sampling seed is the request number, and random() is seeded per test thread.

    • If a layout change put pieces 1 and 2 in two buckets, the test would go red with "2 of three" rather than pass quietly.

  • The control (r4/controls/controls2-r3.log, grouping-off.diff, hash printed): exit 101, with "2 of three extents pointed at request 16, committed false".
  • The test-only switch does not distort the test (mod.rs:1261-1266, #[cfg(test)]).
    • It runs after finish and after the slot is released. Production keeps its order: drop(slot) sits where _slot was dropped before.
    • It changes no transaction's contents and does not change the bytes-before-pointers order. It only puts an entry boundary, with its flushes, at every transaction boundary.
    • Production can have a boundary in the same places, from another task's commit or a full entry. So the crash states this test checks are a superset of production's for this write.
    • What it gives up: the case of this write's three transactions sharing one entry. That is the journal's own atomicity, and the crash suites still cut it: tests/crash.rs:178 with every from 3 to 40.
    • Side effect: with every transaction committed, making pointers before bytes would now turn this test red too. r1 noted that the old form stayed green. That control was not rerun.

Suggestions 9 and 10: verified

  • 9 (tests/data.rs:989-1029):
    • The test asserts 512 KiB buckets, runs six rounds with a remount between rounds, and runs fsck after each round.
    • reused ≥ 64 shows that emptied buckets were taken again; each group of 8 fills one bucket.
    • The empty-bucket count is restored after unlink and release.
    • The refill-keeps-sectors.diff control goes red at round 2.
  • 10 (:897-925, :930-987):
    • The counts are exact: 10 pointer transactions for 80 extents (11 encrypted), and 1 for the group.
    • With grouping off, they are 80 and 8.

Evidence

  • Host checks. The suites and clippy ran on e64953f plus an uncommitted tree, not on the commit itself.
    • tests/data.rs and mod.rs match the hashes printed in the logs.
    • The committed data.rs (a8a8eda3ba4d) differs from the tested one (2db7861d9a62). I put back the two-line fresh-arm comment from the context of per-piece-bucket-change.diff, and the result hashes to 2db7861d9a62 exactly. The reworded comment is therefore the only difference.
    • The engine suite log shows a fresh Compiling bcachefs, and all four new or changed tests pass.
    • Reusing these results is sound.
  • Excluded logs. The superseded logs with swapped figures are rightly set aside.
  • Guest. Run 43070793437 (nvme_disk_boots, head d3d67f8) is still queued.

Findings

No blocker.

Notes on the second A/B's rules.

  • Provenance. rules-ab2.txt is identical to the evidence copy and was written at 19:26. P' was staged at 19:46. P' is ec98464cd, which is d3d67f8 plus only the row commit.
  1. Fast mode is inconsistent.
    • The rules say "the gate A1' is still judged on this A/B's boots only". They also count r4-M toward establishing fast mode, and fast-mode A1' applies "where established".
    • If this A/B gives two fast P' boots and one fast M boot, r4-M is both in and out of the gate.
    • r4-M would also enter without the 5% session-agreement check that the rules require before pooling the earlier M boots.
    • Settle this before reading the results. For example: r4-M counts toward fast-mode M only if this A/B's M boots pass the agreement check.
  2. Fast mode is weaker than r1's condition.
    • r1's landing condition was A1 in fast mode with at least two fast boots an arm. Because P' is new code, that now needs both T14 P' boots to land in fast mode, which the drive decides.
    • The rules let READY hold without it, and the fast top-up request was pulled.
    • This is acceptable only if it is said plainly. If fast mode is not established, the description says so for the landed code, and the record keeps it open (it already does, at issues/a-sequential-durable-write-on-data-runs-at-about-half-linuxs.md:98).
    • The AMD covers the CPU-bound regime, so I would not require more boots. That decision is the orchestrator's.
  3. Free figure. The ABBA order lets the next mount read each P' boot's effect, and the readings are r1's. Two cases are not assigned:
    • A drop after an M boot is not attributed to P'; it is reported.
    • A drop after a P' boot sends the PR back even if a later mount recovers it, as r1 set it.

Suggestion (optional). In the PR description's Checks section, "At da1c40c's code" should read "on the tree committed as da1c40c, but for one comment", as the evidence says.

What must show before landing

  • Guest: run 43070793437, nvme_disk_boots, passes at d3d67f8.
  • Metal (the second A/B), on both benches, with A1' and A2' as the rules state, read with notes 1 to 3:
    • Every boot's verdict is passed and its row judge passes.
    • Every boot mounts DATA once, formats nothing, and logs no DATA error or refusal line. This includes the M boots that mount what P' left.
    • The free figure does not drop after any P' boot.
    • Fast mode is either established (at least two fast boots an arm, with A1' holding) or reported as not established.
    • A3' is not a gate. Still, before landing, the description's gain figures must be replaced with P' figures or labelled as cc718f6's.

LAND, pending the guest row and the second A/B as above.

Both benches are down with no date for their return, and #1216 lands on
review, host checks and its guest row. The second A/B that was to
measure the reviewed code was staged and never ran, so the record says
plainly that the landed code's speed-up is not measured on metal, and
names what main owes when a bench returns: the A/B of main from before
and after #1216 with DATA carried over and every free figure recorded,
the durable rows against Linux on the same drive, and the reboot round
trip on the T14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Orchestrator verification, landing under the benches-down rule (metal unavailable; review + host + guest, metal owed):

  • Review r2 LAND. Head 7ad9bca differs from guest-tested d3d67f8 only in issues/a-sequential-durable-write-on-data-runs-at-about-half-linuxs.md (verified).
  • Guest run 43070793437 at d3d67f8: success (nvme_disk_boots).
  • Owed on metal: the seqwrite2 before/after A/B on both benches, the durable rows against Linux's persist, and bench_data_survives_reboot on the T14 — recorded; the speed-up is not yet measured on metal at this code.

@Japabu
Japabu marked this pull request as ready for review November 11, 2026 20:13
@Japabu
Japabu added this pull request to the merge queue Nov 11, 2026
@Japabu
Japabu removed this pull request from the merge queue due to a manual request Nov 11, 2026
@Japabu
Japabu added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit d004633 Nov 11, 2026
10 checks passed
@Japabu
Japabu deleted the seqwrite branch November 11, 2026 21:57
Japabu added a commit that referenced this pull request Nov 11, 2026
…ved-write pieces

Main moved 64 commits. Four files conflicted, and one call stopped
compiling:

- kernel/src/object/paged.rs: #1245's read returns once a copy reaches
  the read's end. Here that return sits where no write met the read,
  before it reads again.
- kernel/src/syscall/paged.rs: #1246 looks a holder up under the handle
  table's own lock. The lookup keeps that form and checks the right each
  call needs (READ for a read and a statement, WRITE for a write).
- lib/bcachefs/src/engine/tests/data.rs and the memory filesystem's
  record: both sides added beside each other, and both are kept.
- #1216 split write_run into span, fill, seal and point_extent, and gave
  reserved writes their own planned path, dropping the reserved sector
  write_run took. The placed pool's writes take the sector their pool
  holds again: write_run has the optional placement it had, now over
  main's pieces, and the placed write checks the volume is writable as
  a reserved write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Nov 11, 2026
The merge kept a pool's writes on the path #1216 removed: one extent a
transaction, each waiting for its device write before the next, so 8 MiB
of pool write-back cost 128 serial device writes where #1216 sends 4 at
depth. After #1258 the pool carries most small writes to DATA, so the
merge would have undone #1216's gain for them unseen.

A placed write now plans its pieces onto the pool's lowest unwritten
blocks, taking them with a cursor so no two pieces share one, and shares
the reserved write's second phase: every stretch of contiguous sectors
sent once, then the pointers, up to eight fresh ones of a bucket a
transaction, each block counted written by its sector as its pointer
lands. A pool keeps only its runs not written whole; a reservation goes
once spent. write_run is main's again, so the engine writes reserved
sectors one way, and a placed write checks the volume is writable
before its bytes go, as nothing else does before the first transaction.

A placed write over contiguous pool blocks is tested as a reserved one
is: each stretch sent once, no pointer before, and a refused send
leaving the file and the pool's blocks as they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant