Skip to content

tmpdir: each user works in a private directory of a shared base - #1178

Merged
Japabu merged 1 commit into
mainfrom
fix/tmpdir-per-user-base
Nov 4, 2026
Merged

Japabu merged 1 commit into
mainfrom
fix/tmpdir-per-user-base

Conversation

@Japabu

@Japabu Japabu commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

toyos-tmpdir kept its GLOBAL lock and its roots directly in the base: $TMPDIR, or /tmp for TempDir::short. Its module doc assumed every process sharing a base was one user. The guest runner host runs jobs as toyos-runner, and its first job (cargo run -- --ci rows --list, run 41918010376) exited 101 with open /tmp/toyos-tmp.lock: Permission denied. Another user's earlier runs had made that file, mode 644. The same assumption is behind issues/a-dead-scratch-root-another-user-owns-panics-every-sweep-of-its-base.md: one user's killed run left a root in /tmp that every other user's sweep failed to move, which panicked their first TempDir.

Behavior

Every use of a base now goes through toyos_tmpdir::private(base), which returns <base>/toyos-<euid>/. It creates that directory with mode 0700. It panics, naming the path and the reason, unless lstat shows a directory (not a symlink) owned by the effective user with permission bits exactly 0700. The lock, the roots and a sweep's stuck-* leftovers all live inside it.

  • A sweep lists only its own user's directory, so it never meets another user's root. No user can disturb another's roots, and the closed issue's case cannot arise.
  • In a sticky base like /tmp, no other user can rename or replace the directory once it exists. If another user creates toyos-<your uid> first, it is refused with a clear message rather than used.
  • PRIVATE_PREFIX (toyos-) does not start with ROOT_PREFIX (toyos-tmp-). A sweep by an older binary on the same host therefore never reads a private directory as an ownerless root.
  • gone_roots(base) and TempDir::adopt(base, pid) still take the shared base and resolve its private directory themselves.
  • build/ci.rs's leftover check now expects the lock inside the job's $TMPDIR private directory. It names anything left in that directory or beside it.
  • The stale /tmp lock and roots from before this change are left alone and no longer read. On the runner host, jan's /tmp/toyos-tmp.lock stops mattering; nobody has to delete it.

macOS

macOS's $TMPDIR (/var/folders/…/T/) is already per-user and 0700 (checked on the dev host), so the extra level is redundant there. /tmp, the short base, is shared on macOS just as on Linux, so the change is not a no-op on either. I kept one rule for every base and host instead of detecting an already-private base. Detection would add a second layout and buy only one path component.

sun_path

A short path grows by at most toyos-4294967295/, 17 bytes. The worst case with a 32-bit uid is /private/tmp/toyos-4294967295/toyos-tmp-99999-99/boot-999/uart.sock, 67 bytes on Darwin (limit 104). On Linux, with a 7-digit pid, it is /tmp/toyos-4294967295/toyos-tmp-4194304-99/boot-999/uart.sock, 61 bytes (limit 108).

New dependency

libc = "0.2" (already in the workspace lockfile, 0.2.190), for geteuid. std has no effective-uid API. Every consumer of the crate is a dev-dependency on a Unix host, and build/ci.rs already uses std::os::unix unconditionally.

Tests

Host unit tests in lib/tmpdir/src/lib.rs:

  • a_private_directory_is_made_this_users_alone: the directory is created, is a directory, is owned by the euid, has mode 700, and a second use accepts it.
  • a_private_directory_another_user_owns_is_refused: covers the two-user case.
  • a_private_directory_with_another_mode_is_refused: mode 755 is refused.
  • a_symlink_in_place_of_a_private_directory_is_refused: a symlink to a valid 0700 directory is refused.

In lib/tmpdir/tests/reclaim.rs, a_root_without_an_owner_goes_and_nothing_else_is_touched now also places an ownerless root in the shared base and one in another uid's private directory. A real holder process's sweep must leave both alone.

Two users on one host. The host can't run as two users, so private_for(base, uid) takes the user as a parameter. The test calls it as uid+1 on a directory this process owns with mode 0700. Ownership is the only thing that differs, which is exactly what a second user sees when they meet the first user's directory. The OS's own permission checks (sticky /tmp, a 0700 directory) are not re-tested here. The sweep's isolation from other users follows from the sweep listing only the caller's private directory, and the extended reclaim test checks that.

Checks (revision c4e22de, macOS aarch64 dev host)

  • cargo test -p toyos-tmpdir: exit 0 (log).
  • cargo run -- --ci host: exit 1, 107 of 108 steps green (log). "nothing left in $TMPDIR or /tmp" is green under the new layout. The one red step is the build system's cargo test --lib. It failed on the known load flake a_narrow_guest_does_not_overtake_a_wide_one_waiting (issues/a-narrow-guest-test-reads-any-queue-holder-as-the-wide-waiter.md; load average about 9). That test's leftover guest directory then made the last TempDir drop in the process, which happened to be in the_primary_s_fork_checkout_is_moved_to_its_pin, fail to remove its root. Both tests passed alone three times each at the same revision (logs). Neither touches the changed logic beyond taking a TempDir.
  • Negative controls (patches and logs). Each patch was applied, built (exit 0), tested (exit 101), and reversed:
    • removing the owner check reds the other-user test;
    • removing the mode check reds the mode test;
    • following symlinks reds the symlink test;
    • making roots in the shared base reds five reclaim tests, including "a sweep took a root outside its user's private directory".
  • Not run: the guest runner itself. The owner verifies on the runner after landing.

Records

  • Deletes issues/a-dead-scratch-root-another-user-owns-panics-every-sweep-of-its-base.md. Its durable rule is now the module header's "a base may be shared by every user; nothing in it is".
  • Updates the quote and wording in issues/the-tmp-leftover-check-can-name-another-worktrees-killed-run.md. Cross-worktree blame within one user remains open, as does host-ci-blames-its-run-for-another-processs-dead-tmp-root.

Unresolved / noticed, not changed

  • build/qemu.rs still uses fixed shared paths for interactive cargo run: /tmp/toyos-qmp.sock, /tmp/toyos-uart.sock, /tmp/toyos-uart-early.log, /tmp/toyos-qemu-debug.log and /tmp/toyos-qemu-stderr.log. A second user running cargo run on the runner host would collide with the first. Harness rows use TempDir::short and are not affected.
  • TempDir::adopt has no caller outside its own test.
  • A base that is world-writable but not sticky would let another user rename the private directory away after the check. /tmp and macOS's $TMPDIR are not like that; I don't check the base itself.

🤖 Generated with Claude Code

Orchestrator note (review S4). Dead roots made before this change, directly in $TMPDIR as well as in /tmp, are no longer reclaimed by any sweep. On a dev host they may hold boot images; remove toyos-tmp-* there once by hand. toyos-build depends on the crate as a normal dependency, not only as a dev-dependency; the build is Unix-only, so geteuid is available everywhere it runs.

`toyos-tmpdir` kept its GLOBAL lock and its roots directly in the base,
`$TMPDIR` or `/tmp`, assuming every process sharing a base was one user.
On the guest runner host, which runs jobs as `toyos-runner`, the first job
(`cargo run -- --ci rows --list`) exited 101 opening `/tmp/toyos-tmp.lock`,
which another user's earlier runs had made with mode 644. The same
assumption let one user's killed run leave a root in `/tmp` that every
other user's sweep then failed to move, panicking their first `TempDir`.

Every use of a base now goes through `private(base)`:
`<base>/toyos-<euid>/`, made with mode 0700, and refused with a panic
naming the path and the reason unless `lstat` shows a directory, not a
symlink, owned by the effective user with permission bits exactly 0700.
The lock, the roots and a sweep's stuck leftovers live inside it, so a
sweep only ever lists this user's roots. In a sticky base no other user
can rename or replace the directory once made; one they made first under
this user's name is refused rather than used.

The rule is the same on every host. On macOS `$TMPDIR` is already a
per-user 0700 directory, so the extra level is redundant there but
harmless; `/tmp`, the short base, is shared on macOS as on Linux, so the
change is not a no-op on either. A short socket path grows by
`toyos-<uid>/`, at most 17 bytes, and stays far inside Darwin's 104-byte
`sun_path`.

`build/ci.rs`'s leftover check reads the private directory of the job's
`$TMPDIR` for the lock and names what is left in it or beside it.

Closes `issues/a-dead-scratch-root-another-user-owns-panics-every-sweep-of-its-base.md`:
another user's roots are never in the directory a sweep lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #1178 at c4e22de (round 1)

Architecture

The fix is in the right place. toyos-tmpdir owns the scratch layout, and the defect was a layout assumption: one user per base. Giving each effective user a private <base>/toyos-<euid>/ directory, checked with lstat for owner, mode and type, is the standard answer for a predictable name in a sticky directory. tmux's /tmp/tmux-<uid> uses the same check. It removes the shared-lock permission failure and the cross-user sweep panic together. It does not add a second mechanism for either.

One rule for every base and host is simpler than detecting a base that is already private. Production growth is small: private_for is about 40 lines, plus about 15 in left_behind. Tests grow by about 75 lines. The deleted issue is 36 lines. The only new dependency is libc, which toyos-build already has and the lockfile already holds. It is used for geteuid, which std does not expose. The unsafe blocks have correct SAFETY notes.

What I verified

  • Creation and verification in a sticky /tmp. DirBuilder::mode(0o700).create treats AlreadyExists as "inspect it", and symlink_metadata does not follow a link.
    • A hostile pre-created entry is refused loudly. That covers a directory another user owns, a symlink, a plain file and a 0755 directory. The panic names the path and tells the user to have the owner or root remove it.
    • Once the directory exists and passes, the sticky bit stops other non-root users from renaming or unlinking it. Later uses by path therefore cannot be redirected. sweep, Root::remove and stuck reuse the stored root.tmp without checking again, and that is safe for the same reason.
    • Nobody can hard-link a directory into place. Renaming a victim-owned directory into place needs write permission on that directory, which 0700 denies.
    • A umask that strips owner bits produces a mode other than 700 and is refused.
    • The one remaining hole is a world-writable base without the sticky bit. That can only be a user-chosen $TMPDIR, and the PR states it.
  • No sweep reaches another user's roots. gone_under lists one directory without recursing, and Root::make, sweep, adopt and gone_roots all receive the private directory. ROOT_PREFIX (toyos-tmp-) is not a prefix of toyos-<digits>, so a sweep by an older binary of the shared base never takes a private directory. a_root_without_an_owner_goes_and_nothing_else_is_touched covers a root in the shared base and one in another uid's directory. Negative control m4 turns it red on the shared-base root, at reclaim.rs:202.
  • Socket paths. The only TempDir::short user is tests/common/qemu.rs:2765, with boot-<n>/{qmp,vnc,uart}.sock. I recounted the worst case at 67 bytes on Darwin, against a 104-byte limit. The macOS $TMPDIR nesting grows by about 20 bytes, but no host code binds a socket under TempDir::new. osbench's Linux socket uses an address, not a $TMPDIR path.
  • build/ci.rs left_behind. It names everything in the job's $TMPDIR except the private directory, and everything in the private directory except GLOBAL. That is the old check moved down one level, with nothing dropped. A stray old-style lock at top level would still be named. owed now looks in the right place. The real run shows the end-to-end path: "nothing left in $TMPDIR or /tmp" passed with owed true, so the children's lock was found in ci-host-0/toyos-UID/. The two ci.rs unit tests cover both levels.
  • Issue records. The deleted issue's case cannot happen any more: another user's roots are never in the listed directory. Within one user, a rename that fails in a 0700 directory of that user's own roots is a broken internal invariant, and a loud panic is correct for it. No other file references the deleted issue. The edited the-tmp-leftover-check-… issue correctly narrows to one user's worktrees, and its quote matches the new header. host-ci-blames-… keeps its historical /tmp/toyos-tmp-16566-0 path as evidence, which is correct.
  • The red host step. The ci-host.log failure has the recorded issue's exact signature: both guests "waiting for the guest CPU queue", narrow acquired first, then the assertion at buildlock.rs:928. The PR changes only how the root directory is located. It makes one mkdir and lstat per process per base and does not touch buildlock's lock ordering, which is what the flake races on.
    • The cascade is downstream of that failure. The panicked test's still-running guest left buildlock-guest-cpus-queue-15 non-empty, so the process's last TempDir drop failed remove_dir on its root. Root::remove behaves the same before and after this PR. A later process's sweep then reclaimed the ownerless root, which is why the leftover step passed.
    • The isolated reruns are at the same revision.
    • I accept that the PR is independent of the red.
  • Negative controls m1–m4. Each targets the defect its test claims to catch: owner, mode, symlink-following, shared base.

Findings

No blockers.

SUGGESTION

  1. lib/tmpdir/src/lib.rs:542-544: "Here the other user is the one this process is not" is hard to parse. Something like "This process claims uid+1, so the directory it just made, mode 0700, differs from a valid one only in its owner." says the same thing directly.
  2. build/ci.rs:892-893: the reflow left a ragged line, and "keeps in this user's private directory there" reads awkwardly. "…but the lock toyos_tmpdir keeps in this user's private directory of it:" with the paragraph rewrapped would read better. lib/tmpdir/src/lib.rs:312 and the edited issue's lines 27-28 are similarly unwrapped.
  3. lib/tmpdir/src/lib.rs:66: the claim that ROOT_PREFIX is not a prefix of PRIVATE_PREFIX could be a const assertion rather than prose. This is optional; the comment is accurate.
  4. The PR description says the stale /tmp lock and roots are left alone. The same holds for dead pre-change roots directly in $TMPDIR, and on a dev host those can hold harness boot images. No sweep will reclaim them now. A one-line note to remove $TMPDIR/toyos-tmp-* and /tmp/toyos-tmp-* once by hand would make that gap explicit. Separately, the description calls every consumer a dev-dependency, but toyos-build depends on the crate normally. The conclusion still holds because the build is Unix-only. Both are for the orchestrator.
  5. The fixed /tmp/toyos-{qmp,uart}.sock, toyos-audio.wav and similar paths in build/qemu.rs, which interactive cargo run uses, collide across users. The PR noticed this but recorded it only in its description. No current CI flow reaches it, but if it is worth keeping, an issues/ record is its durable home.

Verdict

LAND

@Japabu
Japabu marked this pull request as ready for review November 4, 2026 22:14
@Japabu
Japabu added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit e740f01 Nov 4, 2026
10 checks passed
@Japabu
Japabu deleted the fix/tmpdir-per-user-base branch November 4, 2026 22:31
Japabu added a commit that referenced this pull request Nov 4, 2026
Brings #1178 (each user's tmpdir is a private directory of a shared
base). No conflict; Cargo.lock merged by itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant