Move locking from bootstrap.py to rust bootstrap, using fd-lock

[joshtriplett]

Helps with #94829.

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

This is great, thanks! I have a bad feeling CI will fail because of the new dependencies, but r=me otherwise. If it does fail, we should probably have a discussion with Mark / infra team about whether the dependencies are ok.

[jyn514]

Oh, last thing - can you see what the compile times for bootstrap before and after this PR? You can clear only the bootstrap artifacts (and not the stage0 toolchain) with rm -rf build/bootstrap.

[joshtriplett]

On my system, 16.39s before, 16.83s after. I think the new dependencies build almost entirely in parallel with other things.

[Mark-Simulacrum]

I'm sad we need this many dependencies added, but I suppose they're all fast to compile.

I am a little worried that this is moving the locking into rustbuild, which means that (for example) submodule management is no longer covered by it -- probably amongst several other similar things. Are we confident those behave well when racing two different builds?

[jyn514]

submodule management is no longer covered by it -- probably amongst several other similar things. Are we confident those behave well when racing two different builds

Submodule handling is being moved out of python in #97513, happy to wait until that's merged to land this. The only other major things are downloading the bootstrap toolchain and building rustbuild.

Downloading the toolchain is mostly atomic: we download the files to a temporary directory, then atomically rename them to somewhere in build/cache. Extracting is not atomic but the chance you try and extract the same destination file at once seems vanishingly small - IMO the worst scenario in practice for both of these is doing extra work, which seems ~fine given that this is already best effort.

Cargo already has its own locks for building rustbuild, I'm not worried about that.

[bors]

☔ The latest upstream changes (presumably #97513) made this pull request unmergeable. Please resolve the merge conflicts.

[joshtriplett]

Rebased.

[jyn514]

@bors r+ rollup=iffy

[bors]

📌 Commit 5a30316 has been approved by jyn514

[bors]

⌛ Testing commit 5a30316 with merge 0e21a27...

[bors]

☀️ Test successful - checks-actions
Approved by: jyn514
Pushing 0e21a27 to master...

[rust-timer]

Finished benchmarking commit (0e21a27): comparison url.

Instruction count

• Primary benchmarks: 🎉 relevant improvements found
• Secondary benchmarks: 🎉 relevant improvements found

	mean1	max	count2
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	N/A	N/A	0
Improvements 🎉 (primary)	-1.0%	-1.0%	2
Improvements 🎉 (secondary)	-2.1%	-2.4%	6
All 😿🎉 (primary)	-1.0%	-1.0%	2

Max RSS (memory usage)

Results

• Primary benchmarks: no relevant changes found
• Secondary benchmarks: 😿 relevant regression found

	mean1	max	count2
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	3.9%	3.9%	1
Improvements 🎉 (primary)	N/A	N/A	0
Improvements 🎉 (secondary)	N/A	N/A	0
All 😿🎉 (primary)	N/A	N/A	0

Cycles

Results

• Primary benchmarks: 😿 relevant regression found
• Secondary benchmarks: 🎉 relevant improvements found

	mean1	max	count2
Regressions 😿 (primary)	2.4%	2.4%	1
Regressions 😿 (secondary)	N/A	N/A	0
Improvements 🎉 (primary)	N/A	N/A	0
Improvements 🎉 (secondary)	-3.0%	-3.2%	3
All 😿🎉 (primary)	2.4%	2.4%	1

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

Footnotes

1. the arithmetic mean of the percent change ↩ ↩² ↩³

2. number of relevant changes ↩ ↩² ↩³