bootstrap doesn't handle LLVM submodule updates

[ZuseZ4]

Summary

If we build rustc with the llvm submodule enabled (as opposed to downloading ci-llvm), then we get an LLVM for a specific Version/Commit. If the LLVM submodule gets updated, then old artifacts remain in the llvm build dir, while we build the new llvm. The chances that this causes linker errors are very high. Last time encountered in: #wg-autodiff > ✔ rustc build failing @ 💬, but a few other people I talked to and I also learned it the same way. The solution is to rm -rf build, since x clean doesn't touch the llvm folder.

Command used

any build command

Expected behaviour

builds

Actual behaviour

This error mentions autodiff, but it can be encountered in different versions without autodiff as well.
error: failed to load our autodiff backend: DlOpen { source: "/home/jonathan/Projects/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libEnzyme-22.so: undefined symbol: _ZN4llvm11APFloatBase18semPPCDoubleDoubleE" }

Bootstrap configuration (bootstrap.toml)

any that does not use download-ci-llvm=true

Operating system

linux (but likely others as well)

HEAD

encountered over multiple llvm versions

Additional context

Imho we should check if the submodule checkout matches build artifacts in the llvm build dir, and error if not.
We shouldn't warn, since I don't think it will ever work. We imho also shouldn't delete the llvm build dir for users, as it's slow to build and maybe he would rather do something else (like change branches) than loosing his llvm artifacts.

Build Log

<log>

[Kobzol]

Hmm, I'm not sure if it really won't ever work, e.g. if you are changing LLVM and you want to try several follow-up submodule versions, the build cache likely can be reused.

@nikic Any opinions on this? Do you ever find it useful for bootstrap to reuse the local LLVM build cache, even if the LLVM submodule commit changes?

[nikic]

I don't really get what this issue is about and have never seen this kind of problem. I don't understand why there would be an issue with stale artifacts as long as the build system works correctly?

Reusing the build directory is indeed helpful when changing the submodule. It's not that big a deal if you're using ccache, but it still avoid regenerating all the TableGen artifacts, etc.

[ZuseZ4]

as long as the build system works correctly

That's kind of the thing I'm questioning here. I am surprised that you haven't seen it, pretty much every collaborator I had ran into it earlier or later. I personally just rm the whole build folder as soon as I see an x.py mentioning an llvm submodule update, since the chance of bootstrap failing after an almost full build are too high, so I'd rather directly start clean. Especially since I also started using ccache even on my local laptop, clean build times of LLVM are quite manageable.

I never encountered it on pure llvm clones, only on my rustc clones which have an llvm submodule.

Maybe we can leave the issue open and I direct more people here as they run into it, so we can try to spot a pattern to figure out when exactly this problem happens?

[bjorn3]

Did bootstrap even ran make/ninja for LLVM at all? If not you probably need to remove the stamp file for the LLVM build. That will cause bootstrap to build again while reusing the build directory. I guess we don't invalidate the stamp file in cases where we should.

[ZuseZ4]

Yes, the issue is that we end up with a second set of LLVM artifacts in the same build folder, so ninja must have run again.
I can't say whether I manually delete the stamp or not since I tend to do that quite often during my normal dev workflow.
But I'd guess that @JonathanBrouwer didn't delete that stamp, can you confirm or correct that?

[JonathanBrouwer]

I only ran ./x clean, and performed no other actions

[ZuseZ4]

I just encountered it once again. I updated main which gave me an llvm submodule update. It compiled LLVM from scratch (~5k objects) and then failed afterwards:

error: failed to load our autodiff backend: DlOpen { source: "libLLVM.so.22.1-rust-1.95.0-nightly: cannot open shared object file: No such file or directory" }

rust/compiler/rustc_codegen_llvm/src/lib.rs

Line 238 in 0c68443

Err(llvm::EnzymeLibraryError::LoadFailed { err }) => {

➜  rust git:(main) ls build/x86_64-unknown-linux-gnu/llvm/lib/libLLVM-22*                                                                                                 
build/x86_64-unknown-linux-gnu/llvm/lib/libLLVM-22-rust-1.95.0-nightly.so  build/x86_64-unknown-linux-gnu/llvm/lib/libLLVM-22-rust-1.96.0-nightly.so

Also

➜  rust git:(main) ldd build/x86_64-unknown-linux-gnu/enzyme/lib/libEnzyme-22.so 
	linux-vdso.so.1 (0x000072ef36694000)
	libLLVM.so.22.1-rust-1.95.0-nightly => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000072ef35a00000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000072ef35d17000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000072ef3663e000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000072ef35600000)
	/lib64/ld-linux-x86-64.so.2 (0x000072ef36696000)
➜  rust git:(main) ls -lh build/x86_64-unknown-linux-gnu/enzyme/lib/libEnzyme-22.so
-rw-r--r-- 2 manuel manuel 8.6M Feb 18 14:00 build/x86_64-unknown-linux-gnu/enzyme/lib/libEnzyme-22.so

Despite the up-to-date timestamp of libEnzyme, I can't see any build logs, so it probably was just touched.
So I guess we just don't trigger an Enzyme re-build when we re-build LLVM, which explains why Nikita never ran into it.