Hermetic (offline) build of enola with Nix and Bazel

[dotdoom]

We're making a nix-bazel-nix sandwich:

1. Nix runs bazel to fetch all dependencies, then tars them and supply it as derivation to next step
2. Nix builds the app (offline)

According to https://bazel.build/external/faq#how-do-i-prepare-and-run-an-offline-build this should work:

$ bazel clean

$ REGISTRY=$(mktemp -d)
$ git clone https://github.com/bazelbuild/bazel-central-registry.git "${REGISTRY?}"

$ REPO_CACHE=$(mktemp -d)
$ TARGET=//java/dev/enola/cli:enola_deploy.jar
$ bazel --batch fetch --repository_cache="${REPO_CACHE?}" --registry="file://${REGISTRY?}" "${TARGET?}"

< disconnect network >

$ bazel --batch build --nofetch --repository_cache="${REPO_CACHE?}" --registry="file://${REGISTRY?}" "${TARGET?}"

However on Nix this doesn't work at build:

       > OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
       > Computing main repo mapping:
       > ERROR: Error computing the main repository mapping: error during computation of main repo mapping: to fix, run
       >  bazel fetch //...
       > External repository @@bazel_tools not found and fetching repositories is disabled.
       > Computing main repo mapping:

[vorburger]

I've tried without the registry related stuff, first like this (minor difference re. $REPO_CACHE is just due to Fish):

$ set REPO_CACHE $(mktemp -d)

$ bazelisk --batch fetch --repository_cache="$REPO_CACHE" //...

$ bazelisk --batch build --nofetch --repository_cache="$REPO_CACHE" //...

Without any initial clean and had not disconnected network, yet; and that (as above) worked fine.

Then I've proceeded to disable networking on my workstation, to re-try:

Because of https://github.com/vorburger/vorburger.ch-Notes/blob/develop/linux/systemd-networkd.md :

$ networkctl
IDX LINK    TYPE     OPERATIONAL SETUP
  1 lo      loopback carrier     unmanaged
  2 eno1    ether    routable    configured

  $ systemctl stop systemd-networkd
Stopping 'systemd-networkd.service', but its triggering units are still active:
systemd-networkd.socket

$ networkctl
Failed to connect to network service /run/systemd/netif/io.systemd.Network: No such file or directory

$ ping 8.8.8.8
ping: connect: Network is unreachable

$ bazelisk --batch build --nofetch --repository_cache="$REPO_CACHE" //...
(...) WARNING: (...)
INFO: Analyzed 297 targets (1331 packages loaded, 22080 targets configured).
INFO: Found 297 targets...
INFO: Elapsed time: 7.649s, Critical Path: 0.98s
INFO: 1 process: 3891 action cache hit, 1 internal.
INFO: Build completed successfully, 1 total action

This seems to work great!

Next I'll try with an initial clean, and then also using the registry related stuff, to reproduce it more precisely.

[vorburger]

Next I'll try with an initial clean

$ bazelisk clean --expunge
Starting local Bazel server (8.3.1) and connecting to it...
INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.

$ rm -rf $REPO_CACHE
$ set REPO_CACHE $(mktemp -d)

$ time bazelisk --batch fetch --repository_cache="$REPO_CACHE" //...
INFO: Elapsed time: 58.270s, Critical Path: 0.00s
________________________________________________________
Executed in   59.55 secs    fish           external
   usr time  162.33 secs    5.93 millis  162.33 secs
   sys time   22.06 secs    3.23 millis   22.06 secs

$ du -h $REPO_CACHE
2.0G    /tmp/nix-shell.gGCGoQ/tmp.YFQzmQXe5U

$ systemctl stop systemd-networkd

$ time bazelisk --batch build --nofetch --repository_cache="$REPO_CACHE" //...
ERROR: /home/vorburger/git/github.com/enola-dev/enola/java/dev/enola/common/protobuf/test/BUILD:19:13: Building source jar java/dev/enola/common/protobuf/test/libtest-resources-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //java/dev/enola/common/protobuf/test:test-resources) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/java/dev/enola/common/protobuf/test/libtest-resources-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:516: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x2ae7a370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/models/BUILD:3:13: Building source jar models/libmodels-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //models:models) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/models/libmodels-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:516: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0xdb58370)": No such file or directory
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 5.281s, Critical Path: 0.15s
INFO: 5 processes: 5 internal.
ERROR: Build did NOT complete successfully

________________________________________________________
Executed in    6.17 secs    fish           external
   usr time   49.56 secs    7.13 millis   49.55 secs
   sys time    1.87 secs    2.03 millis    1.86 secs

This is a different error than the one you ran into... but in order to be able to fully reproduce each other, we should state which commit we're working? I suspect that you've done something to switch from remotejdk_21 to a local (Nix installed) Java? Let me do that as well, and push a PR with just that - that way we can ensure that we're "on the same page".

[vorburger]

Oh, and before I forget, there's another non-trivial difference! I'm still running with bazelisk, which currently current will lead to bazel 8.3.1 as per the .bazelversion as of 3a066e. This will change to 8.4.1 once I'll merge #1727! (I'm holding that off for now, to not cause too much variability.)

You're probably using a different Bazel version, from Nix? That could in theory also cause us some differences, of course.

BTW I'm also going to try out bazel vendor, just out of curiosity how that's different from bazel fetch.

[vorburger]

With #1727 to use Bazel 8.4.1 instead of 8.3.1 and using vendor instead of fetch:

$ bazelisk clean --expunge
$ rm -rf $REPO_CACHE
$ set REPO_CACHE $(mktemp -d)

$ time bazelisk vendor --vendor_dir="$REPO_CACHE" //...
INFO: Analyzed 297 targets (1331 packages loaded, 22080 targets configured).
INFO: Found 297 targets...
INFO: Elapsed time: 42.769s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
INFO: Vendoring dependencies for targets...
INFO: All external dependencies for the requested targets vendored successfully.

________________________________________________________
Executed in   69.92 secs      fish           external
   usr time  107.44 millis    7.39 millis  100.06 millis
   sys time   77.65 millis    1.84 millis   75.81 millis

$ du -h $REPO_CACHE
1.7G    /tmp/nix-shell.gGCGoQ/tmp.VbsyGojAn0

Note how with vendor is ~300 MB smaller (1.7G instead of 2.0G).

BTW, just FTR: Using //... at the end of venor is very important, because while it's not neccessary to specify a target, as explained in https://bazel.build/external/vendor#vendor-all-dependencies, without that it throws errors due to: "Some repos may fail to fetch if they are not compatible with the current platform or environment."

$ systemctl stop systemd-networkd

$ time bazelisk --batch build --nofetch --vendor_dir="$REPO_CACHE" //...
ERROR: /home/vorburger/git/github.com/enola-dev/enola/tools/bazel/java_plugin/BUILD:3:12: Building source jar tools/bazel/java_plugin/libautovalue-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //tools/bazel/java_plugin:autovalue) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/tools/bazel/java_plugin/libautovalue-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:525: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x1ad0f370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/java/dev/enola/common/protobuf/test/BUILD:19:13: Building source jar java/dev/enola/common/protobuf/test/libtest-resources-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //java/dev/enola/common/protobuf/test:test-resources) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/java/dev/enola/common/protobuf/test/libtest-resources-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:525: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x2ad4d370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/tools/version/BUILD:20:13: Building source jar tools/version/libversion-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //tools/version:version) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/tools/version/libversion-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:525: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x470a370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/models/BUILD:3:13: Building source jar models/libmodels-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //models:models) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/models/libmodels-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:525: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x39569370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/test/BUILD:8:13: Building source jar test/libtest-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //test:test) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/test/libtest-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:525: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x3e8c7370)": No such file or directory
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.556s, Critical Path: 0.18s
INFO: 16 processes: 16 internal.
ERROR: Build did NOT complete successfully

________________________________________________________
Executed in    7.45 secs    fish           external
   usr time   58.89 secs   11.85 millis   58.88 secs
   sys time    2.04 secs    2.22 millis    2.03 secs

Looks similar. So I'll proceed with removing the remotejdk_21, as it definitely seems like Bazel sucks for offline with Toolchains, with both vendor and fetch.

[vorburger]

BTW there's also a ~/.cache/bazel, I'm not 100% sure if it would be safer for full reproducibility to also wipe that; but let me from now on, just to be on the safe side.

[vorburger]

I'll proceed with removing the remotejdk_21, as it definitely seems like Bazel sucks for offline with Toolchains, with both vendor and fetch.

OK let's retry this all over but now using #1747 (as is, so back to [current] Bazel 8.3.1, for now) and see how it looks, also wiping ~/.cache/bazel, but with vendor instead of with fetch again:

$ bazelisk clean --expunge
$ sudo rm -rf ~/.cache/bazel
$ rm -rf $REPO_CACHE

$ set REPO_CACHE $(mktemp -d)
$ time bazelisk vendor --vendor_dir="$REPO_CACHE" //...

$ systemctl stop systemd-networkd
$ time bazelisk --batch build --nofetch --vendor_dir="$REPO_CACHE" //...

ERROR: /home/vorburger/git/github.com/enola-dev/enola/tools/version/BUILD:20:13: Building source jar tools/version/libversion-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //tools/version:version) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/tools/version/libversion-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:516: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x1faee370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/test/BUILD:8:13: Building source jar test/libtest-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //test:test) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/test/libtest-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:516: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x31246370)": No such file or directory
ERROR: /home/vorburger/git/github.com/enola-dev/enola/java/dev/enola/common/protobuf/test/BUILD:19:13: Building source jar java/dev/enola/common/protobuf/test/libtest-resources-src.jar failed: (Exit 1): singlejar_local failed: error executing JavaSourceJar command (from target //java/dev/enola/common/protobuf/test:test-resources) external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local @bazel-out/k8-fastbuild/bin/java/dev/enola/common/protobuf/test/libtest-resources-src.jar-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:516: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local, 0x1c0a370)": No such file or directory
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.381s, Critical Path: 0.25s
INFO: 4 processes: 4 internal.
ERROR: Build did NOT complete successfully

________________________________________________________
Executed in    7.28 secs    fish           external
   usr time   56.52 secs    0.27 millis   56.52 secs
   sys time    1.94 secs    2.09 millis    1.94 secs

If I'm diffing this correctly to the above, it's at least a little better; looks like x3 instead of x5 ERROR ?

But it's STILL choking on something related to remote_java_tools_linux - even though we're not using it anymore?! 😿

But that's on targets which we might not actually need (to build the Nix package hermetically offline).

Let me (also) use only //java/dev/enola/cli:enola_deploy.jar instead of //... from now on:

$ time bazelisk --batch build --nofetch --vendor_dir="$REPO_CACHE" //java/dev/enola/cli:enola_deploy.jar

INFO: Analyzed target //java/dev/enola/cli:enola_deploy.jar (956 packages loaded, 12398 targets configured).
ERROR: /home/vorburger/.cache/bazel/_bazel_vorburger/a3cf8df34f0c7a76e9926daaad9ffbf4/external/protobuf+/upb/mem/BUILD:12:11: Compiling upb/mem/arena.c [for tool] failed: (Exit 1): gcc failed: error executing CppCompile command (from target @@protobuf+//upb/mem:mem) /nix/store/kaj8d1zcn149m40s9h0xi0khakibiphz-gcc-wrapper-14.3.0/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cc1: fatal error: external/protobuf+/upb/mem/arena.c: No such file or directory
compilation terminated.
Target //java/dev/enola/cli:enola_deploy.jar failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/vorburger/git/github.com/enola-dev/enola/java/dev/enola/cli/BUILD:26:13 Building java/dev/enola/cli/liblib-class.jar (38 source files) and running annotation processors (NativeImageConfigGeneratorProcessor) failed: (Exit 1): gcc failed: error executing CppCompile command (from target @@protobuf+//upb/mem:mem) /nix/store/kaj8d1zcn149m40s9h0xi0khakibiphz-gcc-wrapper-14.3.0/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 5.162s, Critical Path: 0.13s
INFO: 334 processes: 334 internal.
ERROR: Build did NOT complete successfully

________________________________________________________
Executed in    5.97 secs    fish           external
   usr time   46.26 secs    7.74 millis   46.25 secs
   sys time    1.96 secs    2.95 millis    1.95 secs

Now this is an entirely different one. Let me retry with fetch instead of with vendor again, to see if that's any different.

[vorburger]

Now this is an entirely different one. Let me retry with fetch instead of with vendor again, to see if that's any different.

$ bazelisk clean --expunge
$ sudo rm -rf ~/.cache/bazel
$ rm -rf $REPO_CACHE

$ set REPO_CACHE $(mktemp -d)
$ time bazelisk --batch fetch --repository_cache="$REPO_CACHE" //java/dev/enola/cli:enola_deploy.jar

$ systemctl stop systemd-networkd
$ time bazelisk --batch build --nofetch --repository_cache="$REPO_CACHE" //java/dev/enola/cli:enola_deploy.jar

ERROR: /home/vorburger/.cache/bazel/_bazel_vorburger/a3cf8df34f0c7a76e9926daaad9ffbf4/external/zlib+/BUILD.bazel:72:8: Executing genrule @@zlib+//:copy_public_headers [for tool] failed: (Exit 1): bash failed: error executing Genrule command (from target @@zlib+//:copy_public_headers) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cp: cannot stat 'external/zlib+/crc32.h': No such file or directory
cp: cannot stat 'external/zlib+/deflate.h': No such file or directory
cp: cannot stat 'external/zlib+/gzguts.h': No such file or directory
cp: cannot stat 'external/zlib+/inffast.h': No such file or directory
cp: cannot stat 'external/zlib+/inffixed.h': No such file or directory
cp: cannot stat 'external/zlib+/inflate.h': No such file or directory
cp: cannot stat 'external/zlib+/inftrees.h': No such file or directory
cp: cannot stat 'external/zlib+/trees.h': No such file or directory
cp: cannot stat 'external/zlib+/zconf.h': No such file or directory
cp: cannot stat 'external/zlib+/zlib.h': No such file or directory
cp: cannot stat 'external/zlib+/zutil.h': No such file or directory
Target //java/dev/enola/cli:enola_deploy.jar failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/vorburger/.cache/bazel/_bazel_vorburger/a3cf8df34f0c7a76e9926daaad9ffbf4/external/protobuf+/upb/reflection/BUILD:26:28 Compiling external/protobuf+/upb/reflection/stage1/google/protobuf/descriptor.upb_minitable.c [for tool] failed: (Exit 1): bash failed: error executing Genrule command (from target @@zlib+//:copy_public_headers) /bin/bash -c ... (remaining 1 argument skipped)

Now this looks more like an "upstream" issue than something we can fix in Enola, to me.

It seems (to me) like the [PITA] Bazel protoc stuff isn't really ready for offline building.

I think the best next step would be to report this to https://github.com/protocolbuffers/protobuf.

[vorburger]

I think the best next step would be to report this to https://github.com/protocolbuffers/protobuf.

I've created protocolbuffers/protobuf#23504.

The immediate next step is to block this on the protobuf bump WIP in #1689, and then retry if 🤞🏽 that helps (but it probably won't).

[vorburger]

@dotdoom I expect that #1759 will at least partially unblock this, or perhaps entirely fix this PITA.

We should retry this, after that's merged, as with it, there is no more protoc from Bazel.

So unless there are other (new) fetch related problems, this will just "go away".