Docker build doesn't work on macOS with Docker Desktop

[huang195]

Title: Docker build doesn't work

Description:

I followed the instructions here: https://github.com/envoyproxy/envoy/blob/main/bazel/README.md#building-envoy-with-the-ci-docker-image, to build Envoy proxy from the head of the main branch. However, I'm encountering the following error message:

haihuang@Mac:~/works/go/src/github.com/envoyproxy/envoy$ ./ci/run_envoy_docker.sh './ci/do_ci.sh release.server_only'
ENVOY_SRCDIR=/source
ENVOY_BUILD_TARGET=//source/exe:envoy-static
ENVOY_BUILD_ARCH=aarch64
Setting test_tmpdir to /build/tmp.
building using 16 CPUs
building for aarch64
clang toolchain configured: arm64-clang
Existing output directory found (/build/envoy/arm64/bin), removing ...
Building with:
  build options: --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=arm64-clang
  release options:  --stripopt=--strip-all -c opt
  binary dir:  /build/envoy/arm64/bin
Starting local Bazel server and connecting to it...
ERROR: /source/WORKSPACE:21:25: fetching python_repository rule //external:python3_12_aarch64-unknown-linux-gnu: java.io.IOException: unlinkat (/build/bazel_root/base/external/python3_12_aarch64-unknown-linux-gnu/share/terminfo/P) (Directory not empty)
ERROR: no such package '@@python3_12_aarch64-unknown-linux-gnu//': unlinkat (/build/bazel_root/base/external/python3_12_aarch64-unknown-linux-gnu/share/terminfo/P) (Directory not empty)
ERROR: /build/bazel_root/base/external/rules_pkg/pkg/private/tar/BUILD:47:10: @@rules_pkg//pkg/private/tar:build_tar depends on @@python3_12_aarch64-unknown-linux-gnu//:python_runtimes in repository @@python3_12_aarch64-unknown-linux-gnu which failed to fetch. no such package '@@python3_12_aarch64-unknown-linux-gnu//': unlinkat (/build/bazel_root/base/external/python3_12_aarch64-unknown-linux-gnu/share/terminfo/P) (Directory not empty)
ERROR: Analysis of target '//distribution/binary:release' failed; build aborted: Analysis failed
INFO: Elapsed time: 11.719s, Critical Path: 0.71s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED: 
    Fetching repository @@cmake-3.23.2-linux-aarch64; starting

I tried to use the latest tagged version from the repo, but I'm getting similar error messages. Shouldn't the Docker build image have all the necessary tools and dependencies pre-installed or am I missing something obvious here? Thank you!

[huang195]

Also the place of failure sometimes changes, using the same command , e.g.,

$ ./ci/run_envoy_docker.sh './ci/do_ci.sh release.server_only'
ENVOY_SRCDIR=/source
ENVOY_BUILD_TARGET=//source/exe:envoy-static
ENVOY_BUILD_ARCH=aarch64
Setting test_tmpdir to /build/tmp.
building using 16 CPUs
building for aarch64
clang toolchain configured: arm64-clang
Existing output directory found (/build/envoy/arm64/bin), removing ...
Building with:
  build options: --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=arm64-clang
  release options:  --stripopt=--strip-all -c opt
  binary dir:  /build/envoy/arm64/bin
Starting local Bazel server and connecting to it...
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:87:18: target 'grpc_cpp_plugin' is both a rule and a file; please choose another name for the rule
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:93:18: target 'grpc_csharp_plugin' is both a rule and a file; please choose another name for the rule
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:99:18: target 'grpc_node_plugin' is both a rule and a file; please choose another name for the rule
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:105:18: target 'grpc_objective_c_plugin' is both a rule and a file; please choose another name for the rule
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:111:18: target 'grpc_php_plugin' is both a rule and a file; please choose another name for the rule
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:117:18: target 'grpc_python_plugin' is both a rule and a file; please choose another name for the rule
WARNING: /build/bazel_root/base/external/com_github_grpc_grpc/src/compiler/BUILD:123:18: target 'grpc_ruby_plugin' is both a rule and a file; please choose another name for the rule
ERROR: no such package '@@com_github_axboe_liburing//': error globbing [**] op=FILES: /build/bazel_root/base/external/com_github_axboe_liburing/man/IO_URING_VERSION_MINOR.3 (Too many levels of symbolic links)
ERROR: /source/bazel/foreign_cc/BUILD:28:15: no such package '@@com_github_axboe_liburing//': error globbing [**] op=FILES: /build/bazel_root/base/external/com_github_axboe_liburing/man/IO_URING_VERSION_MINOR.3 (Too many levels of symbolic links) and referenced by '//bazel/foreign_cc:liburing'
ERROR: Analysis of target '//distribution/binary:release' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.892s, Critical Path: 1.13s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED: 
    Fetching repository @@cmake-3.23.2-linux-aarch64; starting
    Fetching repository @@python3_12_aarch64-unknown-linux-gnu; starting
    Fetching ...on3_12_aarch64-unknown-linux-gnu; Extracting cpython-3.12.3+20240415-aarch64-unknown-linux-gnu-install_only.tar.gz
    Fetching repository @@boringssl; starting
    Fetching repository @@com_github_fdio_vpp_vcl; starting
    Fetching repository @@com_github_google_quiche; starting
    Fetching /build/bazel_root/base/external/com_github_google_quiche; Extracting 377a484ce2b0d00bd1dd5a6cc0fa7a7032603bba.tar.gz
    Fetching repository @@base_pip3_ply; starting ... (28 fetches)

[phlax]

these errors look ~unrelated - the first seems to be a network transient, and the second issue seems to be an issue related to fs linking

not 100% in my diagnosis - but if im correct then rerunning got you past the first - the second might be a bazel cache issue, but might be a host os issue - are you using docker desktop on macos?

fwiw - this code path is tested in ci for arm and x86

[huang195]

Hi @phlax thanks for your response. I tried to run the command multiple times, and it just fails at some point in time. I'm running this using docker desktop on macos with a m4 chip. Is this a supported setup to compile envoy proxy?

[phlax]

theoretically yes - altho its not a tested setup - we kinda support the docker bit

afaiaa there are others building with this setup - so i believe its possible but also would be good to know when/why it doesnt work - perhaps they are not building the release tarball etc

[huang195]

Usually when a build is done by a Docker build image, my understanding is that the host environment does not matter, however, in this case, it made some differences. At first, I used Podman, but the build script seems to only want Docker, and refused to use Podman. Then after installing Docker Desktop, I ran into the above problems.

I even created a github action so the build can run natively on Ubuntu using Bazel without Docker, and it is taking forever -- 3 hours and 15 mins and still running. I'm scratching my head about what I'm doing wrong that I can't build an Envoy binary locally. Any suggestions on what I should check? Is Macos w/ M4 chip known to have problems? I feel like this is a well-traveled path that I shouldn't run into this many problems.

[phlax]

taking a long time is unfortunately expected - we use a lot of caching and build workers to make ci fast - but if you are doing a fresh build - ie no existing cache - it really does take a lot of resources

... the host environment does not matter ...

to some extent - that at least is the hope/expectation - but it can in some cases i believe - im not a macos/docker desktop user so cant test myself

... a well-traveled path ...

there are certainly a lot of macos devs - and my understanding is that many/most use the container build - so im wondering if its an issue with this target specifically - which is less used in dev

does it always fail fast with the too many links issue? - im assuming that the other one that looked more transient has gone away

[huang195]

so im wondering if its an issue with this target specifically - which is less used in dev

That's interesting. I could build for x86_64 instead of arm64 and see if it makes a difference

does it always fail fast with the too many links issue? - im assuming that the other one that looked more transient has gone away

Yes, it always fail with the same error type, but sometimes with a different library.

[phlax]

so i think the too many links does point to a macos issue - iiuc its uses a linux vm layer - but it also mounts the host fs via virtio or similar so there are a few layers of possible issues before you even get to the macos fs

you might want to try disabling the --experimental_repository_cache_hardlinks one way or other - that might make it work

[huang195]

@phlax thanks for the tips. I will give them a try, and if I succeed, I will report back.

[huang195]

Re-reading the documentation, and it explicitly says:

The ./ci/run_envoy_docker.sh script can be used to set up a Docker container on Linux and Windows to build an Envoy static binary and run tests.

It mentions Linux and Windows, but doesn't mention MacOS at all. So, it looks like my assumption that this is a well-traveled path is not true at all.