Why does Bazel navigate symbolic links when finding BUILD files?

[crorvick]

Description of the problem / feature request:

Bazel navigates into symbolic links when searching for BUILD files. This can be divided into two scenarios:

1. The symbolic link points to another location within the workspace. In this case, all paths reachable through the link are already reachable without it, thus there is no reason to continue.
2. The symbolic link points to a location outside of the workspace. This seems like a dubious use case. Is this really an intended feature? I would expect everything outside of the workspace should be pulled in using a workspace rule.

Description of the problem / feature request:

Do not traverse symbolic links when looking for BUILD files in a workspace.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ git clone https://github.com/crorvick/bazel-hello.git
Cloning into 'bazel-hello'...
remote: Enumerating objects: 22, done.
remote: Total 22 (delta 0), reused 0 (delta 0), pack-reused 22
Unpacking objects: 100% (22/22), done.
$ mkdir empty
$ cd empty/
$ touch WORKSPACE
$ ln -s ../bazel-hello .
$ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 1f6bc1d1-68d2-4c64-ab9c-20f55cfb9ba7
Loading: 
Loading: 0 packages loaded
ERROR: error loading package 'bazel-hello/src/hello': Unable to load package for '//src/optimize:optimize.bzl': BUILD file not found on package path
INFO: Elapsed time: 1.294s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded)
FAILED: Build did NOT complete successfully (2 packages loaded)

A slightly more complex, but less contrived scenario:

$ git clone https://github.com/crorvick/bazel-hello.git
Cloning into 'bazel-hello'...
remote: Enumerating objects: 22, done.
remote: Total 22 (delta 0), reused 0 (delta 0), pack-reused 22
Unpacking objects: 100% (22/22), done.
$ cd bazel-hello/
$ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: d8aee92d-b92a-4198-8ad1-f2db4cde17b2
Loading: 
Loading: 0 packages loaded
Analyzing: 4 targets (5 packages loaded)
Analyzing: 4 targets (5 packages loaded, 0 targets configured)
INFO: Analysed 4 targets (20 packages loaded, 505 targets configured).
INFO: Found 4 targets...
[0 / 5] no action
INFO: Elapsed time: 3.158s, Critical Path: 0.52s
INFO: 7 processes: 7 linux-sandbox.
INFO: Build completed successfully, 20 total actions
INFO: Build completed successfully, 20 total actions
$ cd ..
$ mv bazel-hello bazel-hello-bak
$ cd bazel-hello-bak
$ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 5c1430a6-4ff2-47c1-bf3c-fb3c5600672d
Loading: 
Loading: 0 packages loaded
ERROR: error loading package 'bazel-bazel-hello/external/bazel_tools/tools/jdk': Unable to load package for '//tools/jdk:default_java_toolchain.bzl': BUILD file not found on package path
INFO: Elapsed time: 1.343s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (8 packages loaded)
FAILED: Build did NOT complete successfully (8 packages loaded)

Bazel is tripping over its bazel-<project> symbolic link in the above example after the parent directory is renamed. A confound in resolving this particular issue is that it adding /bazel-* to the list of ignored file patterns by Git (or whatever VCS is used) is a reasonable thing to do since the bazel-<project> link is not predictable from what is under Git's control.

[laszlocsomor]

/sub

[crorvick]

bump

[laszlocsomor]

This is great question!

I don't know for sure why Bazel does this. My theory is that following these symlinks lets Bazel track changes in them or in their target, and correctly rebuild rules that depend on the link (e.g. a genrule with srcs=["a"] where a is a symlink). I also suspect that this behavior may have historical reasons, like supporting a latest symlink in some tool's package that always points to the newest checked-in version of that tool (while old versions are also kept in the depot).

Whether symlinks in the source tree are good practice or not is another question. (Especially if said source tree is also built on Windows, not just Linux.)

@ulfjack : do you have any thoughts on this topic?

[ulfjack]

Funny, I just discussed this with @buchgr. He was working on related problems, but I'm not sure what his current plan is.

[crorvick]

@buchgr @ulfjack, do you have any further thoughts on this? This continues to be an infrequent source of confusion, and it's still not clear to me why Bazel should be navigating symbolic links at all. Thanks!

[crorvick]

@jin @ulfjack, is there a good story for why Bazel navigates symbolic links? If not, I think this deserves a 1.0 tag so this can be cleaned up before it becomes difficult to do so.

[ulfjack]

Is there a good story for why it shouldn't treat symlinks like files or directories? I'm not convinced that changing this will be less confusing overall.

[crorvick]

Well, I tried to tell one in the description of this ticket. It seems like the only use case for navigating a symbolic link is a hacky way to pull something into a workspace. This feels like an odd feature for Bazel to support.

I'm not really sure what it means for Bazel to navigate links within a workspace ... it seems to just result in Bazel compiling and linking the same thing multiple times.

In summary, I cannot see how symbolic links provide anything beyond the ability to shoot yourself in the foot. It seems out of character for the Bazel project to not have a strong opinion about such a thing.

[laszlocsomor]

Related bug: #6350