eBPF module fails to load on Fedora 32

[roysjosh]

Describe the bug
falco fails to start when configured to use the BPF module with a verifier exception.

How to reproduce it

# FALCO_BPF_PROBE="" falco -v
...
jump out of range from insn 8 to 584
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
jump out of range from insn 8 to 584
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
Thu Jun  4 17:09:56 2020: Runtime error: bpf_load_program() err=22 event=filler/sys_single message=jump out of range from insn 8 to 584
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
. Exiting.

Expected behaviour
Successful start of falco with eBPF functionality.

Environment

• Falco version:

Falco version: 0.23.0
Driver version: 96bd9bc560f67742738eb7255aeb4d03046b8045

• System info:

{
  "machine": "x86_64",
  "nodename": "falco.example.com",
  "release": "5.6.15-300.fc32.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Fri May 29 14:23:59 UTC 2020"
}

• Cloud provider or hardware configuration: KVM VM
• OS:

NAME=Fedora
VERSION="32 (Cloud Edition)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora 32 (Cloud Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Cloud Edition"
VARIANT_ID=cloud

• Kernel:
  Linux falco.example.com 5.6.15-300.fc32.x86_64 #1 SMP Fri May 29 14:23:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Installation method:
  RPM

Additional context

[fntlnz]

Thanks a lot for bringing this to our attention! Can you post the clang version ? @roysjosh

[roysjosh]

clang-10.0.0-1.fc32.x86_64

[leodido]

I remember both me and @fntlnz experienced this too with clang 10. Would you try to compile it with clang 9 please?

On Wed, 10 Jun 2020 at 17:20, Joshua Roys ***@***.***> wrote: clang-10.0.0-1.fc32.x86_64 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1251 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5J443ZFHKHFJ3NS5LDLLRV6QC7ANCNFSM4NS34HZQ> .

-- L.

[fntlnz]

Just noticed again the same again with Clang 10 on Arch Linux.

jump out of range from insn 8 to 584
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
jump out of range from insn 8 to 584
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
bpf_load_program() err=22 event=filler/sys_single message=jump out of range from insn 8 to 584
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

clang --version                                                                                                       
clang version 10.0.0 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[leodido]

I was about to call it a day when I randomly encountered an LLVM BPF backend bug that somehow seems to be possibly related to this issue (the fix is available into LLVM 11 trunk). I'm especially referring to the CORE_MEM pseudo op.

Clearly I'm not sure at all, but I leave the link here for next week:
https://reviews.llvm.org/D78466

[fntlnz]

@leodido when you got some time, can you dump here a bit more information on why you think this is related? It's the generated code? I'd love to see a diff of what the two revisions of the backend generate.

Maybe there's a visualization tool like bpftool for this phase? We can't use bpftool because the program doesn't even get loaded. :/

[yunbo-xufeng]

I also met the same problem with LLVM-11

[fntlnz]

/assign @leodido
/assign @fntlnz

[fntlnz]

We (@leodido and I) are working on this, it seems like with newer kernels and newer versions of clang we jump out of range when we have more than one subprogram.

[krisnova]

See commentary in #1463 for replication steps.

However I was able to remedy to problem by using a different version of clang to compile the BPF probe.

Arch Linux

pacaur -R clang
pacaur -S llvm90 clang90 # <--- These packages are installed in /opt/llvm90 
PATH=$PATH:/opt/llvm90/bin
cd ${HOME}
git clone git@github.com:falcosecurity/falco.git
cd falco
mkdir build
cd build
cmake ../ \
      -DBUILD_BPF="ON" \
      -DBUILD_WARNINGS_AS_ERRORS="OFF" \
      -DCMAKE_BUILD_TYPE="Release" \
      -DCMAKE_INSTALL_PREFIX="/usr" \
      -DFALCO_ETC_DIR="/etc/falco" \
      -DUSE_BUNDLED_DEPS=ON
make bpf
make falco
sudo cp driver/bpf/probe.o /root/.falco/falco-bpf.o
sudo cp userspace/falco/falco /usr/bin/falco
sudo FALCO_BPF_PROBE="" falco

Works for me on the following kernel

Linux emily 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux

with this version of clang

clang version 9.0.1 (https://aur.archlinux.org/clang90 27393b8bd3cb3d511808bd0b60f79dd0d2f26286)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm90/bin