nvidia pods for driver, device plugin, dcgm exporter, feature discovery in crash loop after host reboot

[tusharrobin]

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

1. Quick Debug Checklist

• Are you running on an Ubuntu 18.04 node? => Centos 7.9
• [x ] Are you running Kubernetes v1.13+? => v1.20.5
• Are you running Docker (>= 18.06) or CRIO (>= 1.13+)? ==> 19.03.6
• Do you have i2c_core and ipmi_msghandler loaded on the nodes?
• Did you apply the CRD (kubectl describe clusterpolicies --all-namespaces) ==> Yes, its deployed via GPU operator

1. Issue or feature description

On Centos 7.9, GPU operator installed successfully and all pods became ready but after rebooting of node, pods went into crashloopbackoff state.

gpu-operator-resources gpu-feature-discovery-qcdp4 0/1 Init:CrashLoopBackOff 10 44m
gpu-operator-resources nvidia-container-toolkit-daemonset-rkg4b 0/1 Init:CrashLoopBackOff 10 44m
gpu-operator-resources nvidia-cuda-validator-ssgbh 0/1 Completed 0 42m
gpu-operator-resources nvidia-dcgm-exporter-kj45b 0/1 Init:CrashLoopBackOff 10 44m
gpu-operator-resources nvidia-device-plugin-daemonset-zdc4w 0/1 Init:CrashLoopBackOff 11 44m
gpu-operator-resources nvidia-device-plugin-validator-qbhtk 0/1 Completed 0 42m
gpu-operator-resources nvidia-driver-daemonset-svsmn 0/1 CrashLoopBackOff 10 44m
gpu-operator-resources nvidia-operator-validator-j9m2z 0/1 Init:CrashLoopBackOff 11 44m

2. Steps to reproduce the issue

1. Install Kubernetes
2. Install GPU operator and make sure all pods are running. Test sample pod with GPU to see if everything is working.
3. Reboot node
4. Pod go into crashloopbackoff

3. Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods --all-namespaces
• kubernetes daemonset status: kubectl get ds --all-namespaces
• If a pod/ds is in an error state or pending state kubectl describe pod -n NAMESPACE POD_NAME
  kubectl describe pod -n gpu-operator-resources nvidia-driver-daemonset-svsmn

Warning BackOff 3m16s (x143 over 34m) kubelet Back-off restarting failed container

  Reason:       ContainerCannotRun
  **Message:      OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli.real: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown**

• [] If a pod/ds is in an error state or pending state kubectl logs -n NAMESPACE POD_NAME

• Output of running a container on the GPU machine: docker run -it alpine echo foo

• [ ]
  $ docker run -it alpine echo foo
  foo

• Docker configuration file: cat /etc/docker/daemon.json

• Docker runtime configuration: docker info | grep runtime

docker info| grep -i runtime
Runtimes: nvidia runc
Default Runtime: nvidia

• NVIDIA shared directory: ls -la /run/nvidia
• NVIDIA packages directory: ls -la /usr/local/nvidia/toolkit
• NVIDIA driver directory: ls -la /run/nvidia/driver
• kubelet logs journalctl -u kubelet > kubelet.logs

[shivamerla]

@tusharrobin Can you provide the toolkit version you used? v1.7.0 ships with v0.5.0 toolkit which ensures to fallback to runc when driver modules are not loaded. Looks like old version of toolkit might be used here?

[tusharrobin]

@shivamerla I am using toolkit.version=1.4.0-ubi8. I am using 1.4.0 since I faced glibc issues mentioned in #72 when using default toolkit version shipped with 1.7.

[bstrdsmkr]

I'm working through this same problem after attempting an upgrade from helm chart version 1.6.0. I had this working using the 1.4.7-ubi8 toolkit. FYI, the available toolkit versions can be found here: https://hub.docker.com/r/nvidia/container-toolkit/tags?page=1&ordering=last_updated

It looks like 1.5.0 is the most recent. Theoretically, the "-ubi8" images are what you need for CentOS, though you'll have to do a bit of trial and error to find one compiled with a matching version of GLIBC

Also, according to the previous bugs, there were plans to have 1.5.0 compiled with a statically linked GLIBC though that apparently didn't happen: NVIDIA/libnvidia-container#81

[shivamerla]

Yes, thats the best option to switch to 1.5.0-ubi8 toolkit version. To recover from current state, you can change default-runtime to runc in docker daemon.json and restart docker. Toolkit will set the runtime again on startup.

[bstrdsmkr]

@shivamerla can we reopen this? I'm seeing this issue still on a RHEL 7.9 host with toolkit version 1.5.0-ubi8

[shivamerla]

@bstrdsmkr sure, i will re-open this. So, the solution works only with 1.4.7-ubi8 toolkit version?

[bstrdsmkr]

RHEL 7.8 worked with 1.4.7-ubi, but neither appears to work with RHEL 7.9 sadly :(

[shivamerla]

got it. seems like RHEL7.9 GLIBC version doesn't match with one packaged with either of those toolkit images. statically linking nvidia-container-cli will be the proper fix for this. Will try to get this fixed soon.

[bstrdsmkr]

@shivamerla thanks for that!

I'm definitely not great with makefiles, but looking at this comment: NVIDIA/libnvidia-container#81 (comment) and this makefile: https://github.com/NVIDIA/libnvidia-container/blob/master/Makefile it looks to me like I could build a custom image by cloning the repo and running make -f mk/docker.mk centos7 BIN_LDLIBS='-l:$(LIB_STATIC) -lseccomp -lcap -ldl' or possibly make -f mk/docker.mk centos7.9.2009 (based on the tags available for base images here: https://hub.docker.com/_/centos?tab=tags&page=1&ordering=last_updated)

Does that seem like an appropriate path or have I missed something?

[shivamerla]

I will let @klueska @elezar comment on this.

[elezar]

@bstrdsmkr unfortunately defining the contents of the make variables from the command line as you propose will not work as these are not conditionally set in the make files.

If you are building a custom image, what would be the reason for building statically? Would the centos7-x86_64 target not be sufficient in your case or is static linking a must-have even if you are able to produce a compatible library?

[bstrdsmkr]

@elezar The problem I'm trying to solve is the toolkit crashlooping:

Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: contai
ner init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: /usr/local/nvidia/toolkit/nvidia-container-cli.real: /lib64/libc.so.6: version 'GLIBC_2.27' not found (required by /usr/local
/nvidia/toolkit/libnvidia-container.so.1): unknown

This is a CentOS 7.9 node using either to 1.4.7-ubi8 or 1.5.0-ubi8 toolkit images. On CentOS 7.8, 1.4.7-ubi8 worked fine, but it appears that none of the ubi images are linked against CentOS 7.9's glibc 2.17:

[root@arm-login01 containerd]# ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[root@arm-login01 containerd]# cat /etc/*release
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
[root@arm-login01 containerd]#

I take from this thread: NVIDIA/libnvidia-container#81 (comment) that building a staticly linked binary is the long term fix for this problem, though until now using the ubi images were an acceptable workaround. Until there is solve for this, our ML platform is is broken :(

[janeczku]

Given that there is no workaround that would allow GPU operator to work on CentOS 7.9, is there a timeline for shipping the fix mentioned above ?