Failures on Debian with `ldconfig`

[brycelelbach]

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Also, before reporting a new issue, please make sure that:

• You read carefully the documentation and frequently asked questions.
• You searched for a similar issue and this is not a duplicate of an existing one.
• This issue is not related to NGC, otherwise, please use the devtalk forums instead.
• You went through the troubleshooting steps.

---

1. Issue or feature description

On Debian 10 and Debian unstable, nvidia-docker fails to run programs that use CUDA inside of containers UNLESS ldconfig is run first in the container to rebuild the the ldconfig cache.

Example failure:

[17:07:54]:wash@voyager:/home/wash/development/nvidia/cuda_linux_p4/sw/gpgpu/thrust/ci:0:$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi           NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.
Please also try adding directory that contains libnvidia-ml.so to your system PATH.
[17:07:57]:wash@voyager:/home/wash/development/nvidia/cuda_linux_p4/sw/gpgpu/thrust/ci:12:$ tail -n 5 /var/log/nvidia-container-toolkit.log
I1014 00:07:56.925518 4001429 nvc_ldcache.c:359] executing /sbin/ldconfig from host at /var/lib/docker/overlay2/1b23287eb935d89df1baab6e66ded34209ac3f6a371ccb11c307a553bd11cff4/merged
E1014 00:07:56.926236 1 nvc_ldcache.c:390] could not start /sbin/ldconfig: process execution failed: no such file or directory
I1014 00:07:56.943973 4001429 nvc.c:337] shutting down library context
I1014 00:07:56.944378 4001435 driver.c:156] terminating driver service
I1014 00:07:56.944613 4001429 driver.c:196] driver service terminated successfully

If I run ldconfig within the container to rebuild ld.so.cache first, everything works:

[17:09:00]:wash@voyager:/home/wash/development/nvidia/cuda_linux_p4/sw/gpgpu/thrust/ci:0:$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base bash -c "ldconfig && nvidia-smi"
Wed Oct 14 00:11:34 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.10       Driver Version: 455.10       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GT 710      On   | 00000000:04:00.0 N/A |                  N/A |
| 40%   41C    P8    N/A /  N/A |      1MiB /  2002MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            On   | 00000000:17:00.0 Off |                  N/A |
| 23%   30C    P8     8W / 250W |      1MiB / 12196MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Quadro GV100        On   | 00000000:65:00.0  On |                  Off |
| 32%   44C    P0    26W / 250W |      0MiB / 32505MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

This seems related to:

• Failed to run nvidia/cuda:8.0 via nvidia-docker2. nvidia-container-cli: ldcache error: process /sbin/ldconfig failed with error code: 127 #587
• Debian buster: nnvidia-container-cli: ldcache error: process /sbin/ldconfig failed with error code: 1 #660
• Couldn't find libnvidia-ml.so library in your system nvidia-container-toolkit#299

2. Steps to reproduce the issue

3. Information to attach (optional if deemed irrelevant)

• Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info

I1014 00:13:02.642845 4001640 nvc.c:282] initializing library context (version=1.3.0, build=16315ebdf4b9728e899f615e208b50c41d7a5d15)
I1014 00:13:02.642869 4001640 nvc.c:256] using root /
I1014 00:13:02.642873 4001640 nvc.c:257] using ldcache /etc/ld.so.cache
I1014 00:13:02.642876 4001640 nvc.c:258] using unprivileged user 1000:1000
I1014 00:13:02.642887 4001640 nvc.c:299] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I1014 00:13:02.642982 4001640 nvc.c:301] dxcore initialization failed, continuing assuming a non-WSL environment
W1014 00:13:02.644169 4001641 nvc.c:187] failed to set inheritable capabilities
W1014 00:13:02.644192 4001641 nvc.c:188] skipping kernel modules load due to failure
I1014 00:13:02.644291 4001642 driver.c:101] starting driver service
I1014 00:13:02.645319 4001640 nvc_info.c:680] requesting driver information with ''
I1014 00:13:02.646053 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.455.10
I1014 00:13:02.646118 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.455.10
I1014 00:13:02.646144 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.455.10
I1014 00:13:02.646162 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.455.10
I1014 00:13:02.646182 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.455.10
I1014 00:13:02.646210 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.455.10
I1014 00:13:02.646239 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.455.10
I1014 00:13:02.646258 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.455.10
I1014 00:13:02.646278 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.455.10
I1014 00:13:02.646305 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.455.10
I1014 00:13:02.646335 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.455.10
I1014 00:13:02.646354 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.455.10
I1014 00:13:02.646374 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.455.10
I1014 00:13:02.646393 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.455.10
I1014 00:13:02.646421 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.455.10
I1014 00:13:02.646448 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.455.10
I1014 00:13:02.646467 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.455.10
I1014 00:13:02.646487 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.455.10
I1014 00:13:02.646514 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cbl.so.455.10
I1014 00:13:02.646532 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.455.10
I1014 00:13:02.646562 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.455.10
I1014 00:13:02.646670 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.455.10
I1014 00:13:02.646765 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.455.10
I1014 00:13:02.646786 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.455.10
I1014 00:13:02.646807 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.455.10
I1014 00:13:02.646828 4001640 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.455.10
I1014 00:13:02.646853 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/vdpau/libvdpau_nvidia.so.455.10
I1014 00:13:02.646880 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-tls.so.455.10
I1014 00:13:02.646898 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-ptxjitcompiler.so.455.10
I1014 00:13:02.646925 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-opticalflow.so.455.10
I1014 00:13:02.646952 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-opencl.so.455.10
I1014 00:13:02.646971 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-ml.so.455.10
I1014 00:13:02.646998 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-ifr.so.455.10
I1014 00:13:02.647027 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-glvkspirv.so.455.10
I1014 00:13:02.647045 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-glsi.so.455.10
I1014 00:13:02.647064 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-glcore.so.455.10
I1014 00:13:02.647083 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-fbc.so.455.10
I1014 00:13:02.647110 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-encode.so.455.10
I1014 00:13:02.647137 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-eglcore.so.455.10
I1014 00:13:02.647156 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-compiler.so.455.10
I1014 00:13:02.647175 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-allocator.so.455.10
I1014 00:13:02.647204 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvcuvid.so.455.10
I1014 00:13:02.647242 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libcuda.so.455.10
I1014 00:13:02.647278 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libGLX_nvidia.so.455.10
I1014 00:13:02.647297 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libGLESv2_nvidia.so.455.10
I1014 00:13:02.647317 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libGLESv1_CM_nvidia.so.455.10
I1014 00:13:02.647337 4001640 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libEGL_nvidia.so.455.10
W1014 00:13:02.647351 4001640 nvc_info.c:350] missing library libnvidia-fatbinaryloader.so
W1014 00:13:02.647356 4001640 nvc_info.c:354] missing compat32 library libnvidia-cfg.so
W1014 00:13:02.647360 4001640 nvc_info.c:354] missing compat32 library libnvidia-fatbinaryloader.so
W1014 00:13:02.647363 4001640 nvc_info.c:354] missing compat32 library libnvidia-ngx.so
W1014 00:13:02.647366 4001640 nvc_info.c:354] missing compat32 library libnvidia-rtcore.so
W1014 00:13:02.647370 4001640 nvc_info.c:354] missing compat32 library libnvoptix.so
W1014 00:13:02.647373 4001640 nvc_info.c:354] missing compat32 library libnvidia-cbl.so
I1014 00:13:02.655816 4001640 nvc_info.c:276] selecting /usr/bin/nvidia-smi
I1014 00:13:02.655827 4001640 nvc_info.c:276] selecting /usr/bin/nvidia-debugdump
I1014 00:13:02.655839 4001640 nvc_info.c:276] selecting /usr/bin/nvidia-persistenced
I1014 00:13:02.655849 4001640 nvc_info.c:276] selecting /usr/bin/nvidia-cuda-mps-control
I1014 00:13:02.655859 4001640 nvc_info.c:276] selecting /usr/bin/nvidia-cuda-mps-server
I1014 00:13:02.655874 4001640 nvc_info.c:438] listing device /dev/nvidiactl
I1014 00:13:02.655877 4001640 nvc_info.c:438] listing device /dev/nvidia-uvm
I1014 00:13:02.655882 4001640 nvc_info.c:438] listing device /dev/nvidia-uvm-tools
I1014 00:13:02.655887 4001640 nvc_info.c:438] listing device /dev/nvidia-modeset
I1014 00:13:02.655928 4001640 nvc_info.c:317] listing ipc /run/nvidia-persistenced/socket
W1014 00:13:02.655937 4001640 nvc_info.c:321] missing ipc /tmp/nvidia-mps
I1014 00:13:02.655941 4001640 nvc_info.c:745] requesting device information with ''
I1014 00:13:02.661724 4001640 nvc_info.c:628] listing device /dev/nvidia0 (GPU-858ec672-5669-6e20-d0e8-194029d32d2c at 00000000:04:00.0)
I1014 00:13:02.667422 4001640 nvc_info.c:628] listing device /dev/nvidia1 (GPU-2da062d6-3b80-9750-0af9-85d39d0b010b at 00000000:17:00.0)
I1014 00:13:02.673130 4001640 nvc_info.c:628] listing device /dev/nvidia2 (GPU-58a70c9d-1070-2a96-e5b3-cbee8d19d9e3 at 00000000:65:00.0)
NVRM version:   455.10
CUDA version:   11.1

Device Index:   0
Device Minor:   0
Model:          GeForce GT 710
Brand:          GeForce
GPU UUID:       GPU-858ec672-5669-6e20-d0e8-194029d32d2c
Bus Location:   00000000:04:00.0
Architecture:   3.5

Device Index:   1
Device Minor:   1
Model:          TITAN Xp
Brand:          GeForce
GPU UUID:       GPU-2da062d6-3b80-9750-0af9-85d39d0b010b
Bus Location:   00000000:17:00.0
Architecture:   6.1

Device Index:   2
Device Minor:   2
Model:          Quadro GV100
Brand:          Quadro
GPU UUID:       GPU-58a70c9d-1070-2a96-e5b3-cbee8d19d9e3
Bus Location:   00000000:65:00.0
Architecture:   7.0
I1014 00:13:02.673174 4001640 nvc.c:337] shutting down library context
I1014 00:13:02.673531 4001642 driver.c:156] terminating driver service
I1014 00:13:02.673681 4001640 driver.c:196] driver service terminated successfully

• Kernel version from uname -a

Linux voyager 5.5.0-1-amd64 NVIDIA/nvidia-docker#1 SMP Debian 5.5.13-2 (2020-03-30) x86_64 GNU/Linux

• Driver information from nvidia-smi -a

Timestamp                                 : Tue Oct 13 17:15:06 2020
Driver Version                            : 455.10
CUDA Version                              : 11.1

Attached GPUs                             : 3
GPU 00000000:04:00.0
    Product Name                          : GeForce GT 710
    Product Brand                         : GeForce
    Display Mode                          : N/A
    Display Active                        : N/A
    Persistence Mode                      : Enabled
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : N/A
    Accounting Mode Buffer Size           : N/A
    Driver Model
        Current                           : N/A
        Pending                           : N/A
    Serial Number                         : N/A
    GPU UUID                              : GPU-858ec672-5669-6e20-d0e8-194029d32d2c
    Minor Number                          : 0
    VBIOS Version                         : 80.28.A6.00.12
    MultiGPU Board                        : N/A
    Board ID                              : N/A
    GPU Part Number                       : N/A
    Inforom Version
        Image Version                     : N/A
        OEM Object                        : N/A
        ECC Object                        : N/A
        Power Management Object           : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GPU Virtualization Mode
        Virtualization Mode               : N/A
        Host VGPU Mode                    : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x04
        Device                            : 0x00
        Domain                            : 0x0000
        Device Id                         : 0x128B10DE
        Bus Id                            : 00000000:04:00.0
        Sub System Id                     : 0x27123842
        GPU Link Info
            PCIe Generation
                Max                       : N/A
                Current                   : N/A
            Link Width
                Max                       : N/A
                Current                   : N/A
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : N/A
        Rx Throughput                     : N/A
    Fan Speed                             : 40 %
    Performance State                     : P8
    Clocks Throttle Reasons               : N/A
    FB Memory Usage
        Total                             : 2002 MiB
        Used                              : 1 MiB
        Free                              : 2001 MiB
    BAR1 Memory Usage
        Total                             : N/A
        Used                              : N/A
        Free                              : N/A
    Compute Mode                          : Default
    Utilization
        Gpu                               : N/A
        Memory                            : N/A
        Encoder                           : N/A
        Decoder                           : N/A
    Encoder Stats
        Active Sessions                   : N/A
        Average FPS                       : N/A
        Average Latency                   : N/A
    FBC Stats
        Active Sessions                   : N/A
        Average FPS                       : N/A
        Average Latency                   : N/A
    Ecc Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Errors
        Volatile
            Single Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
            Double Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
        Aggregate
            Single Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
            Double Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows                         : N/A
    Temperature
        GPU Current Temp                  : 41 C
        GPU Shutdown Temp                 : N/A
        GPU Slowdown Temp                 : N/A
        GPU Max Operating Temp            : N/A
        Memory Current Temp               : N/A
        Memory Max Operating Temp         : N/A
    Power Readings
        Power Management                  : N/A
        Power Draw                        : N/A
        Power Limit                       : N/A
        Default Power Limit               : N/A
        Enforced Power Limit              : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A
    Clocks
        Graphics                          : N/A
        SM                                : N/A
        Memory                            : N/A
        Video                             : N/A
    Applications Clocks
        Graphics                          : 954 MHz
        Memory                            : 900 MHz
    Default Applications Clocks
        Graphics                          : 954 MHz
        Memory                            : 900 MHz
    Max Clocks
        Graphics                          : N/A
        SM                                : N/A
        Memory                            : N/A
        Video                             : N/A
    Max Customer Boost Clocks
        Graphics                          : N/A
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Processes                             : None

GPU 00000000:17:00.0
    Product Name                          : TITAN Xp
    Product Brand                         : GeForce
    Display Mode                          : Disabled
    Display Active                        : Disabled
    Persistence Mode                      : Enabled
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : N/A
        Pending                           : N/A
    Serial Number                         : N/A
    GPU UUID                              : GPU-2da062d6-3b80-9750-0af9-85d39d0b010b
    Minor Number                          : 1
    VBIOS Version                         : 86.02.49.00.00
    MultiGPU Board                        : No
    Board ID                              : 0x1700
    GPU Part Number                       : N/A
    Inforom Version
        Image Version                     : G001.0000.01.04
        OEM Object                        : 1.1
        ECC Object                        : N/A
        Power Management Object           : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x17
        Device                            : 0x00
        Domain                            : 0x0000
        Device Id                         : 0x1B0210DE
        Bus Id                            : 00000000:17:00.0
        Sub System Id                     : 0x11DF10DE
        GPU Link Info
            PCIe Generation
                Max                       : 3
                Current                   : 1
            Link Width
                Max                       : 16x
                Current                   : 8x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 0 KB/s
        Rx Throughput                     : 0 KB/s
    Fan Speed                             : 23 %
    Performance State                     : P8
    Clocks Throttle Reasons
        Idle                              : Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    FB Memory Usage
        Total                             : 12196 MiB
        Used                              : 1 MiB
        Free                              : 12195 MiB
    BAR1 Memory Usage
        Total                             : 256 MiB
        Used                              : 5 MiB
        Free                              : 251 MiB
    Compute Mode                          : Default
    Utilization
        Gpu                               : 0 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    Ecc Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Errors
        Volatile
            Single Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
            Double Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
        Aggregate
            Single Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
            Double Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows                         : N/A
    Temperature
        GPU Current Temp                  : 30 C
        GPU Shutdown Temp                 : 99 C
        GPU Slowdown Temp                 : 96 C
        GPU Max Operating Temp            : N/A
        Memory Current Temp               : N/A
        Memory Max Operating Temp         : N/A
    Power Readings
        Power Management                  : Supported
        Power Draw                        : 9.77 W
        Power Limit                       : 250.00 W
        Default Power Limit               : 250.00 W
        Enforced Power Limit              : 250.00 W
        Min Power Limit                   : 125.00 W
        Max Power Limit                   : 300.00 W
    Clocks
        Graphics                          : 139 MHz
        SM                                : 139 MHz
        Memory                            : 405 MHz
        Video                             : 544 MHz
    Applications Clocks
        Graphics                          : 1404 MHz
        Memory                            : 5705 MHz
    Default Applications Clocks
        Graphics                          : 1404 MHz
        Memory                            : 5705 MHz
    Max Clocks
        Graphics                          : 1911 MHz
        SM                                : 1911 MHz
        Memory                            : 5705 MHz
        Video                             : 1620 MHz
    Max Customer Boost Clocks
        Graphics                          : N/A
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Processes                             : None

GPU 00000000:65:00.0
    Product Name                          : Quadro GV100
    Product Brand                         : Quadro
    Display Mode                          : Enabled
    Display Active                        : Enabled
    Persistence Mode                      : Enabled
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : N/A
        Pending                           : N/A
    Serial Number                         : 0320319013520
    GPU UUID                              : GPU-58a70c9d-1070-2a96-e5b3-cbee8d19d9e3
    Minor Number                          : 2
    VBIOS Version                         : 88.00.5A.00.03
    MultiGPU Board                        : No
    Board ID                              : 0x6500
    GPU Part Number                       : 900-5G500-0000-000
    Inforom Version
        Image Version                     : G500.0500.00.05
        OEM Object                        : 1.1
        ECC Object                        : 5.0
        Power Management Object           : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x65
        Device                            : 0x00
        Domain                            : 0x0000
        Device Id                         : 0x1DBA10DE
        Bus Id                            : 00000000:65:00.0
        Sub System Id                     : 0x121A10DE
        GPU Link Info
            PCIe Generation
                Max                       : 3
                Current                   : 3
            Link Width
                Max                       : 16x
                Current                   : 16x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 0 KB/s
        Rx Throughput                     : 0 KB/s
    Fan Speed                             : 32 %
    Performance State                     : P0
    Clocks Throttle Reasons
        Idle                              : Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    FB Memory Usage
        Total                             : 32505 MiB
        Used                              : 0 MiB
        Free                              : 32505 MiB
    BAR1 Memory Usage
        Total                             : 256 MiB
        Used                              : 3 MiB
        Free                              : 253 MiB
    Compute Mode                          : Default
    Utilization
        Gpu                               : 0 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    Ecc Mode
        Current                           : Disabled
        Pending                           : Disabled
    ECC Errors
        Volatile
            Single Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
            Double Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
        Aggregate
            Single Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
            Double Bit
                Device Memory             : N/A
                Register File             : N/A
                L1 Cache                  : N/A
                L2 Cache                  : N/A
                Texture Memory            : N/A
                Texture Shared            : N/A
                CBU                       : N/A
                Total                     : N/A
    Retired Pages
        Single Bit ECC                    : 0
        Double Bit ECC                    : 0
        Pending Page Blacklist            : No
    Remapped Rows                         : N/A
    Temperature
        GPU Current Temp                  : 44 C
        GPU Shutdown Temp                 : 90 C
        GPU Slowdown Temp                 : 88 C
        GPU Max Operating Temp            : 87 C
        Memory Current Temp               : 42 C
        Memory Max Operating Temp         : 95 C
    Power Readings
        Power Management                  : Supported
        Power Draw                        : 26.83 W
        Power Limit                       : 250.00 W
        Default Power Limit               : 250.00 W
        Enforced Power Limit              : 250.00 W
        Min Power Limit                   : 100.00 W
        Max Power Limit                   : 250.00 W
    Clocks
        Graphics                          : 135 MHz
        SM                                : 135 MHz
        Memory                            : 850 MHz
        Video                             : 555 MHz
    Applications Clocks
        Graphics                          : 1132 MHz
        Memory                            : 850 MHz
    Default Applications Clocks
        Graphics                          : 1132 MHz
        Memory                            : 850 MHz
    Max Clocks
        Graphics                          : 1912 MHz
        SM                                : 1912 MHz
        Memory                            : 850 MHz
        Video                             : 1717 MHz
    Max Customer Boost Clocks
        Graphics                          : 1912 MHz
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Processes                             : None

• Docker version from docker version

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:26:02 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:23 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version      Architecture Description
+++-=============================-============-============-=====================================================
un  libgldispatch0-nvidia         <none>       <none>       (no description available)
ii  libnvidia-container-tools     1.3.0-1      amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64    1.3.0-1      amd64        NVIDIA container runtime library
ii  nvidia-container-runtime      3.4.0-1      amd64        NVIDIA container runtime
un  nvidia-container-runtime-hook <none>       <none>       (no description available)
ii  nvidia-container-toolkit      1.3.0-1      amd64        NVIDIA container runtime hook
un  nvidia-docker                 <none>       <none>       (no description available)
ii  nvidia-docker2                2.5.0-1      all          nvidia-docker CLI wrapper
un  nvidia-libopencl1-dev         <none>       <none>       (no description available)

My installation of the display driver and CUDA is a local debug build from source and is rougly CUDA 11.0 / R455.

• NVIDIA container library version from nvidia-container-cli -V

version: 1.3.0
build date: 2020-09-16T12:33+00:00
build revision: 16315ebdf4b9728e899f615e208b50c41d7a5d15
build compiler: x86_64-linux-gnu-gcc-8 8.3.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

[dualvtable]

@klueska - do you have any insight into this issue?

[nvjmayo]

In your example I see you use sudo:
sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Sudo should not be necessary to run this, and I won't be using it in my examples below.

Here's an attempt with an old CUDA 10.2 image that does work on Debian 10 and an older version of nvidia-docker (1.2.0):

~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
~$ nvidia-container-cli -V
version: 1.2.0
build date: 2020-07-08T19:33+00:00
build revision: d22237acaea94aa5ad5de70aac903534ed598819
build compiler: x86_64-linux-gnu-gcc-8 8.3.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
~$ docker run --rm --gpus all nvidia/cuda:10.2-base nvidia-smi
Unable to find image 'nvidia/cuda:10.2-base' locally
10.2-base: Pulling from nvidia/cuda
171857c49d0f: Pull complete
419640447d26: Pull complete
61e52f862619: Pull complete
c118dad7e37a: Pull complete
29c091e4be16: Pull complete
d85c81a4428d: Pull complete
Digest: sha256:1774efa6e102a9bdfa393521c6b2254ea751dc1af9cff25c51376128bfb08d5e
Status: Downloaded newer image for nvidia/cuda:10.2-base
Fri Oct 30 19:59:35 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.31       Driver Version: 440.31       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 710      Off  | 00000000:65:00.0 N/A |                  N/A |
| 50%   40C    P0    N/A /  N/A |      0MiB /  2001MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

I'm leaning towards these possibilities:

• insufficient support in the version of nvidia-docker you have for CUDA 11 library paths.
• failure of nvidia-docker to detect ubuntu/debian versus RHEL/CentOS path conventions. (seems unlikely, but the logic can be a little fragile so worth me looking into)
• regression in debian support on 1.3.0 versus 1.2.0 (see above)

[klueska]

I am also unable to reproduce this on a fresh debian10 system with the latest drivers and the latest nvidia-docker2.

My entire history of commands from the time I brought the system online until I executed docker:

    1  sudo apt-get install build-essential
    2  sudo apt-get install linux-headers-$(uname -r)
    3  wget https://us.download.nvidia.com/tesla/450.80.02/NVIDIA-Linux-x86_64-450.80.02.run
    4  chmod a+x NVIDIA-Linux-x86_64-450.80.02.run
    5  sudo ./NVIDIA-Linux-x86_64-450.80.02.run
    6  curl https://get.docker.com | sh   && sudo systemctl start docker   && sudo systemctl enable docker
    7  distribution=$(. /etc/os-release;echo $ID$VERSION_ID)    && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -    && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
    8  sudo apt-get update
    9  sudo apt-get install -y nvidia-docker2
   10  sudo systemctl restart docker
   11  sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

With output:

$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Unable to find image 'nvidia/cuda:11.0-base' locally
11.0-base: Pulling from nvidia/cuda
54ee1f796a1e: Pull complete
f7bfea53ad12: Pull complete
46d371e02073: Pull complete
b66c17bbf772: Pull complete
3642f1a6dfb3: Pull complete
e5ce55b8b4b9: Pull complete
155bc0332b0a: Pull complete
Digest: sha256:774ca3d612de15213102c2dbbba55df44dc5cf9870ca2be6c6e9c627fa63d67a
Status: Downloaded newer image for nvidia/cuda:11.0-base
Fri Oct 30 22:00:59 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  T4 32GB             Off  | 00000000:65:00.0 Off |                    0 |
| N/A   38C    P0    19W / 150W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

System info:

$ sudo lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10

Package info:

$ dpkg -l '*nvidia*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                             Version      Architecture Description
+++-================================-============-============-=====================================================
un  libgldispatch0-nvidia            <none>       <none>       (no description available)
ii  libnvidia-container-tools        1.3.0-1      amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64       1.3.0-1      amd64        NVIDIA container runtime library
ii  nvidia-container-runtime         3.4.0-1      amd64        NVIDIA container runtime
un  nvidia-container-runtime-hook    <none>       <none>       (no description available)
ii  nvidia-container-toolkit         1.3.0-1      amd64        NVIDIA container runtime hook
un  nvidia-docker                    <none>       <none>       (no description available)
ii  nvidia-docker2                   2.5.0-1      all          nvidia-docker CLI wrapper
un  nvidia-legacy-304xx-vdpau-driver <none>       <none>       (no description available)
un  nvidia-legacy-340xx-vdpau-driver <none>       <none>       (no description available)
un  nvidia-vdpau-driver              <none>       <none>       (no description available)

Contents of /etc/nvidia-container-runtime/config.toml (the default for debian10):

disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
#accept-nvidia-visible-devices-as-volume-mounts = false

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
#user = "root:video"
ldconfig = "@/sbin/ldconfig"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"

[xkszltl]

Kind of off the topic but, am I seeing a 16GB GPU named as T4 32GB?

[3ronco]

I don't think it's a problem with ldconfig but with the docker-nvidia-runtime prehook which provides the necessary driver libs from the host dynamically and depends on correct configuration of NVIDIA_VISIBLE_DEVICES & NVIDIA_DRIVER_CAPABILITIES. Notice that my config.toml is in original state (with the @ in ldconfig):

disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
#accept-nvidia-visible-devices-as-volume-mounts = false

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
#user = "root:video"
ldconfig = "@/sbin/ldconfig"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"

1) Logically fails when you omit both NVIDIA_VISIBLE_DEVICES & NVIDIA_DRIVER_CAPABILITIES:

docker run -ti --rm --privileged \
	--runtime=nvidia \
	nvidia/libnvidia-container/debian10-amd64;
me@container:/tmp/libnvidia-container# nvidia-container-cli list
nvidia-container-cli: error while loading shared libraries: libnvidia-container.so.1: cannot open shared object file: No such file or directory

2) With an NVIDIA_VISIBLE_DEVICES set to first device.

docker run -ti --rm --privileged --runtime=nvidia \
		-e NVIDIA_VISIBLE_DEVICES=0 \
		nvidia/libnvidia-container/debian10-amd64;

So by default without NVIDIA_DRIVER_CAPABILITIES it loads the config & management libs:

me@container:/tmp/libnvidia-container# ll /usr/lib/x86_64-linux-gnu/ | grep libnv
lrwxrwxrwx 1 root root   27 Nov 29 16:29 libnvidia-cfg.so.1 -> libnvidia-cfg.so.418.152.00
-rw-r--r-- 1 root root 187K Jun  2 02:42 libnvidia-cfg.so.418.152.00
lrwxrwxrwx 1 root root   26 Nov 29 16:29 libnvidia-ml.so.1 -> libnvidia-ml.so.418.152.00
-rw-r--r-- 1 root root 1.6M Jun  2 02:46 libnvidia-ml.so.418.152.00

By the way, don't confuse the index order with the order at host even when your host has two GPUs with the nvidia device being the second one the index still starts at 0 as NVIDIA_VISIBLE_DEVICES naturally sees only nvidia devices.

3) Here's a strange case, you may give an invalid list for NVIDIA_DRIVER_CAPABILITIES

docker run -ti --rm --privileged --runtime=nvidia \
		-e NVIDIA_VISIBLE_DEVICES=0 \
		-e NVIDIA_DRIVER_CAPABILITIES=video,compat32,graphics,display \
		nvidia/libnvidia-container/debian10-amd64;
me@container:/tmp/libnvidia-container# nvidia-container-cli list
nvidia-container-cli: initialization error: driver error: failed to process request

Although some of the libs got loaded, ...

me@container:/tmp/libnvidia-container# ll /usr/lib/x86_64-linux-gnu/ | grep libnv
-rw-r--r-- 1 root root  25M Jun  2 03:07 libnvidia-eglcore.so.418.152.00
-rw-r--r-- 1 root root  27M Jun  2 02:51 libnvidia-glcore.so.418.152.00
-rw-r--r-- 1 root root 653K Jun  2 02:49 libnvidia-glsi.so.418.152.00
-rw-r--r-- 1 root root  14M Jun  2 03:03 libnvidia-glvkspirv.so.418.152.00
-rw-r--r-- 1 root root  15K Jun  2 02:30 libnvidia-tls.so.418.152.00

... nvidia-container-cli still fails because i've omitted utility in the list (!)

4) While with all capabilites utility,video,compat32,graphics,display or simply all

docker run -ti --rm --privileged --runtime=nvidia \
		-e NVIDIA_VISIBLE_DEVICES=0 \
		-e NVIDIA_DRIVER_CAPABILITIES=utility,video,compat32,graphics,display \
		nvidia/libnvidia-container/debian10-amd64;

me@container:tmp/libnvidia-container# ll /usr/lib/x86_64-linux-gnu/ | grep libnv
lrwxrwxrwx 1 root root   27 Nov 29 16:49 libnvidia-cfg.so.1 -> libnvidia-cfg.so.418.152.00
-rw-r--r-- 1 root root 187K Jun  2 02:42 libnvidia-cfg.so.418.152.00
-rw-r--r-- 1 root root  25M Jun  2 03:07 libnvidia-eglcore.so.418.152.00
-rw-r--r-- 1 root root  27M Jun  2 02:51 libnvidia-glcore.so.418.152.00
-rw-r--r-- 1 root root 653K Jun  2 02:49 libnvidia-glsi.so.418.152.00
-rw-r--r-- 1 root root  14M Jun  2 03:03 libnvidia-glvkspirv.so.418.152.00
lrwxrwxrwx 1 root root   26 Nov 29 16:49 libnvidia-ml.so.1 -> libnvidia-ml.so.418.152.00
-rw-r--r-- 1 root root 1.6M Jun  2 02:46 libnvidia-ml.so.418.152.00
-rw-r--r-- 1 root root  15K Jun  2 02:30 libnvidia-tls.so.418.152.00

me@container:tmp/libnvidia-container# nvidia-container-cli list
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset
/dev/nvidia0
/usr/bin/nvidia-smi
/usr/bin/nvidia-debugdump
/usr/bin/nvidia-persistenced
/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.418.152.00
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.418.152.00
/usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.418.152.00
/usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.418.152.00
/usr/lib/x86_64-linux-gnu/libnvidia-tls.so.418.152.00
/usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.418.152.00
/usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.418.152.00
/usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.418.152.00
/usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.418.152.00
/usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.418.152.00
/usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.418.152.00
/run/nvidia-persistenced/socket

It works as expected. Since the nvidia-container-cli depends on the management lib libnvidia-ml.so.1 you should consider providing that lib always regardless of what is given in NVIDIA_DRIVER_CAPABILITIES env var.

[klueska]

@3ronco What you are saying is mostly correct, except that the exact error being reported by this issue:

could not start /sbin/ldconfig: process execution failed: no such file or directory

and not:

nvidia-container-cli: error while loading shared libraries: libnvidia-container.so.1: cannot open shared object file: No such file or directory
or
nvidia-container-cli: initialization error: driver error: failed to process request

Which means they are running a container that definitely triggers the nvidia-container-toolkit to be invoked properly. In fact, they are running the nvidia/cuda:11.0-base image, which definitely has sufficient values for NVIDIA_DRIVER_CAPABILITIES and NVIDIA_VISIBILE_DEVICE set in it.

Also, this issue only seems to appear on debian 10 systems and not others.
Unfortunately, I am unable to reproduce this issue (even on debian 10), so it is hard for me to root cause it.
If anyone can give me more info on the particulars of their system so I can reproduce it, that would go a long way towards resolving this (somewhat longstanding) issue.

[3ronco]

@klueska ok, i see ... i've used an image built from source (nvidia/libnvidia-container/debian10-amd64) ... may explain the missing setting for the env vars in my setup but like you i'm unable to reproduce the issue the debug output from /var/log/nvidia-container-toolkit.log gives no error.

... nvc_ldcache.c:359] executing /sbin/ldconfig from host at /var/lib/docker/btrfs/subvolumes/c068652...

[dikkepanda]

I am having similar problems. This is on a Debian installed with OMV . When I change @/ldconfig to /ldconfig (which I often encounter), I have the following message in my logs:

/bin/sh: error while loading shared libraries: /lib/x86_64-linux-gnu/libc.so.6: cannot read file data: Operation not permitted

[rjeli]

it's because when trying to execveat host ldconfig, security_bprm_check in the kernel returns -ENOENT. not sure why yet.

[rjeli]

it's not AppArmor, it still happens with apparmor=0 boot param. not sure how else to trace who is denying the LSM hook.