2.00.21 - ffmpeg v6 requires newer drivers for nvenc than ffmpeg v5

[mbentley]

Describe the bug
nvenc is no longer transcoding on my machine after the latest 2.00.21 update which upgrades ffmpeg.

To Reproduce

1. Start with a host running Debian 11 (bullseye) with the nvidia-driver package from the Debian repos
2. Start with tdarr 2.00.20 in Docker, verify it can transcode.
3. Upgrade to 2.00.21 and try to transcode a file and it will fail.
4. Roll back to 2.00.20 and it works again.

nvidia-driver version info:

# dpkg -l nvidia-driver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  nvidia-driver  470.182.03-1 amd64        NVIDIA metapackage

Expected behavior
ffmpeg would either fall back to the previous version or a documented workaround of other sorts would exist. I imagine that I could mount in a version of ffmpeg that works but that somewhat defeats the purpose of a container abstraction.

Screenshots
n/a

Please provide the following information:

• Config files [can be found in /app/configs/ when using Docker or in the /configs folder next to Tdarr_Updater if not using Docker]

# cat Tdarr_Server_Config.json
{
  "serverPort": "8266",
  "webUIPort": "8265",
  "serverIP": "172.20.0.36",
  "handbrakePath": "",
  "ffmpegPath": "",
  "logLevel": "INFO",
  "mkvpropeditPath": "",
  "ccextractorPath": "",
  "openBrowser": true
}

• Log files [can be found in /app/logs/ when using Docker or in the /logs folder next to Tdarr_Updater if not using Docker]

-Worker error [can be found on the 'Tdarr' tab by pressing the 'i' button on a failed item in the staged file section or in the transcode error section at the bottom]

2023-06-15T15:15:30.423Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:[Step W05] [C2] Launching subworker
2023-06-15T15:15:30.424Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Preparing to launch subworker
2023-06-15T15:15:30.424Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker launched
2023-06-15T15:15:30.424Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:[1/3] Sending command to subworker
2023-06-15T15:15:30.424Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:[2/3] tdarr-ffmpeg -c:v h264_cuvid -i "/temp/Gravity - 2K Trailer-new3-TdarrCacheFile-Xx5qG0yuD.mp4" -map 0 -dn -c:v hevc_nvenc -pix_fmt p010le -qmin 0 -cq:V 31 -b:v 2500k -maxrate:v 5000k -preset slow -rc-lookahead 32 -spatial_aq:v 1 -aq-strength:v 8 -a53cc 0 -c:a copy -c:s copy "/temp/Gravity - 2K Trailer-new3-TdarrCacheFile-xfYdDsMbi.mkv"
2023-06-15T15:15:30.424Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:[3/3] Command sent
2023-06-15T15:15:30.425Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:To see live CLI output, enable 'Log full FFmpeg/HandBrake output' in the staging section on the Tdarr tab before the job starts. Note this could increase the job report size substantially.
2023-06-15T15:15:30.425Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker:Online
2023-06-15T15:15:30.425Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker:Receiving transcode settings
2023-06-15T15:15:30.425Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker:Running CLI
2023-06-15T15:15:31.425Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker:a.Thread closed, code: 1
2023-06-15T15:15:31.426Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker exit approved, killing subworker
2023-06-15T15:15:31.426Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Subworker killed
2023-06-15T15:15:31.426Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:b.Thread closed, code: 1
2023-06-15T15:15:31.427Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:CLI code: 1
2023-06-15T15:15:31.427Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:Last 200 lines of CLI log:
2023-06-15T15:15:31.427Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
2023-06-15T15:15:31.427Z built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
2023-06-15T15:15:31.427Z configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z libavutil 58. 2.100 / 58. 2.100
2023-06-15T15:15:31.427Z libavcodec 60. 3.100 / 60. 3.100
2023-06-15T15:15:31.427Z libavformat 60. 3.100 / 60. 3.100
2023-06-15T15:15:31.427Z libavdevice 60. 1.100 / 60. 1.100
2023-06-15T15:15:31.427Z libavfilter 9. 3.100 / 9. 3.100
2023-06-15T15:15:31.427Z libswscale 7. 1.100 / 7. 1.100
2023-06-15T15:15:31.427Z libswresample 4. 10.100 / 4. 10.100
2023-06-15T15:15:31.427Z libpostproc 57. 1.100 / 57. 1.100
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z [mov,mp4,m4a,3gp,3g2,mj2 @ 0x558a1983f100] stream 0, timescale not set
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/media/test/Gravity - 2K Trailer-new3.mp4':
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z major_brand : mp42
2023-06-15T15:15:31.427Z minor_version : 0
2023-06-15T15:15:31.427Z compatible_brands: mp42isomavc1
2023-06-15T15:15:31.427Z creation_time : 2013-12-01T03:59:56.000000Z
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z genre : Trailer
2023-06-15T15:15:31.427Z artist : Warner Bros.
2023-06-15T15:15:31.427Z title : Gravity - 2K Trailer
2023-06-15T15:15:31.427Z encoder : HandBrake 0.9.9 2013051800
2023-06-15T15:15:31.427Z date : 2013
2023-06-15T15:15:31.427Z Duration: 00:02:27.07, start: 0.000000, bitrate: 20296 kb/s
2023-06-15T15:15:31.427Z Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 2048x858, 20149 kb/s, 23.98 fps, 23.98 tbr, 90k tbn (default)
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z creation_time : 2013-12-01T03:59:56.000000Z
2023-06-15T15:15:31.427Z vendor_id : [0][0][0][0]
2023-06-15T15:15:31.427Z encoder : JVT/AVC Coding
2023-06-15T15:15:31.427Z Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 153 kb/s (default)
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z creation_time : 2013-12-01T03:59:56.000000Z
2023-06-15T15:15:31.427Z vendor_id : [0][0][0][0]
2023-06-15T15:15:31.427Z Stream #0:2[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 102x150 [SAR 72:72 DAR 17:25], 90k tbr, 90k tbn (attached pic)
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Output #0, mp4, to '/temp/Gravity - 2K Trailer-new3-TdarrCacheFile-Xx5qG0yuD.mp4':
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z major_brand : mp42
2023-06-15T15:15:31.427Z minor_version : 0
2023-06-15T15:15:31.427Z compatible_brands: mp42isomavc1
2023-06-15T15:15:31.427Z date : 2013
2023-06-15T15:15:31.427Z genre : Trailer
2023-06-15T15:15:31.427Z artist : Warner Bros.
2023-06-15T15:15:31.427Z title : Gravity - 2K Trailer
2023-06-15T15:15:31.427Z encoder : Lavf60.3.100
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 2048x858, q=2-31, 20149 kb/s, 23.98 fps, 23.98 tbr, 90k tbn (default)
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z creation_time : 2013-12-01T03:59:56.000000Z
2023-06-15T15:15:31.427Z vendor_id : [0][0][0][0]
2023-06-15T15:15:31.427Z encoder : JVT/AVC Coding
2023-06-15T15:15:31.427Z Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 153 kb/s (default)
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z creation_time : 2013-12-01T03:59:56.000000Z
2023-06-15T15:15:31.427Z vendor_id : [0][0][0][0]
2023-06-15T15:15:31.427Z Stream mapping:
2023-06-15T15:15:31.427Z Stream #0:0 -> #0:0 (copy)
2023-06-15T15:15:31.427Z Stream #0:1 -> #0:1 (copy)
2023-06-15T15:15:31.427Z Press [q] to stop, [?] for help
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z frame= 0 fps=0.0 q=-1.0 size= 0kB time=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
2023-06-15T15:15:31.427Z frame= 3525 fps=0.0 q=-1.0 Lsize= 364391kB time=00:02:27.05 bitrate=20299.7kbits/s speed= 448x
2023-06-15T15:15:31.427Z video:361517kB audio:2760kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.031461%
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
2023-06-15T15:15:31.427Z built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
2023-06-15T15:15:31.427Z configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z libavutil 58. 2.100 / 58. 2.100
2023-06-15T15:15:31.427Z libavcodec 60. 3.100 / 60. 3.100
2023-06-15T15:15:31.427Z libavformat 60. 3.100 / 60. 3.100
2023-06-15T15:15:31.427Z libavdevice 60. 1.100 / 60. 1.100
2023-06-15T15:15:31.427Z libavfilter 9. 3.100 / 9. 3.100
2023-06-15T15:15:31.427Z libswscale 7. 1.100 / 7. 1.100
2023-06-15T15:15:31.427Z libswresample 4. 10.100 / 4. 10.100
2023-06-15T15:15:31.427Z libpostproc 57. 1.100 / 57. 1.100
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/temp/Gravity - 2K Trailer-new3-TdarrCacheFile-Xx5qG0yuD.mp4':
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z major_brand : isom
2023-06-15T15:15:31.427Z minor_version : 512
2023-06-15T15:15:31.427Z compatible_brands: isomiso2avc1mp41
2023-06-15T15:15:31.427Z title : Gravity - 2K Trailer
2023-06-15T15:15:31.427Z artist : Warner Bros.
2023-06-15T15:15:31.427Z date : 2013
2023-06-15T15:15:31.427Z encoder : Lavf60.3.100
2023-06-15T15:15:31.427Z genre : Trailer
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Duration: 00:02:27.07, start: 0.000000, bitrate: 20296 kb/s
2023-06-15T15:15:31.427Z Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 2048x858, 20143 kb/s, 23.98 fps, 23.98 tbr, 90k tbn (default)
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z handler_name : VideoHandler
2023-06-15T15:15:31.427Z vendor_id : [0][0][0][0]
2023-06-15T15:15:31.427Z encoder : JVT/AVC Coding
2023-06-15T15:15:31.427Z Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 153 kb/s (default)
2023-06-15T15:15:31.427Z Metadata:
2023-06-15T15:15:31.427Z handler_name : SoundHandler
2023-06-15T15:15:31.427Z vendor_id : [0][0][0][0]
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Stream mapping:
2023-06-15T15:15:31.427Z Stream #0:0 -> #0:0 (h264 (h264_cuvid) -> hevc (hevc_nvenc))
2023-06-15T15:15:31.427Z Stream #0:1 -> #0:1 (copy)
2023-06-15T15:15:31.427Z Press [q] to stop, [?] for help
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z [hevc_nvenc @ 0x5626962d7ec0] Driver does not support the required nvenc API version. Required: 12.0 Found: 11.1
2023-06-15T15:15:31.427Z [hevc_nvenc @ 0x5626962d7ec0] The minimum required Nvidia driver for nvenc is 520.56.06 or newer
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z [vost#0:0/hevc_nvenc @ 0x5626962b5e80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z Conversion failed!
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.427Z
2023-06-15T15:15:31.428Z U2gS6HYXr:Node[node1]:Worker[khaki-kid]:[-error-]

• OS: Debian 11 (bullseye)
• Browser: n/a
• Version: 2.00.21

Additional context
How I run my container:

docker run -d \
  --privileged \
  --name tdarr \
  --network nginx-int \
  --ip 172.20.0.36 \
  --restart unless-stopped \
  --cpus 4 \
  --memory 16G \
  --memory-swap 17G \
  --gpus 'all,"capabilities=compute,video,utility"' \
  --device /dev/nvidia0 \
  --device /dev/nvidiactl \
  -e internalNode="true" \
  -e serverIP="172.20.0.36" \
  -e serverPort="8266" \
  -e webUIPort="8265" \
  -e THREADS_WORKER_INIT_TIMEOUT="20000" \
  -e TZ="US/Eastern" \
  -e PUID=501 \
  -e PGID=1501 \
  --mount type=bind,source=/zfs/apps/tdarr/server,destination=/app/server,readonly=false \
  --mount type=bind,source=/zfs/apps/tdarr/configs,destination=/app/configs,readonly=false \
  --mount type=bind,source=/zfs/apps/tdarr/logs,destination=/app/logs,readonly=false \
  --mount type=bind,source=/zfs/storage/media,destination=/media,readonly=false \
  --mount type=bind,source=/zfs/apps/tdarr/transcode_cache,destination=/temp,readonly=false \
  haveagitgat/tdarr

[mbentley]

2023-06-15T15:15:31.427Z [hevc_nvenc @ 0x5626962d7ec0] Driver does not support the required nvenc API version. Required: 12.0 Found: 11.1
2023-06-15T15:15:31.427Z [hevc_nvenc @ 0x5626962d7ec0] The minimum required Nvidia driver for nvenc is 520.56.06 or newer

Not necessarily driver related but ffmpeg related on the version bump it seems.

I copied ffmpeg from the last working version (2.00.20) to the most recent and it goes back to transcoding fine for me. I imagine there might be other dependencies that would need to be messed with but just as a super quick test, I am guessing it's a ffmpeg related change in v6.

# STAGE 0: use version 2.00.20 to pull ffmpeg from
FROM haveagitgat/tdarr:2.00.20 AS base

# STAGE 1: copy over ffmpeg + dependency
FROM haveagitgat/tdarr:latest
COPY --from=base /usr/lib/jellyfin-ffmpeg/ffmpeg /usr/lib/jellyfin-ffmpeg/ffmpeg
COPY --from=base /usr/lib/jellyfin-ffmpeg/lib/libplacebo.so.233 /usr/lib/jellyfin-ffmpeg/lib/libplacebo.so.233

I build and ran this image and it works as expected.

[supersnellehenk]

Generally speaking you shouldn't install the nvidia driver from the Debian repo. Your version 470.182.03-1 is very old, the latest on the Nvidia website is 535.54.03. So I wouldn't really call this a Tdarr issue myself.

[mbentley]

That directly contradicts with what the suggestion is from NVIDIA:

NVIDIA Drivers
Before you get started, make sure you have installed the NVIDIA driver for your Linux distribution. The recommended way to install drivers is to use the package manager for your distribution but other installer mechanisms are also available (e.g. by downloading .run installers from NVIDIA Driver Downloads).

I would love to upgrade to bookworm which has 525.105.17-1 but nvidia-docker hasn't been released yet and backports is only up to 470.103.01-1~bpo11+1. Dealing with the drivers outside of apt & the Debian packages is just less than ideal.

[supersnellehenk]

Yeah in your case you'd be stuck with either manually swapping out ffmpeg 6 for ffmpeg 5, or sticking with 20.00.20.1. Tdarr can't really influence what Nvidia driver is available in the package manager of your distro. (Nor would Nvidia probably expect the package manager to have such an outdated version)

[HaveAGitGat]

Yeah suppose only solution from my side would be to include ffmpeg 5 as well or have a separate container for it but neither are ideal,

[quagsirus]

This drops support for Kepler GPUs such as the GK208B (revised GT 730) since the latest supported drivers for that series is 470.xx
It really would be good to include a different version of FFmpeg built with the sdk/11.1 branch of https://github.com/FFmpeg/nv-codec-headers to maintain compatibility

edit: I'm pretty sure it would still be possible to use the latest version of FFmpeg (>=6) just built with the different headers.

[triestelaporte]

Running Tdarr in docker on Truenas Scale 22.12.3.2, every plugin that attempts to transcode fails with the following errors:

2023-07-26T13:40:16.651Z [hevc_nvenc @ 0x55e10e2bb540] Driver does not support the required nvenc API version. Required: 12.0 Found: 11.1
2023-07-26T13:40:16.651Z [hevc_nvenc @ 0x55e10e2bb540] The minimum required Nvidia driver for nvenc is 520.56.06 or newer

The version showing in nvidia-smi is 515.65.01. Seems to be related.

[Edit: Can confirm that running 2.00.20.1 docker image "fixes" this issue, or at least allows me to transcode using GPU again.]

[HaveAGitGat]

@triestelaporte yes you need to update your host nvidia drivers for FFmpeg 6 which is what Tdarr is using.

[triestelaporte]

Sadly that is not a realistic option for most users of truenas. I do hope ix updates the drivers in truenas though.

[TristanHarms]

For now, it's actually "solved" the other way around. the Tdarr chart's image was rolled back to ensure the latest version of the Tdarr chart is usable for TrueNAS Scale users. The image version was also pinned to make sure it doesn't break until IX updates the NVIDIA drivers. The issue of the outdated NVIDIA drivers is known with IX, so hopefully they'll update them soon 🤞 .

This means that you should be able to safely update to the latest version of the chart.

[HaveAGitGat]

Added acceptance containers with ffmpeg 5:

haveagitgat/tdarr_acc:dev_2.12.01_2023_09_29T07_43_49z_ffmpeg5
haveagitgat/tdarr_node_acc:dev_2.12.01_2023_09_29T05_47_32z_ffmpeg5

Will be included alongside next release