Legacy `sws_scale()` prevents multi-threaded color conversion

[mscheltienne]

🚀 The feature

Following up on #1215, the current implementation of sws_scale is a bottleneck when decoding 4K videos. On the plot below, you can see that decoding is capped at ~20ms regardless of the number of threads and seeking mode of the decoder (CPU decoding).

After some investigation, it turns out the YUV -> RGB conversion is dominating the decoding time and is very consistent. torchcodec uses the legacy sws_scale() FFmpeg function for YUV→RGB color conversion. This function is always single-threaded. FFmpeg 7+ introduced sws_scale_frame() with a threads option on SwsContext that parallelizes the conversion across CPU cores. PyAV and the ffmpeg CLI already use this new API, achieving roughly 2x faster conversion on 4K content.

Motivation, pitch

No response

[mollyxu]

Hi @mscheltienne, thanks for flagging this and for running the benchmarks! This is something we could potentially consider, here's how we're thinking about it at the moment: While the speedups with sws_scale_frame + the threads option are great, we try to avoid backend specific parameters as much as possible. Enabling a threads option for sws_scale_frame forces us to expose another backend specific parameter to the user in VideoDecoder. Additionally, these benchmarks are decoding a single video, but might not be applicable to decoding multiple videos in parallel. Let us know what your thoughts are on this!