[Need feedback] What dtype do we output for HDR videos?

[NicolasHug]

Opening this to gather feedback from https://x.com/hug_nicolas/status/2036138642360439080

We currently return torch.uint8 which is lossy. If we were to support native resolution for HDR videos, what should we output?

• torch.uint16. Users can't do anything with a uint16 tensor, the only thing they can do is to immediately convert to int32 or float32.
• torch.int32
• torch.float32

[bhack]

We could still go to float16 or bfloat16 from uint16.
E.g. what kind of precision needs torch Vision transforms on 10bit 12bit HDR?

[NicolasHug]

TorchVision transforms are agnostic to the video source, but they expect either uint8 images in [0, 255], or float images in [0, 1] (float32, or float64).

[luckyspacetraveller]

Hi there :)

I hope you guys don't mind if I share my perspective here.

I would recommend going float32. At least that's what we are doing in the world of VFX. If the data would be linear (gamma curve), float16 could be enough for transferring color information, but as soon as you start doing operations on it, 32 is the way to go and you would convert again. And as most videos are of course encoded in some other gamma than linear/1, you could loose precision, with float16. Especially when the video is encoded in some log colorspace, which some phone apps can do.

I can imagine, and I think that is the goal here, rather we give full quality with the decode and the user decides to limit it (like converting to float16) than having again sth that reduces precision while decoding already. :)