Investigate value differences between our `ConfusionMatrix` and YOLOv8 implementation

[SkalskiP]

Search before asking

• I have searched the Supervision issues and found no similar bug report.

Bug

We recently added object detection ConfusionMatrix to supervision. However, we noticed that our ConfusionMatrix value tends to differ from the one you get from YOLOv8. We would like to understand why that is happening. And fix any potential bugs in our codebase. Heda more here.

[hardikdava]

@SkalskiP I found that yolov8 has prediction on x-axis. I hope there is no such bug in reverse calculation.

[SkalskiP]

Hi, @hardikdava 👋🏻! Yeah, I know the result matrix is transposed, but that's not the issue. :) But even if you consider that, there are still some differences. No worries, we will figure it out. 👍🏻

[hardikdava]

@SkalskiP I think I have found the issue. There are two issue.

• Somehow supervision is ignoring while loading dataset.
• Yolov8 is adding is also adding penalty when there is no detections found check here. supervision is missing that.

[SkalskiP]

• Somehow supervision is ignoring while loading dataset.

I'm not sure I understand this part. :)

[hardikdava]

@SkalskiP I am investigating the issue with coco128 dataset from yolov8 but it contains 2 background images means there is not objects. Our dataset mechanism sv.DetectionDataset.from_yolo() is loading 126 images. So there is a bug in dataset loading.

[SkalskiP]

@hardikdava Oh, interesting. So if there are no detections, we are dropping the image from the dataset instead of loading sv.Dataset.empty()?

[hardikdava]

@hardikdava Oh, interesting. So if there are no detections, we are dropping the image from the dataset instead of loading sv.Dataset.empty()?

Yeah, also the dataset has issue. I found that 1 image is present but relative annotation file is missing, also 1 annotation file is present and relative image is missing. 😆

[hardikdava]

@SkalskiP Summary of my experiments:

• Original implementation of ConfusionMatrix is adopted from here.
• sv.ConfusionMatrix == kaanakan.ConfusionMatrix == onemetric.ConfusionMatrix
• yolov8 is adding penalties extra check here

I do not see any other difference in the implementation.

[SkalskiP]

Excellent work, @hardikdava! Thanks a lot for the time you spend investigating the issue. So the takeaways are:

• We are pretty sure our implementation is correct.
• We understand where the difference is coming from.

[hardikdava]

@SkalskiP Please convert this issue as Discussion for future references.