Object Detection Confusion Matrix - Evaluation API

[SkalskiP]

Description

Let's start our work on Evaluation API with sv.ConfusionMatrix. Here is the expected API:

@dataclass
class ConfusionMatrix:
    matrix: np.ndarray
    classes: List[str]
    conf_threshold: float
    iou_threshold: float

    @classmethod
    def from_detections(
        cls,
        predictions: List[sv.Detections],
        target: List[sv.Detections],
        classes: List[str],
        conf_threshold: float = 0.3,
        iou_threshold: float = 0.5
    ) -> ConfusionMatrix:
        pass

   @classmethod
    def benchmark(
        cls,
        dataset: sv.DetectionDataset,
        callback: Callable[[np.ndarray], sv.Detections],
        conf_threshold: float = 0.3,
        iou_threshold: float = 0.5
    ) -> ConfusionMatrix:
        pass

    def plot(self, target_path: str) -> None:
        pass

Usage example

from_detections

>>> import supervision as sv
>>> from ultralytics import YOLO

>>> model = YOLO('yolov8s.pt')

>>> dataset = sv.DetectionDataset.from_yolo(
...     images_directory_path='...',
...     annotations_directory_path='...',
...     data_yaml_path='...'
... )

>>> predictions, target = [], []

>>> for _, image, labels in dataset:
...    result = model(image)[0]
...    detections = sv.Detections.from_yolov8(result)
...    predictions.append(detections)
...    target.append(labels)

>>> matrix = ConfusionMatrix.from_detections(
...     predictions=predictions,
...     target=target,
...     classes=dataset.classes
... )
>>> matrix.plot('...')

benchmark

>>> import supervision as sv
>>> from ultralytics import YOLO

>>> model = YOLO('yolov8s.pt')

>>> dataset = sv.DetectionDataset.from_yolo(
...     images_directory_path='...',
...     annotations_directory_path='...',
...     data_yaml_path='...'
... )

>>> def wrapper(image: np.ndarray) -> sv.Detections:
...     result = model(image)[0]
...     return sv.Detections.from_yolov8(result)

>>> matrix = ConfusionMatrix.benchmark(
...     dataset=dataset,
...     callback=wrapper
... )
>>> matrix.plot('...')

Additional

• Code should live in supervision.metrics.detection
• We can use code in onemetric pip package as starting point
• We shouldn't need any external dependencies. Only numpy and matplotlib.

[kirilllzaitsev]

@SkalskiP, @capjamesg, hi, are you working on this feature at the moment? If not, I'd like to give it a shot.

[SkalskiP]

Hi @kirilllzaitsev 👋🏻 I guess no one started that line of work yet. Feel free to submit PR with the initial implementation. Please make sure to use box_iou_batch function that is already part of supervision to make confusion matrix calculation faster.