RFDETR CrowdHuman fine tuning

[frittatelle]

Hi!
I'm trying to fine tune a RFDETRSmall model on the CrowdHuman dataset that contains a bunch of overlapping boxes and a high density of detections in each image of the dataset.
The dataset is quite big (15000 images for training, ~3000 val, ~3000 test) and I'm trying to find the best parameters for fine tuning the model.
This is my simple current setup (RTX3090).

    # model
    model = RFDETRSmall(
        pretrain_weights="checkpoints/rf-detr-small-topdown-hearty-armadillo-5.pth"
    )

    model.train(
        dataset_dir=dataset_dir,
        epochs=30,
        batch_size=8,
        grad_accum_steps=4,
        lr=5e-5,
        patience=10,
        early_stopping=True,
        output_dir=output_dir,
        wandb=True,
    )

The final goal would be to distinguish hugged people for a security application. The pretrained RFDETRSmall often fails this task and classifies hugged people as a single blob, therefore I'm trying to get better results by training on a high density detection dataset.
The problem I'm facing is that the loss is basically flat and the cardinality error which is my most important metric here is never going down.
The following is a snippet of the logs of the last training experiment.

Epoch: [6]  [130/375]  eta: 0:02:57  lr: 0.000020  class_error: 0.00  loss: 5.8841 (6.2072)  loss_ce: 0.6394 (0.6561)  loss_bbox: 0.1946 (0.2135)  loss_giou: 0.5716 (0.6327)  loss_ce_0: 0.6761 (0.6853)  loss_bbox_0: 0.2103 (0.2254)  loss_giou_0: 0.5967 (0.6541)  loss_ce_1: 0.6477 (0.6630)  loss_bbox_1: 0.1962 (0.2170)  loss_giou_1: 0.5819 (0.6404)  loss_ce_enc: 0.6746 (0.6771)  loss_bbox_enc: 0.2334 (0.2524)  loss_giou_enc: 0.6250 (0.6903)  loss_ce_unscaled: 0.6394 (0.6561)  class_error_unscaled: 0.0000 (0.0000)  loss_bbox_unscaled: 0.0389 (0.0427)  loss_giou_unscaled: 0.2858 (0.3163)  cardinality_error_unscaled: 24.5000 (27.5401)  loss_ce_0_unscaled: 0.6761 (0.6853)  loss_bbox_0_unscaled: 0.0421 (0.0451)  loss_giou_0_unscaled: 0.2983 (0.3271)  cardinality_error_0_unscaled: 24.5000 (27.5401)  loss_ce_1_unscaled: 0.6477 (0.6630)  loss_bbox_1_unscaled: 0.0392 (0.0434)  loss_giou_1_unscaled: 0.2909 (0.3202)  cardinality_error_1_unscaled: 24.5000 (27.5401)  loss_ce_enc_unscaled: 0.6746 (0.6771)  loss_bbox_enc_unscaled: 0.0467 (0.0505)  loss_giou_enc_unscaled: 0.3125 (0.3451)  cardinality_error_enc_unscaled: 24.5000 (27.5401)  time: 0.7002  data: 0.0175  max mem: 18269
2026-02-12 11:36:35

Epoch: [6]  [140/375]  eta: 0:02:47  lr: 0.000020  class_error: 0.00  loss: 5.8714 (6.1737)  loss_ce: 0.6487 (0.6544)  loss_bbox: 0.1979 (0.2141)  loss_giou: 0.5658 (0.6250)  loss_ce_0: 0.6803 (0.6841)  loss_bbox_0: 0.2095 (0.2260)  loss_giou_0: 0.5844 (0.6460)  loss_ce_1: 0.6527 (0.6615)  loss_bbox_1: 0.2055 (0.2178)  loss_giou_1: 0.5665 (0.6326)  loss_ce_enc: 0.6733 (0.6764)  loss_bbox_enc: 0.2305 (0.2533)  loss_giou_enc: 0.6226 (0.6825)  loss_ce_unscaled: 0.6487 (0.6544)  class_error_unscaled: 0.0000 (0.0000)  loss_bbox_unscaled: 0.0396 (0.0428)  loss_giou_unscaled: 0.2829 (0.3125)  cardinality_error_unscaled: 22.5000 (26.9663)  loss_ce_0_unscaled: 0.6803 (0.6841)  loss_bbox_0_unscaled: 0.0419 (0.0452)  loss_giou_0_unscaled: 0.2922 (0.3230)  cardinality_error_0_unscaled: 22.5000 (26.9663)  loss_ce_1_unscaled: 0.6527 (0.6615)  loss_bbox_1_unscaled: 0.0411 (0.0436)  loss_giou_1_unscaled: 0.2833 (0.3163)  cardinality_error_1_unscaled: 22.5000 (26.9663)  loss_ce_enc_unscaled: 0.6733 (0.6764)  loss_bbox_enc_unscaled: 0.0461 (0.0507)  loss_giou_enc_unscaled: 0.3113 (0.3413)  cardinality_error_enc_unscaled: 22.5000 (26.9663)  time: 0.6730  data: 0.0167  max mem: 18269

Do you have any suggestion on how to proceed?
Thanks in advance

[IchiruTake]

May I know how many people in the image in advance? As I deep down into the architecture, the model has removed NMS post processing and apply 300 query tokens, associated to at most 300 objects. If you want like 400 or more then a full model training with expansion of query token is better, start pretrain with Object365, RF100VL, LVD142M then perform fine tuning

Otherwise you can pick YOLO (not v10 as YOLOv10 remove post processing) which allows at most 8400 instances per class.

However if the maximum expecting number is around 80 to 150, then I think you need to prepare the dataset having hundred of person in single image. As most dataset dont expect 20-30 person in a single image (unlike supermarket CCTV)

Hope it helps

[isaacrob]

You probably need a larger RF-DETR version than small for your use case, especially since the larger versions use a higher resolution which might be better for distinguishing highly overlapping small people. Are you latency constrained?

Also, why are you considering cardinality error your most important metric? It's not something that gets explicitly optimized.

[frittatelle]

Thank you for the responses @isaacrob @IchiruTake!
I'll give you a little bit more context. 100 detections is absolutely fine, the only dataset that has a high density of overlapping boxes is the crowduman one (~22 detections/image), I thought this was my best shot to have a model that identifies two people hugging / person hiding behind someone else as two people.
The final HW target is a Jetson orin nano that also runs other models so yes, @isaacrob latency is a constrain. I can still try with a larger model as an experiment to see if I get better results.
I was considering cardinality error to be my most important metric because for my application low precision is as problematic as low recall, therefore seeing how wrong is the model at guessing the number of objects in an image is a big deal. On the other hand I don't really care about box accuracy.

[frittatelle]

The following are the results of my last training on RFDETRSmall using the parameters I posted before.

I'll try with a bigger size model in the hope of getting something out of it. These metrics are still really poor

[isaacrob]

What's the state of the art for this dataset?

[frittatelle]

I'm using the Roboflow version here:

https://universe.roboflow.com/crowd-human/all-chr4t

The links on the official site are broken

https://www.crowdhuman.org/

[IchiruTake]

You can first try base Object365 version, and Medium pretrained model first. If both not help, with latency constrained then you may want to annotate data more. Find COCO or similar human dataset, add extra bounding box help about person head, and defined an "estimated" head with some shoulders visible to be labels as "human"

In CrowdHuman dataset, if you want to stick with that, then most of the time you can only see a person head. For benchmarking, I think it is OK to try though I think that you may need to label more on the image; but most of the time crowded counting is hard to count, track and measure right on downstream tasks

[frittatelle]

I'll try to move to a bigger model and possibly make custom annotations if is not enough.

[frittatelle]

I naively tried to finetune a RFDETRLarge without worrying too much about latency for now to understand how far can I push my model and I got a better result in terms of AP50, but overall the performance is still very poor:

{
  "Loss/Test": 5.259323565444198,
  "Loss/Train": 4.886974736397213,
  "Metrics/Base/AP50": 0.720079527803547,
  "Metrics/Base/AP50_90": 0.4674453524568765,
  "Metrics/Base/AR50_90": 0.5306972847865138,
  "Metrics/EMA/AP50": 0.7237014393616996,
  "Metrics/EMA/AP50_90": 0.47315914767673056,
  "Metrics/EMA/AR50_90": 0.5346111993920133,
  "_runtime": 54594,
  "_step": 30,
  "_timestamp": 1771168692.889844,
  "_wandb.runtime": 54594,
  "epoch": 30
}

For my application, what I really need to detect is people hugging or trying to hide. Maybe the way to go is actually making a dataset with custom classes as hugging, hiding, crouching etc instead of relying on the simple "person" class.
Do you think this will be more effective or should I stick to the generic person class and add some head annotations as suggested by @IchiruTake?

[IchiruTake]

For Object Detection, RFDETR does not learn much about human "specific" action and the effort spent on labeling those actions as classes may not give you as many benefits due to tremendous labeling effort. Reason: The pre-trained version is "object detection", learning about color, and object structure, not about human action interaction or so. If you really want to take a try to push to limit, then initialized the weight from random and train from it instead of finetuning on a pretrained model.

The normal strategy is to label some person "head", change the class ID of "head" into "person". For action such as hugging or crouching (like CCTV on retail/store, you can do the same but with a portion of the human body (like right shoulder to right leg), and estimate the "human" bounding box size (I don't apply when people is "hiding" or being occuluded). For downstream task, you may want to apply a cached detection with FPS of 8-10 FPS on 20-30 FPS camera scene, with a customized ByteTrack version to adjust the track_buffer and confidence on-the-fly to re-count person, and apply line/zone counting.

If you have the target "video scene" or camera scene (RTSP from supermarket, let's say), then you can ask and collect the data legally at different time of the day. Use LabelStudio, FiftyOne, or any labeling tool that can support AI detection with ByteTrack, YOLO, and doing some labeling work.

[IchiruTake]

For my application, what I really need to detect is people hugging or trying to hide. Maybe the way to go is actually making a dataset with custom classes as hugging, hiding, crouching etc instead of relying on the simple "person" class.

Is your task "action recognition" or "people counting"? For the latter one, then if after you re-worked the dataset with adjusted annotation, and result is OK then fine, but for the first one, some easiest option is go with YOLOv8-pose (which included person detection with keypoint), and apply keypoint-based action recognition; or a DETRPose for pure human keypoint generation (you can prune the classification head and bounding box head of YOLO to minimize training and inference overhead, though I never try this)

[isaacrob]

@frittatelle what is the best known result on this dataset, ie what is state of the art? You need to understand how your target compares against the published best and what latency is required to accomplish that best. This is a common dataset, it is very likely there are specific published results somewhere.

The pretraining should be good for finding 'hugging' or something of the like, no need to think about random weights. How were you planning on using this dataset to determine hugging or hiding? If nontrivial post processing is involved, it's often easier to try to determine the goal behavior directly.