Skip to content

Commit 6c2ee01

Browse files
nemonamelessnemonameless
authored
add swin focalnet backbone rtdetr models (#8309)
1 parent 579254f commit 6c2ee01

4 files changed

Lines changed: 199 additions & 6 deletions

File tree

‎configs/rtdetr/README.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22

33
## 最新动态
44

5+
- 发布RT-DETR-Swin和RT-DETR-FocalNet模型
56
- 发布RT-DETR-R50和RT-DETR-R101的代码和预训练模型
6-
- 发布RT-DETR-L和RT-DETR-X的代码和预训练模型
7+
- **发布RT-DETR-L和RT-DETR-X的代码和预训练模型**
78
- 发布RT-DETR-R50-m模型（scale模型的范例）
89
- 发布RT-DETR-R34模型
910
- 发布RT-DETR-R18模型
@@ -17,7 +18,7 @@ RT-DETR是第一个实时端到端目标检测器。具体而言，我们设计
1718
<img src="https://github.com/PaddlePaddle/PaddleDetection/assets/17582080/3184a08e-aa4d-49cf-9079-f3695c4cc1c3" width=500 />
1819
</div>
1920

20-
## 模型
21+
## 基础模型
2122

2223
| Model | Epoch | backbone | input shape | $AP^{val}$ | $AP^{val}_{50}$| Params(M) | FLOPs(G) | T4 TensorRT FP16(FPS) | Pretrained Model | config |
2324
|:--------------:|:-----:|:----------:| :-------:|:--------------------------:|:---------------------------:|:---------:|:--------:| :---------------------: |:------------------------------------------------------------------------------------:|:-------------------------------------------:|
@@ -29,10 +30,17 @@ RT-DETR是第一个实时端到端目标检测器。具体而言，我们设计
2930
| RT-DETR-L | 6x | HGNetv2 | 640 | 53.0 | 71.6 | 32 | 110 | 114 | [download](https://bj.bcebos.com/v1/paddledet/models/rtdetr_hgnetv2_l_6x_coco.pdparams) | [config](rtdetr_hgnetv2_l_6x_coco.yml)
3031
| RT-DETR-X | 6x | HGNetv2 | 640 | 54.8 | 73.1 | 67 | 234 | 74 | [download](https://bj.bcebos.com/v1/paddledet/models/rtdetr_hgnetv2_x_6x_coco.pdparams) | [config](rtdetr_hgnetv2_x_6x_coco.yml)
3132

33+
## 高精度模型
34+
35+
| Model | Epoch | backbone | input shape | $AP^{val}$ | $AP^{val}_{50}$ | Pretrained Model | config |
36+
|:-----:|:-----:|:---------:| :---------:|:-----------:|:---------------:|:----------------:|:------:|
37+
| RT-DETR-Swin | 3x | Swin_L_384 | 640 | 56.2 | 73.5 | [download](https://bj.bcebos.com/v1/paddledet/models/rtdetr_swin_L_384_3x_coco.pdparams) | [config](./rtdetr_swin_L_384_3x_coco.yml)
38+
| RT-DETR-FocalNet | 3x | FocalNet_L_384 | 640 | 56.9 | 74.3 | [download](https://bj.bcebos.com/v1/paddledet/models/rtdetr_focalnet_L_384_3x_coco.pdparams) | [config](./rtdetr_focalnet_L_384_3x_coco.yml)
3239

3340
**注意事项:**
34-
- RT-DETR 使用4个GPU训练。
41+
- RT-DETR 基础模型均使用4个GPU训练。
3542
- RT-DETR 在COCO train2017上训练，并在val2017上评估。
43+
- 高精度模型RT-DETR-Swin和RT-DETR-FocalNet使用8个GPU训练，显存需求较高。
3644

3745
## 快速开始
3846

‎configs/rtdetr/rtdetr_focalnet_L_384_3x_coco.yml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
_BASE_: [
2+
'../datasets/coco_detection.yml',
3+
'../runtime.yml',
4+
'_base_/optimizer_6x.yml',
5+
'_base_/rtdetr_r50vd.yml',
6+
'_base_/rtdetr_reader.yml',
7+
]
8+
9+
weights: output/rtdetr_focalnet_L_384_3x_coco/model_final
10+
find_unused_parameters: True
11+
log_iter: 100
12+
snapshot_epoch: 2
13+
14+
pretrain_weights: https://bj.bcebos.com/v1/paddledet/models/pretrained/focalnet_large_fl4_pretrained_on_o365.pdparams
15+
DETR:
16+
backbone: FocalNet
17+
neck: HybridEncoder
18+
transformer: RTDETRTransformer
19+
detr_head: DINOHead
20+
post_process: DETRPostProcess
21+
22+
FocalNet:
23+
arch: 'focalnet_L_384_22k_fl4'
24+
out_indices: [1, 2, 3]
25+
26+
HybridEncoder:
27+
hidden_dim: 256
28+
use_encoder_idx: [2]
29+
num_encoder_layers: 6 #
30+
encoder_layer:
31+
name: TransformerLayer
32+
d_model: 256
33+
nhead: 8
34+
dim_feedforward: 2048
35+
dropout: 0.
36+
activation: 'gelu'
37+
expansion: 1.0
38+
39+
40+
RTDETRTransformer:
41+
num_queries: 300
42+
position_embed_type: sine
43+
feat_strides: [8, 16, 32]
44+
num_levels: 3
45+
nhead: 8
46+
num_decoder_layers: 6
47+
dim_feedforward: 2048 #
48+
dropout: 0.0
49+
activation: relu
50+
num_denoising: 100
51+
label_noise_ratio: 0.5
52+
box_noise_scale: 1.0
53+
learnt_init_query: False
54+
query_pos_head_inv_sig: True #
55+
56+
DINOHead:
57+
loss:
58+
name: DINOLoss
59+
loss_coeff: {class: 1, bbox: 5, giou: 2}
60+
aux_loss: True
61+
use_vfl: True
62+
matcher:
63+
name: HungarianMatcher
64+
matcher_coeff: {class: 2, bbox: 5, giou: 2}
65+
66+
DETRPostProcess:
67+
num_top_queries: 300
68+
69+
70+
epoch: 36
71+
LearningRate:
72+
base_lr: 0.0001
73+
schedulers:
74+
- !PiecewiseDecay
75+
gamma: 0.1
76+
milestones: [36]
77+
use_warmup: false
78+
79+
OptimizerBuilder:
80+
clip_grad_by_norm: 0.1
81+
regularizer: false
82+
optimizer:
83+
type: AdamW
84+
weight_decay: 0.0001
85+
param_groups:
86+
- params: ['absolute_pos_embed', 'relative_position_bias_table', 'norm']
87+
weight_decay: 0.0

‎configs/rtdetr/rtdetr_swin_L_384_3x_coco.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
_BASE_: [
2+
'../datasets/coco_detection.yml',
3+
'../runtime.yml',
4+
'_base_/optimizer_6x.yml',
5+
'_base_/rtdetr_r50vd.yml',
6+
'_base_/rtdetr_reader.yml',
7+
]
8+
9+
weights: output/rtdetr_swin_L_384_3x_coco/model_final
10+
find_unused_parameters: True
11+
log_iter: 100
12+
snapshot_epoch: 2
13+
14+
pretrain_weights: https://bj.bcebos.com/v1/paddledet/models/dino_swin_large_384_4scale_3x_coco.pdparams
15+
DETR:
16+
backbone: SwinTransformer
17+
neck: HybridEncoder
18+
transformer: RTDETRTransformer
19+
detr_head: DINOHead
20+
post_process: DETRPostProcess
21+
22+
23+
SwinTransformer:
24+
arch: 'swin_L_384' # ['swin_T_224', 'swin_S_224', 'swin_B_224', 'swin_L_224', 'swin_B_384', 'swin_L_384']
25+
ape: false
26+
drop_path_rate: 0.2
27+
patch_norm: true
28+
out_indices: [1, 2, 3]
29+
30+
HybridEncoder:
31+
hidden_dim: 256
32+
use_encoder_idx: [2]
33+
num_encoder_layers: 6 #
34+
encoder_layer:
35+
name: TransformerLayer
36+
d_model: 256
37+
nhead: 8
38+
dim_feedforward: 2048 #
39+
dropout: 0.
40+
activation: 'gelu'
41+
expansion: 1.0
42+
43+
RTDETRTransformer:
44+
num_queries: 300
45+
position_embed_type: sine
46+
feat_strides: [8, 16, 32]
47+
num_levels: 3
48+
nhead: 8
49+
num_decoder_layers: 6
50+
dim_feedforward: 2048 #
51+
dropout: 0.0
52+
activation: relu
53+
num_denoising: 100
54+
label_noise_ratio: 0.5
55+
box_noise_scale: 1.0
56+
learnt_init_query: False
57+
58+
DINOHead:
59+
loss:
60+
name: DINOLoss
61+
loss_coeff: {class: 1, bbox: 5, giou: 2}
62+
aux_loss: True
63+
use_vfl: True
64+
matcher:
65+
name: HungarianMatcher
66+
matcher_coeff: {class: 2, bbox: 5, giou: 2}
67+
68+
DETRPostProcess:
69+
num_top_queries: 300
70+
71+
72+
epoch: 36
73+
LearningRate:
74+
base_lr: 0.0001
75+
schedulers:
76+
- !PiecewiseDecay
77+
gamma: 0.1
78+
milestones: [36]
79+
use_warmup: false
80+
81+
OptimizerBuilder:
82+
clip_grad_by_norm: 0.1
83+
regularizer: false
84+
optimizer:
85+
type: AdamW
86+
weight_decay: 0.0001
87+
param_groups:
88+
- params: ['absolute_pos_embed', 'relative_position_bias_table', 'norm']
89+
weight_decay: 0.0

‎ppdet/modeling/transformers/rtdetr_transformer.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -218,14 +218,19 @@ def forward(self,
218218
score_head,
219219
query_pos_head,
220220
attn_mask=None,
221-
memory_mask=None):
221+
memory_mask=None,
222+
query_pos_head_inv_sig=False):
222223
output = tgt
223224
dec_out_bboxes = []
224225
dec_out_logits = []
225226
ref_points_detach = F.sigmoid(ref_points_unact)
226227
for i, layer in enumerate(self.layers):
227228
ref_points_input = ref_points_detach.unsqueeze(2)
228-
query_pos_embed = query_pos_head(ref_points_detach)
229+
if not query_pos_head_inv_sig:
230+
query_pos_embed = query_pos_head(ref_points_detach)
231+
else:
232+
query_pos_embed = query_pos_head(
233+
inverse_sigmoid(ref_points_detach))
229234

230235
output = layer(output, ref_points_input, memory,
231236
memory_spatial_shapes, memory_level_start_index,
@@ -276,6 +281,7 @@ def __init__(self,
276281
label_noise_ratio=0.5,
277282
box_noise_scale=1.0,
278283
learnt_init_query=True,
284+
query_pos_head_inv_sig=False,
279285
eval_size=None,
280286
eval_idx=-1,
281287
eps=1e-2):
@@ -321,6 +327,7 @@ def __init__(self,
321327
if learnt_init_query:
322328
self.tgt_embed = nn.Embedding(num_queries, hidden_dim)
323329
self.query_pos_head = MLP(4, 2 * hidden_dim, hidden_dim, num_layers=2)
330+
self.query_pos_head_inv_sig = query_pos_head_inv_sig
324331

325332
# encoder head
326333
self.enc_output = nn.Sequential(
@@ -464,7 +471,9 @@ def forward(self, feats, pad_mask=None, gt_meta=None):
464471
self.dec_bbox_head,
465472
self.dec_score_head,
466473
self.query_pos_head,
467-
attn_mask=attn_mask)
474+
attn_mask=attn_mask,
475+
memory_mask=None,
476+
query_pos_head_inv_sig=self.query_pos_head_inv_sig)
468477
return (out_bboxes, out_logits, enc_topk_bboxes, enc_topk_logits,
469478
dn_meta)
470479

0 commit comments

Comments
 (0)