VIT Adapter

[aselimc]

Hello,

I propose adding ViT adapter option in TimmUniversalEncoder for ViTs that does not have multi-scale features, so that SMP can now support a whole new set of backbones. The following links can be used as a reference/starting point.

DINOv3_Adapter
DINOv2_Adapter
Original VIT Adapter Repo
AdaptersHub
Adapters HF Version

Or perhaps these type of adapters might actually better be on timm? @rwightman @qubvel

[ogencoglu]

+1

[ha405]

Hi @aselimc @ogencoglu 👋

I’d like to take this up and work on adding ViT adapter support in TimmUniversalEncoder. Before I begin, could you please confirm the intended integration point: should the adapter layer live within the TimmUniversalEncoder itself (as an internal submodule), or be attached externally similar to how the DINOv2_Adapter repo structures it?

Thanks for opening this.

[ogencoglu]

Any preferences @qubvel ?

[aselimc]

I agree this is a design decision in which project owner @qubvel should comment on it.