Feature/remove timm clip flags

[jain18ayush]

reconfigured the flags for #98 based off the transformer configs

[themachinefan]

It's messy but is_clip actually still loads from hugging face transformer library, we don't use the clip library.
Based on the following code it seems like maybe all you need to do is check if AutoConfig has an attribute 'vision_config'

elif is_clip:  # Extract vision encoder from dual-encoder CLIP model.
    hf_config = AutoConfig.from_pretrained(model_name).vision_config

[jain18ayush]

Ok. Thanks for the tip!

[lowlorenz]

What is the status of this PR? @soniajoseph is there something preventing us from merging this?

[soniajoseph]

Apologies for the delay here and thank you @jain18ayush for the code!

We've moved largely to openclip models so it's not clear to me that the list-matching works here. This is something to adapt as soon as Karolis pushes the OpenCLIP adaptation code which should come with a list of models to check. I'll tag him here as soon as he accepts my invite to this repo.