Feature proposal: unsupervised manifold learning to understand un-labelled datasets ?

[ltetrel]

nrtk currently helps to evaluate the robustness of existing models or algorithms. The goal would be to expend its functionality to enable insights from datasets alone.
Using unsupervised manifold learning, one can automatically extract insights from an un-annotated dataset by projecting the n-dimensionnal latent space to a 3D/2D reduce space (with PCA, UMAP or T-SNE as you are already doing).

The additionnal features for nrtk would be:

1. Select an "embedding model" such as a convolutionnal VAE, or classical non-reference image quality like opencv BRISQUE.
2. Train the model if required (for ML-based techniques like CVAE)
3. Infer the model on an un-labelled dataset (optionally with augmentation) (not sure if nrtk support no annotations)
4. Visualize the features on the new space

We are currently working on this topic with ifremer, you can find a first POC here. The current visualization is really basic with a static file (html) exported from bokeh.

Let us know what you think,

[brianhhu]

Hi @ltetrel, thanks for your input! I wanted to point you to our corresponding trame-based tool, nrtk-explorer, which has a lot of these features already. There is the ability to ingest and visualize low-dimensional embeddings of datasets, perform model inference on those datasets, and also export the results. We also provide the ability to apply NRTK-provided perturbations and visualize the impact on model performance.

I don't think I have access to the Gitlab repo, but it might make sense to schedule a call to better understand your exact needs. We have a small amount of scope on our current effort to build out workflows that leverage NRTK and enable model robustness and dataset analyses. Thanks~

cc'ing @alesgenova @jourdain @stephencrowell

[ltetrel]

Thanks Brian,

low-dimensional embeddings of datasets

So basically you are using existing pre-trained models to extract those if I understand correctly ?

[brianhhu]

Yes, I believe we use features provided by ImageNet pre-trained models using the timm package (see docs here). As these models are trained on natural images, they form a good starting point for most use cases, but domain-specific features may work better depending on how different your datasets are from natural images.