Will this work for ZeroShotClassification?

[premthomas]

Hello Charles,

I used the example from your medium article with one change. On replacing the DistilBert model with the Bart model, I get an error that the attribute does not exist. The model I am trying to use is "facebook/bart-large-mnli".

File "C:\TextClassificationV5\lib\site-packages\transformers_interpret\explainers\sequence_classification.py", line 129, in _calculate_attributions
attribute = getattr(self.model, self.model_type)
File "C:\TextClassificationV5\lib\site-packages\torch\nn\modules\module.py", line 779, in __getattr__
type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'BartForSequenceClassification' object has no attribute 'bart'"

The model name is BartForSequenceClassification
And the model_type is Bart

Have I missed something?

[cdpierse]

Hi Prem,

Thanks for bringing this to my attention. I've just taken a look at this and have a few notes about the issue:

• The ZeroShotClassification model although it is an instance of BartForSequenceClassification won't work like regular classification models as the model does some custom work with selecting labels - see the model card's description of manually getting prediction labels for an example of this.

• The model you mention is for Natural Language Inference (NLI) and the Facebook/Huggingface team do some interesting tweaks of it's logic to make it function as zero shot classification. It's certainly a custom behavior I could and would be interested in building in eventually. I've a couple of features and pieces of cleanup on the regular classification explainer to do first, but I'm going to leave this issue open so that I eventually give this a proper look.

• Funnily enough the specific issue you ran into here occurred for a different reason and is something I've started to remedy with Replace model name with base model prefix #20 which is that the model_name was not uniform across all Huggingface models. For instance, Bert and Distilbert models have attribute signatures like my_bert_model.bert and my_distilbert_model.distilbert that allow access to the modules etc. Encoder & Decoder models in the Huggingface library seem to behave differently, Bart models for instance look like my_bart_model.model. So when I tried to access the model by that attribute name it would naturally not work.

• I'm currently working on a fix for Use model.embeddings.word_embeddings as in LIG  #14 that should allow support for normal BartForSequenceClassification models by using the same method call for all models to get just the input (word) embeddings. This does change the attributions somewhat in some cases though so I'm trying to make sure I understand fully why they are different before I implement it.

• Once I have that issue fixed you should at least be able to use Bart models like textattack/facebook-bart-large-MNLI which outputs whether a premise and hypothesis are a Contradiction, Neutral or Entailment to one another.

Thanks again for pointing this out, as you can see there are a couple of things getting in the way of being able to do this specific use case right now, but it is certainly something I hope I can get to!
Charles

[lalitpagaria]

First I would like to thank you for awesome work.
I am also looking forward for this work and planning to integrate with https://github.com/lalitpagaria/obsei

I have one query what will happen if I modify model id2label and label2id config before initialisation of explainer?
Like this

model.config.id2label = {0:"politics", 1:"public health", 2:"economics"}
model.config.label2id = {"politics":0, "public health":1, "economics":2}

Is this correct way to use it for zero shot predictions?

Also I suspect memory leak some where, as when ever I run attributions = cls_explainer(text) on colab, RAM usage increase and notebook crash eventually after sometime.

[lalitpagaria]

@cdpierse Just checking about if you need any help in this task. At least I can support in testing it :)

[cdpierse]

@lalitpagaria I haven't started on the zero shot explainer just yet, but I should be able to get a start on it soon as I have just finished up witht the question answering explainer.

[cdpierse]

Just a quick update on this, I've started progress on this explainer and have figured out how to make it work I think, hoping to have a PR together in the next week or two, the explainer will likely end up being a very slight modification of the classification explainer.

[premthomas]

Hi Charles,

First, thank you for the detailed reply to the query I had posted on the ZeroShot Classification.

@lalitpagaria On initial testing, yes. On populating just the id2label and label2id fields with the labels that you are passing, there are some results being populated.

However, there are still some further tweaks that are required.

1. There is a limit on the number of words. On passing larger paragraphs with a set of sentence-based classes, I am getting memory issues.
2. On reducing the number of words, the selected class is not the same as that of the class suggested by the ZeroShot Classifier. Obviously, since the number of parameters is reduced, we should expect the classes to be different and now I am trying to reduce the number of words in my input to test if I get the same results.

Thank you so much, Charles. Looking forward to hearing from you. And I can also help with some testing if you so require.

[cdpierse]

Hi everyone,

So I have the bones of a basic implementation done, but still have one or two design decisions to make so I've started a discussion over at #40 if you would like to participate I'd really love to hear your thoughts.

[lalitpagaria]

Thank you @cdpierse

Apologies for not replying earlier, I am currently stuck with some task.

I will certainly check this out and share feedback.