BERT :dropout(): argument 'input' (position 1) must be Tensor, not str

Question

I am new to NLP and would like to build a BERT model for sentiment analysis so I am following this tutorial.

However, I am getting the error below:

F.softmax(model(input_ids, attention_mask), dim = 1)

When I would like to execute this cell I get the error:

 dropout(): argument 'input' (position 1) must be Tensor, not str

Answer 1

How to obtain the same behavior as v3.x in v4.x In order to obtain the same behavior as version v3.x, you should install sentencepiece additionally:

In version v3.x:

pip install transformers

to obtain the same in version v4.x:

pip install transformers[sentencepiece] or

pip install transformers sentencepiece

Answer 2

Which transformer version are you using? I had to pin mine to transformer == 3.5.1 to mitigate that problem, when the hugging face team updated their transformer to 4.0 things started to break.

Hope it helps

Answer 3

The transformers library uses complex output objects instead of plain tuples as return type since one of the updates after 3.5.1.:

from transformers import BertModel, BertTokenizer

t = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

o = t.encode_plus('this is a sample sentence', return_tensors='pt')

mo= model(**o)
print(type(mo))
print(mo.keys())

Output:

<class 'transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions'>
odict_keys(['last_hidden_state', 'pooler_output'])

You have to change your Sentimentclassifier to either return to the previous behavior by specifying return_dict=False or use the BaseModelOutputWithPoolingAndCrossAttentions (recommended):

class SentimentClassifier(nn.Module):
  def __init__(self, n_classes):
    super(SentimentClassifier, self).__init__()
    self.bert = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)
    self.drop = nn.Dropout(p=0.3)
    self.out = nn.Linear(self.bert.config.hidden_size, n_classes)

  def forward(self, input_ids, attention_mask):
    bertOutput = self.bert(
      input_ids=input_ids,
      attention_mask=attention_mask
    )
    output = self.drop(bertOutput['pooler_output'])

    return self.out(output)

Please have a look at this stackoverflow post in case you do not want to use my recommended solution.