I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following:

from transformers import BertModel
hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)


token_ids with Shape : [1, 4, 47]
attn_mask with Shape : [1, 4, 47]
seg_ids with Shape : [1, 4, 47]

but I have an error :

TypeError                                 Traceback (most recent call last)
<ipython-input-74-5fa632122cc7> in <module>()
      1 from transformers import BertModel
----> 2 hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

TypeError: __init__() got an unexpected keyword argument 'attention_mask'

How can I fix this error??

and how can I find the word embedding and the similarity of a given word using this model?

I could not find any tutorials or similar codes for this task.

  • $\begingroup$ note that BERT is designed to assign embeddings to words in the context while Word2Vec or GloVe doesn't need a context. $\endgroup$ Nov 2, 2021 at 13:27

2 Answers 2


First of all, I think you are confused with pretrained and finetuned.

BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text.

BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other datasets : the knowledge does not apply.

You have a custom dataset. So you don't want to use a finetuned model, but you want to still use a pretrained model (to have access to the general knowledge of BERT).

If you use a BERT model from scratch (not pretrained), you're basically creating an empty model, randomly initialized : it's useless.

Now that we have clarified this, let's see your problem.

What you are doing is calling the constructor of BertModel with the arguments of the forward pass. You first need to create the model, and then use the forward pass :

from transformers import BertModel, BertConfig
config = BertConfig(...)    # Here your parameters to initialize your BERT as you want, such as number of heads, etc...
model = BertModel(config)
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

BUT, as I explained in the first part, you should not do this : it's completely useless and your results are going to be random. You want to use a pretrained BERT, to have some meaningful results :

from transformers import BertModel
model = BertModel.from_pretrained('bert-base-uncased')
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

Note : Your results can be improved further if you finetune your BERT model on your custom dataset.

Final point : Don't reinvent the wheel

People already tried to use BERT for word similarity. Instead of implementing this from scratch, using only a pretrained model, potentially adding bug to your own implementation, just use some already existing code !

I'm not sure about NER, but for word similarity, you can take a look at BERT Score. Their model is already finetuned for word similarity, so it's likely that you will get higher score using this approach, and also much easier path to take.

  • $\begingroup$ Can you elaborate on how to use BERT Score for work similarity? Based on what I read, it seems BERT Score only can be used for sentence similarity. $\endgroup$
    – etang
    Jan 30, 2021 at 7:02

If obtaining word similarity is your main goal, I would suggest you to look at Word2Vec from genism.

You can even train a simple network on your corpus and fetch the most similar word(vectors).

  • $\begingroup$ I want to see the result in BERT, and then make a comparison among some models. $\endgroup$
    – AFB
    Nov 16, 2019 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.