Similarity of words using BERTMODEL

Question

I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following:

from transformers import BertModel
hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

where

token_ids with Shape : [1, 4, 47]
attn_mask with Shape : [1, 4, 47]
seg_ids with Shape : [1, 4, 47]

but I have an error :

TypeError                                 Traceback (most recent call last)
<ipython-input-74-5fa632122cc7> in <module>()
      1 from transformers import BertModel
----> 2 hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

TypeError: __init__() got an unexpected keyword argument 'attention_mask'

How can I fix this error??

and how can I find the word embedding and the similarity of a given word using this model?

I could not find any tutorials or similar codes for this task.

Answer 1

First of all, I think you are confused with pretrained and finetuned.

BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text.

BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other datasets : the knowledge does not apply.

You have a custom dataset. So you don't want to use a finetuned model, but you want to still use a pretrained model (to have access to the general knowledge of BERT).

If you use a BERT model from scratch (not pretrained), you're basically creating an empty model, randomly initialized : it's useless.

---

Now that we have clarified this, let's see your problem.

What you are doing is calling the constructor of BertModel with the arguments of the forward pass. You first need to create the model, and then use the forward pass :

from transformers import BertModel, BertConfig
config = BertConfig(...)    # Here your parameters to initialize your BERT as you want, such as number of heads, etc...
model = BertModel(config)
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

---

BUT, as I explained in the first part, you should not do this : it's completely useless and your results are going to be random. You want to use a pretrained BERT, to have some meaningful results :

from transformers import BertModel
model = BertModel.from_pretrained('bert-base-uncased')
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

Note : Your results can be improved further if you finetune your BERT model on your custom dataset.

---

Final point : Don't reinvent the wheel

People already tried to use BERT for word similarity. Instead of implementing this from scratch, using only a pretrained model, potentially adding bug to your own implementation, just use some already existing code !

I'm not sure about NER, but for word similarity, you can take a look at BERT Score. Their model is already finetuned for word similarity, so it's likely that you will get higher score using this approach, and also much easier path to take.

Answer 2

If obtaining word similarity is your main goal, I would suggest you to look at Word2Vec from genism.

You can even train a simple network on your corpus and fetch the most similar word(vectors).