# How to JUST represent words as embeddings by pretrained BERT?

I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.

After the representation of words I want to clusterize it, find similar vectors (i.e. words). Maybe even then make a classification to a given classes (classification there unsupervised --- since I don't have labeled data to train on).

I know that almost any task can be solved "inside" BERT, i.e. using fine-tuning in final layer.

Since all described above, I have two QUESTIONS; answers/hints/anything really appreciated since i'm stuck on that:

1. How to just extract embeddings from BERT using some dictionary of words and use word representations for futher work?
2. Can we solve inside BERT using fine-tuning the next problem: a). Load dictionary of words into BERT b). Load given classes (words representing each class. E.g. "fashion", "nature"). c) Make an unsupervised classification task?
– noe
Apr 14, 2021 at 6:40
• @noe that means that I have classes labels. But instead of standard classification, we're not given labeled data for training. In supervised case we would then make error function, (LogLoss) to maximize posterior probabilities taking derivatives and renewing our parameters. LogLoss uses given labeled data in $y_i$. So this is supervised classification -- using real labels to train on. In my case there is only labels of classes, to which we need to split the data Apr 14, 2021 at 8:37
• I understand that you want to use BERT do word tagging, not sentence classification, right?
– noe
Apr 14, 2021 at 18:06
• @noe, yes, I simply don't have sentences. Or we can assume that I have sentences with volume equal to one (one word in each sentence). Apr 14, 2021 at 18:20

1. BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.

2. You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-speech (POS) (e.g. this) tagging or named entity recognition (NER) (e.g. this). Normally, you associate the tag to either the first or the last subword token in the word. If you prepare a dataset that way, you could fine-tune BERT to perform word tagging with the classes you need. If you only have the words, you could find some text corpus (ideally of the intended domains) and apply the described data preparation process.

• Thanks for response, but searching for 4k datasets won't be a good case I think... Maybe I could find some dataset with images (that's what I have --- dictionary of images labels) from some social network, I could try that approach. But since I can't use Instagram DS (you answered this question if you can remember), I don't know what to do. On google's service for datasets searching I didn't find anything related and valid for my task. Apr 16, 2021 at 16:54
• excuse me i got from your answer that BERT works on sub-words, not Word. so the method for concatenating the last four layers of BERT will represent sub-words ? Jan 14 at 4:16
• @user5520049 yes
– noe
Jan 14 at 9:30
• thanks a lot for replying. appreciating it really as I'm confused on this point .. so the method which called get_bert_embed_matrix for getting word_embed_6 = torch.cat([hidden_states[i] for i in [-1,-2,-3,-4]], dim=-1) for sub-word and i need to calculate the mean for those values to get word-embedding Jan 14 at 13:38
• Please open a new question with all the details so that we can answer appropriately.
– noe
Jan 14 at 13:40

With regard to a dictionary of words, there can be no single dictionary for BERT because the BERT embeddings incorporate contextual information (i.e. the surrounding words in the sentence change the embedding for your target word). In theory, you could construct a dictionary for your words by passing single word sentences (though a single word may be broken down into multiple tokens).

If you're looking for an easy practical way to get the pretrained Bert embeddings, HuggingFace makes it easy.

I have given a simple code snippet below using python and specifically pytorch:

from transformers import BertTokenizer, BertModel
import torch

my_sentence = "Whatever your sentence is"
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

input_ids = tokenizer(my_sentence, return_tensors="pt")
output = model(**input_ids)

final_layer = output.last_hidden_state


The final_layer tensor will now hold the embeddings (768 dimensional) for each token in your input sentence. Note that the zeroth token is a start token (CLS) and the last token is an end token.

If you have a list of sentences (of single words in your case perhaps if you are making a dictionary), you can use the above code in a batched manner. However, you will need to extract a mask from the tokenizer and pass this to your model (to account for different length sentences).

Example:

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')