I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.

After the representation of words I want to clusterize it, find similar vectors (i.e. words). Maybe even then make a classification to a given classes (classification there unsupervised --- since I don't have labeled data to train on).

I know that almost any task can be solved "inside" BERT, i.e. using fine-tuning in final layer.

Since all described above, I have two QUESTIONS; answers/hints/anything really appreciated since i'm stuck on that:

  1. How to just extract embeddings from BERT using some dictionary of words and use word representations for futher work?
  2. Can we solve inside BERT using fine-tuning the next problem: a). Load dictionary of words into BERT b). Load given classes (words representing each class. E.g. "fashion", "nature"). c) Make an unsupervised classification task?
  • $\begingroup$ Why do you mean by "unsupervised classification task"? Please elaborate. $\endgroup$
    – noe
    Commented Apr 14, 2021 at 6:40
  • $\begingroup$ @noe that means that I have classes labels. But instead of standard classification, we're not given labeled data for training. In supervised case we would then make error function, (LogLoss) to maximize posterior probabilities taking derivatives and renewing our parameters. LogLoss uses given labeled data in $y_i$. So this is supervised classification -- using real labels to train on. In my case there is only labels of classes, to which we need to split the data $\endgroup$
    – taciturno
    Commented Apr 14, 2021 at 8:37
  • $\begingroup$ I understand that you want to use BERT do word tagging, not sentence classification, right? $\endgroup$
    – noe
    Commented Apr 14, 2021 at 18:06
  • $\begingroup$ @noe, yes, I simply don't have sentences. Or we can assume that I have sentences with volume equal to one (one word in each sentence). $\endgroup$
    – taciturno
    Commented Apr 14, 2021 at 18:20

2 Answers 2


With regard to a dictionary of words, there can be no single dictionary for BERT because the BERT embeddings incorporate contextual information (i.e. the surrounding words in the sentence change the embedding for your target word). In theory, you could construct a dictionary for your words by passing single word sentences (though a single word may be broken down into multiple tokens).

If you're looking for an easy practical way to get the pretrained Bert embeddings, HuggingFace makes it easy.

I have given a simple code snippet below using python and specifically pytorch:

from transformers import BertTokenizer, BertModel
import torch

my_sentence = "Whatever your sentence is"
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

input_ids = tokenizer(my_sentence, return_tensors="pt")
output = model(**input_ids)

final_layer = output.last_hidden_state

The final_layer tensor will now hold the embeddings (768 dimensional) for each token in your input sentence. Note that the zeroth token is a start token (CLS) and the last token is an end token.

If you have a list of sentences (of single words in your case perhaps if you are making a dictionary), you can use the above code in a batched manner. However, you will need to extract a mask from the tokenizer and pass this to your model (to account for different length sentences).


tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encoded_inputs = tokenizer(list_of_sentences, padding=True, 
                            truncation=True, return_tensors="pt")
ids = encoded_inputs['input_ids']
mask = encoded_inputs['attention_mask']

output = model(ids, mask)
final_layer = output.last_hidden_state
  • 1
    $\begingroup$ Thanks, that's valuable. For first part: I don't want to make a single dictionary for all BERT's model -- cause it doesn't make sense :). For second part: I can't understand for what purpose it's necessary to use mask for my case, if I have sentences contains one word each? $\endgroup$
    – taciturno
    Commented Apr 16, 2021 at 16:46
  • $\begingroup$ However, your code seems not working: some attribute error $\endgroup$
    – taciturno
    Commented Apr 17, 2021 at 13:56
  • 2
    $\begingroup$ I founded out problems with it, I will edit your answer to correct your errors $\endgroup$
    – taciturno
    Commented Apr 17, 2021 at 14:41
  1. BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.

  2. You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-speech (POS) (e.g. this) tagging or named entity recognition (NER) (e.g. this). Normally, you associate the tag to either the first or the last subword token in the word. If you prepare a dataset that way, you could fine-tune BERT to perform word tagging with the classes you need. If you only have the words, you could find some text corpus (ideally of the intended domains) and apply the described data preparation process.

  • $\begingroup$ Thanks for response, but searching for 4k datasets won't be a good case I think... Maybe I could find some dataset with images (that's what I have --- dictionary of images labels) from some social network, I could try that approach. But since I can't use Instagram DS (you answered this question if you can remember), I don't know what to do. On google's service for datasets searching I didn't find anything related and valid for my task. $\endgroup$
    – taciturno
    Commented Apr 16, 2021 at 16:54
  • $\begingroup$ excuse me i got from your answer that BERT works on sub-words, not Word. so the method for concatenating the last four layers of BERT will represent sub-words ? $\endgroup$ Commented Jan 14, 2022 at 4:16
  • $\begingroup$ @user5520049 yes $\endgroup$
    – noe
    Commented Jan 14, 2022 at 9:30
  • $\begingroup$ thanks a lot for replying. appreciating it really as I'm confused on this point .. so the method which called get_bert_embed_matrix for getting word_embed_6 = torch.cat([hidden_states[i] for i in [-1,-2,-3,-4]], dim=-1) for sub-word and i need to calculate the mean for those values to get word-embedding $\endgroup$ Commented Jan 14, 2022 at 13:38
  • 1
    $\begingroup$ Please open a new question with all the details so that we can answer appropriately. $\endgroup$
    – noe
    Commented Jan 14, 2022 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.