I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.

After the representation of words I want to clusterize it, find similar vectors (i.e. words). Maybe even then make a classification to a given classes (classification there unsupervised --- since I don't have labeled data to train on).

I know that almost any task can be solved "inside" BERT, i.e. using fine-tuning in final layer.

Since all described above, I have two QUESTIONS; answers/hints/anything really appreciated since i'm stuck on that:

  1. How to just extract embeddings from BERT using some dictionary of words and use word representations for futher work?
  2. Can we solve inside BERT using fine-tuning the next problem: a). Load dictionary of words into BERT b). Load given classes (words representing each class. E.g. "fashion", "nature"). c) Make an unsupervised classification task?
  • $\begingroup$ Why do you mean by "unsupervised classification task"? Please elaborate. $\endgroup$ – noe Apr 14 at 6:40
  • $\begingroup$ @noe that means that I have classes labels. But instead of standard classification, we're not given labeled data for training. In supervised case we would then make error function, (LogLoss) to maximize posterior probabilities taking derivatives and renewing our parameters. LogLoss uses given labeled data in $y_i$. So this is supervised classification -- using real labels to train on. In my case there is only labels of classes, to which we need to split the data $\endgroup$ – taciturno Apr 14 at 8:37
  • $\begingroup$ I understand that you want to use BERT do word tagging, not sentence classification, right? $\endgroup$ – noe Apr 14 at 18:06
  • $\begingroup$ @noe, yes, I simply don't have sentences. Or we can assume that I have sentences with volume equal to one (one word in each sentence). $\endgroup$ – taciturno Apr 14 at 18:20
  1. BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.

  2. You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-speech (POS) (e.g. this) tagging or named entity recognition (NER) (e.g. this). Normally, you associate the tag to either the first or the last subword token in the word. If you prepare a dataset that way, you could fine-tune BERT to perform word tagging with the classes you need. If you only have the words, you could find some text corpus (ideally of the intended domains) and apply the described data preparation process.

  • $\begingroup$ Thanks for response, but searching for 4k datasets won't be a good case I think... Maybe I could find some dataset with images (that's what I have --- dictionary of images labels) from some social network, I could try that approach. But since I can't use Instagram DS (you answered this question if you can remember), I don't know what to do. On google's service for datasets searching I didn't find anything related and valid for my task. $\endgroup$ – taciturno Apr 16 at 16:54

With regard to a dictionary of words, there can be no single dictionary for BERT because the BERT embeddings incorporate contextual information (i.e. the surrounding words in the sentence change the embedding for your target word). In theory, you could construct a dictionary for your words by passing single word sentences (though a single word may be broken down into multiple tokens).

If you're looking for an easy practical way to get the pretrained Bert embeddings, HuggingFace makes it easy.

I have given a simple code snippet below using python and specifically pytorch:

from transformers import BertTokenizer, BertModel
import torch

my_sentence = "Whatever your sentence is"
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

input_ids = tokenizer(my_sentence, return_tensors="pt")
output = model(**input_ids)

final_layer = output.last_hidden_state

The final_layer tensor will now hold the embeddings (768 dimensional) for each token in your input sentence. Note that the zeroth token is a start token (CLS) and the last token is an end token.

If you have a list of sentences (of single words in your case perhaps if you are making a dictionary), you can use the above code in a batched manner. However, you will need to extract a mask from the tokenizer and pass this to your model (to account for different length sentences).


tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encoded_inputs = tokenizer(list_of_sentences, padding=True, 
                            truncation=True, return_tensors="pt")
ids = encoded_inputs['input_ids']
mask = encoded_inputs['attention_mask']

output = model(ids, mask)
final_layer = output.last_hidden_state
  • $\begingroup$ Thanks, that's valuable. For first part: I don't want to make a single dictionary for all BERT's model -- cause it doesn't make sense :). For second part: I can't understand for what purpose it's necessary to use mask for my case, if I have sentences contains one word each? $\endgroup$ – taciturno Apr 16 at 16:46
  • $\begingroup$ However, your code seems not working: some attribute error $\endgroup$ – taciturno Apr 17 at 13:56
  • 1
    $\begingroup$ I founded out problems with it, I will edit your answer to correct your errors $\endgroup$ – taciturno Apr 17 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.