1
$\begingroup$

I need to get word-vectors using BERT and got this function that i think it should be the one i need

def get_bert_embed_matrix(sentences):
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model_config = transformers.AutoConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
    model = transformers.AutoModel.from_pretrained('bert-base-uncased', config=model_config)
    tokenizer = transformers.AutoTokenizer.from_pretrained('bert-base-uncased')  
   for i in sentences:
        tokenized_text = tokenizer.tokenize(i)
        indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)        
        tokens_tensor = torch.tensor([indexed_tokens])
        model.eval()
        outputs = model(tokens_tensor)
        hidden_states = outputs[2]
        word_embed_6 = torch.cat([hidden_states[i] for i in [-1,-2,-3,-4]], dim=-1)
    return word_embed_6

Does the method return vectors for sub-word or word ?

Improve this question
$\endgroup$
4
  • $\begingroup$ You need to provide more context in order to get a proper answer on this. $\endgroup$
    – Peter
    Commented Jan 14, 2022 at 15:21
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Jan 14, 2022 at 15:21
  • $\begingroup$ thanks for replying to the post . i clarified my problem, please see it . Thanks $\endgroup$
    – user5520049
    Commented Jan 14, 2022 at 15:30
  • $\begingroup$ A full discussion on BERT Word Embeddings can be found here mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial with a colab notebook here colab.research.google.com/drive/… $\endgroup$
    – Manuel Alves
    Commented Oct 8, 2022 at 9:41

1 Answer 1

Reset to default
2
$\begingroup$

About the first piece of code you posted:

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.

About the second piece of code you posted:

It seems to add up the subword embeddings of each word (only the last BERT layer) and concatenate each resulting vector into a tensor for the whole sentence (whose length would be the number of words).

Improve this answer
$\endgroup$
4
  • $\begingroup$ thank a lot for answering but excuse me do you mean that i need to loop in words not sentences right ? i mean here in this line for w in words: instead of for i in sentences: if so how can i get words in the sentences , does tokenization return them ? $\endgroup$
    – user5520049
    Commented Jan 14, 2022 at 17:22
  • 1
    $\begingroup$ I updated my answer referring to the second piece of code you posted. $\endgroup$
    – noe
    Commented Jan 15, 2022 at 9:21
  • $\begingroup$ thanks a lot for replying. my specific task if i need to represent the embedding layer for image captioning task i need to represent the vectors for each word in the sentence so if you please do you see that the second code is suitable for this task ? i updated my question too with my result $\endgroup$
    – user5520049
    Commented Jan 15, 2022 at 13:59
  • $\begingroup$ exucse me i read that the word embedding by concatenating the last four layers(word_emb_6), giving us a single word vector per token. Each vector will have a length 4 x 768 = 3,072. All other word embeddings have the 768 length vectors per token. I'm confused about sub-words and words embedding in BERT $\endgroup$
    – user5520049
    Commented Jan 15, 2022 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.