0
$\begingroup$

I need to get word-vectors using BERT and got this function that i think it should be the one i need

def get_bert_embed_matrix(sentences):
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model_config = transformers.AutoConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
    model = transformers.AutoModel.from_pretrained('bert-base-uncased', config=model_config)
    tokenizer = transformers.AutoTokenizer.from_pretrained('bert-base-uncased')  
   for i in sentences:
        tokenized_text = tokenizer.tokenize(i)
        indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)        
        tokens_tensor = torch.tensor([indexed_tokens])
        model.eval()
        outputs = model(tokens_tensor)
        hidden_states = outputs[2]
        word_embed_6 = torch.cat([hidden_states[i] for i in [-1,-2,-3,-4]], dim=-1)
    return word_embed_6

Does the method return vectors for sub-word or word ?

another function like

for s in sentences :                 
             tokenized_s = tokenizer.tokenize(s)                
             indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_s)
             tokens_tensor = torch.tensor([indexed_tokens]).to(device)
             with torch.no_grad():
                encoded_layers, _ = model(tokens_tensor)
             bert_embedding = encoded_layers[11].squeeze(0)
             split_s = s.split()
             tokens_embedding = []
             j = 0
             for full_token in split_s:
                 curr_token = ''
                 x = 0
                 for i,_ in enumerate(tokenized_s[1:]): # disregard CLS
                     token = tokenized_cap[i+j]
                     piece_embedding = bert_embedding[i+j]
                        
                        # full token
                     if token == full_token and curr_token == '' :
                         tokens_embedding.append(piece_embedding)
                         j += 1
                         break
                     else: # partial token
                         x += 1
                         if curr_token == '':
                             tokens_embedding.append(piece_embedding)
                             curr_token += token.replace('#', '')
                         else:
                             tokens_embedding[-1] = torch.add(tokens_embedding[-1], piece_embedding)
                             curr_token += token.replace('#', '')
                             if curr_token == full_token: # end of partial
                                j += x
                                break                            

             s_embedding = torch.stack(tokens_embedding) 
             embeddings.append(s_embedding)
    embedding_matrix = torch.stack(embeddings)

I'm confused , does the second function for word-embedding or sub-word too.

i tried to use the second code in image captioning task but got result like

CLS r o o m l i v i n g [ S E P ] [ S E P ] [ S E P ] [ S E P ]
[ S E P ] [ S E P ] [ S E P ] s e p s e p s e p s e p s e p 
s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p
s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p 
s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p 
s e p s e p s e p s e p s e p 

i got words not full sentences . i think i missed something too . i tried to pass the results from the function using

inputs2 = Input(shape=(max_length,))
   
sent = Embedding(vocab_size, 3072, mask_zero=True)(inputs2)
    
lstm3 = LSTM(512)(sent)

model.layers[1].set_weights([embedding_matrix])

model.layers[1].trainable = False
$\endgroup$
3
  • $\begingroup$ You need to provide more context in order to get a proper answer on this. $\endgroup$
    – Peter
    Jan 14 at 15:21
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Jan 14 at 15:21
  • $\begingroup$ thanks for replying to the post . i clarified my problem, please see it . Thanks $\endgroup$ Jan 14 at 15:30
1
$\begingroup$

About the first piece of code you posted:

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.


About the second piece of code you posted:

It seems to add up the subword embeddings of each word (only the last BERT layer) and concatenate each resulting vector into a tensor for the whole sentence (whose length would be the number of words).

$\endgroup$
4
  • $\begingroup$ thank a lot for answering but excuse me do you mean that i need to loop in words not sentences right ? i mean here in this line for w in words: instead of for i in sentences: if so how can i get words in the sentences , does tokenization return them ? $\endgroup$ Jan 14 at 17:22
  • 1
    $\begingroup$ I updated my answer referring to the second piece of code you posted. $\endgroup$
    – noe
    Jan 15 at 9:21
  • $\begingroup$ thanks a lot for replying. my specific task if i need to represent the embedding layer for image captioning task i need to represent the vectors for each word in the sentence so if you please do you see that the second code is suitable for this task ? i updated my question too with my result $\endgroup$ Jan 15 at 13:59
  • $\begingroup$ exucse me i read that the word embedding by concatenating the last four layers(word_emb_6), giving us a single word vector per token. Each vector will have a length 4 x 768 = 3,072. All other word embeddings have the 768 length vectors per token. I'm confused about sub-words and words embedding in BERT $\endgroup$ Jan 15 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.