# How can i get the vector of word using BERT?

I need to get word-vectors using BERT and got this function that i think it should be the one i need

def get_bert_embed_matrix(sentences):
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model_config = transformers.AutoConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
model = transformers.AutoModel.from_pretrained('bert-base-uncased', config=model_config)
tokenizer = transformers.AutoTokenizer.from_pretrained('bert-base-uncased')
for i in sentences:
tokenized_text = tokenizer.tokenize(i)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])
model.eval()
outputs = model(tokens_tensor)
hidden_states = outputs[2]
word_embed_6 = torch.cat([hidden_states[i] for i in [-1,-2,-3,-4]], dim=-1)
return word_embed_6


Does the method return vectors for sub-word or word ?

another function like

for s in sentences :
tokenized_s = tokenizer.tokenize(s)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_s)
tokens_tensor = torch.tensor([indexed_tokens]).to(device)
encoded_layers, _ = model(tokens_tensor)
bert_embedding = encoded_layers[11].squeeze(0)
split_s = s.split()
tokens_embedding = []
j = 0
for full_token in split_s:
curr_token = ''
x = 0
for i,_ in enumerate(tokenized_s[1:]): # disregard CLS
token = tokenized_cap[i+j]
piece_embedding = bert_embedding[i+j]

# full token
if token == full_token and curr_token == '' :
tokens_embedding.append(piece_embedding)
j += 1
break
else: # partial token
x += 1
if curr_token == '':
tokens_embedding.append(piece_embedding)
curr_token += token.replace('#', '')
else:
curr_token += token.replace('#', '')
if curr_token == full_token: # end of partial
j += x
break

s_embedding = torch.stack(tokens_embedding)
embeddings.append(s_embedding)
embedding_matrix = torch.stack(embeddings)


I'm confused , does the second function for word-embedding or sub-word too.

i tried to use the second code in image captioning task but got result like

CLS r o o m l i v i n g [ S E P ] [ S E P ] [ S E P ] [ S E P ]
[ S E P ] [ S E P ] [ S E P ] s e p s e p s e p s e p s e p
s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p
s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p
s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p
s e p s e p s e p s e p s e p


i got words not full sentences . i think i missed something too . i tried to pass the results from the function using

inputs2 = Input(shape=(max_length,))

lstm3 = LSTM(512)(sent)

model.layers[1].set_weights([embedding_matrix])

model.layers[1].trainable = False

• You need to provide more context in order to get a proper answer on this. Jan 14 at 15:21
• Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
– Community Bot
Jan 14 at 15:21
• thanks for replying to the post . i clarified my problem, please see it . Thanks Jan 14 at 15:30

About the first piece of code you posted:

At least from the apparent behavior, I would say your code computes the average of all subword vectors in a sentence, not for each word.

To compute word-level representations, you should average only the subwords belonging to a specific word, not all subwords in the sentence.

As a side note, I would suggest not to reuse variable names, as it makes the code confusing. In your code, you reuse i.

About the second piece of code you posted:

It seems to add up the subword embeddings of each word (only the last BERT layer) and concatenate each resulting vector into a tensor for the whole sentence (whose length would be the number of words).

• thank a lot for answering but excuse me do you mean that i need to loop in words not sentences right ? i mean here in this line for w in words: instead of for i in sentences: if so how can i get words in the sentences , does tokenization return them ? Jan 14 at 17:22
• I updated my answer referring to the second piece of code you posted.
– noe
Jan 15 at 9:21
• thanks a lot for replying. my specific task if i need to represent the embedding layer for image captioning task i need to represent the vectors for each word in the sentence so if you please do you see that the second code is suitable for this task ? i updated my question too with my result Jan 15 at 13:59
• exucse me i read that the word embedding by concatenating the last four layers(word_emb_6), giving us a single word vector per token. Each vector will have a length 4 x 768 = 3,072. All other word embeddings have the 768 length vectors per token. I'm confused about sub-words and words embedding in BERT Jan 15 at 22:31