Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?
$\begingroup$
$\endgroup$
0
asked Nov 17, 2020 at 19:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
9
BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors.
Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.
ELMo is very different: it ingests characters and generate word-level representations. The fact that it ingests the characters of each word instead of a single token for representing the whole word is what grants ELMo the ability to handle unseen words.
-
$\begingroup$ BERT provides word-level embeddings, not sentence embedding. You are correct about averaging word embedding to get the sentence embedding part. My doubt is regarding out of vocabulary words and how pre-trained BERT handles it. If it is able to generate word embedding for words that are not present in the vocabulary. Do you happen to know anything about that? $\endgroup$ Nov 17, 2020 at 22:26
-
2$\begingroup$ Again, BERT does not provide word embeddings but subword embeddings. Here you can see BERT's subword vocabulary. There, for instance, you can find tokens
recurringand##ly, which would be the subwords used to represent the wordrecurringly, which is not in the vocabulary. BERT would give you separate vectors forrecurringand##ly. $\endgroup$– noeNov 17, 2020 at 23:05 -
1$\begingroup$ Also, BERT does provide sentence-level embeddings in the first position of the output, which is where the special token
[CLS]is placed in the input. $\endgroup$– noeNov 17, 2020 at 23:06 -
$\begingroup$ Thank you so much @ncasas for sharing vocabulary and recurringly example with me :) Also, do you know any python package available for bert embeddings? Currently, I am using pypi.org/project/bert-embedding $\endgroup$ Nov 17, 2020 at 23:22
-
1$\begingroup$ Thank you, your explanation helped me to understand that we were using Bert's contextual representations in a wrong way. We used the vectors to calculate their semantic distance to other others we have for a dictionary (cossim), and my student thought it was weird that some words have kinda distant results, but what happened is that the vector representation was given by the subwords not by the word itself. So we are updating our method, accordingly. $\endgroup$– ThiagoApr 14, 2022 at 10:41