Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?
$\begingroup$
$\endgroup$
asked Nov 17 '20 at 19:34
$\begingroup$
$\endgroup$
BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors.
Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.
ELMo is very different: it ingests characters and generate word-level representations. The fact that it ingests the characters of each word instead of a single token for representing the whole word is what grants ELMo the ability to handle unseen words.
-
$\begingroup$ BERT provides word-level embeddings, not sentence embedding. You are correct about averaging word embedding to get the sentence embedding part. My doubt is regarding out of vocabulary words and how pre-trained BERT handles it. If it is able to generate word embedding for words that are not present in the vocabulary. Do you happen to know anything about that? $\endgroup$ – Sayali Sonawane Nov 17 '20 at 22:26
-
2$\begingroup$ Again, BERT does not provide word embeddings but subword embeddings. Here you can see BERT's subword vocabulary. There, for instance, you can find tokens
recurringand##ly, which would be the subwords used to represent the wordrecurringly, which is not in the vocabulary. BERT would give you separate vectors forrecurringand##ly. $\endgroup$ – ncasas Nov 17 '20 at 23:05 -
$\begingroup$ Also, BERT does provide sentence-level embeddings in the first position of the output, which is where the special token
[CLS]is placed in the input. $\endgroup$ – ncasas Nov 17 '20 at 23:06 -
$\begingroup$ Thank you so much @ncasas for sharing vocabulary and recurringly example with me :) Also, do you know any python package available for bert embeddings? Currently, I am using pypi.org/project/bert-embedding $\endgroup$ – Sayali Sonawane Nov 17 '20 at 23:22
-
$\begingroup$ The most "famous" BERT python package is huggingface's transformers: github.com/huggingface/transformers $\endgroup$ – ncasas Nov 18 '20 at 7:28
