Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?


BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors.

Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.

ELMo is very different: it ingests characters and generate word-level representations. The fact that it ingests the characters of each word instead of a single token for representing the whole word is what grants ELMo the ability to handle unseen words.

  • $\begingroup$ BERT provides word-level embeddings, not sentence embedding. You are correct about averaging word embedding to get the sentence embedding part. My doubt is regarding out of vocabulary words and how pre-trained BERT handles it. If it is able to generate word embedding for words that are not present in the vocabulary. Do you happen to know anything about that? $\endgroup$ – Sayali Sonawane Nov 17 '20 at 22:26
  • 2
    $\begingroup$ Again, BERT does not provide word embeddings but subword embeddings. Here you can see BERT's subword vocabulary. There, for instance, you can find tokens recurring and ##ly, which would be the subwords used to represent the word recurringly, which is not in the vocabulary. BERT would give you separate vectors for recurring and ##ly. $\endgroup$ – ncasas Nov 17 '20 at 23:05
  • $\begingroup$ Also, BERT does provide sentence-level embeddings in the first position of the output, which is where the special token [CLS] is placed in the input. $\endgroup$ – ncasas Nov 17 '20 at 23:06
  • $\begingroup$ Thank you so much @ncasas for sharing vocabulary and recurringly example with me :) Also, do you know any python package available for bert embeddings? Currently, I am using pypi.org/project/bert-embedding $\endgroup$ – Sayali Sonawane Nov 17 '20 at 23:22
  • $\begingroup$ The most "famous" BERT python package is huggingface's transformers: github.com/huggingface/transformers $\endgroup$ – ncasas Nov 18 '20 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.