Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?

  • 1
    $\begingroup$ Might be worth adding a new 'oov' tag for out-of-vocabulary. $\endgroup$ Commented Mar 11 at 20:52

1 Answer 1


BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors.

Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.

ELMo is very different: it ingests characters and generate word-level representations. The fact that it ingests the characters of each word instead of a single token for representing the whole word is what grants ELMo the ability to handle unseen words.

  • $\begingroup$ BERT provides word-level embeddings, not sentence embedding. You are correct about averaging word embedding to get the sentence embedding part. My doubt is regarding out of vocabulary words and how pre-trained BERT handles it. If it is able to generate word embedding for words that are not present in the vocabulary. Do you happen to know anything about that? $\endgroup$ Commented Nov 17, 2020 at 22:26
  • 3
    $\begingroup$ Again, BERT does not provide word embeddings but subword embeddings. Here you can see BERT's subword vocabulary. There, for instance, you can find tokens recurring and ##ly, which would be the subwords used to represent the word recurringly, which is not in the vocabulary. BERT would give you separate vectors for recurring and ##ly. $\endgroup$
    – noe
    Commented Nov 17, 2020 at 23:05
  • 1
    $\begingroup$ Also, BERT does provide sentence-level embeddings in the first position of the output, which is where the special token [CLS] is placed in the input. $\endgroup$
    – noe
    Commented Nov 17, 2020 at 23:06
  • $\begingroup$ Thank you so much @ncasas for sharing vocabulary and recurringly example with me :) Also, do you know any python package available for bert embeddings? Currently, I am using pypi.org/project/bert-embedding $\endgroup$ Commented Nov 17, 2020 at 23:22
  • 1
    $\begingroup$ Thank you, your explanation helped me to understand that we were using Bert's contextual representations in a wrong way. We used the vectors to calculate their semantic distance to other others we have for a dictionary (cossim), and my student thought it was weird that some words have kinda distant results, but what happened is that the vector representation was given by the subwords not by the word itself. So we are updating our method, accordingly. $\endgroup$
    – Thiago
    Commented Apr 14, 2022 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.