# How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?

• Again, BERT does not provide word embeddings but subword embeddings. Here you can see BERT's subword vocabulary. There, for instance, you can find tokens recurring and ##ly, which would be the subwords used to represent the word recurringly, which is not in the vocabulary. BERT would give you separate vectors for recurring and ##ly. – ncasas Nov 17 '20 at 23:05
• Also, BERT does provide sentence-level embeddings in the first position of the output, which is where the special token [CLS] is placed in the input. – ncasas Nov 17 '20 at 23:06