How can word2vec or BERT be used for previously unseen words

Question

Is there any way to modify word2vec or BERT to extend finding out embeddings for words that were not in the training data? My data is extremely domain-specific and I don't really expect pre-trained models to work very well. I also don't have access to huge amounts of this data so cannot train word2vec on my own. I was thinking something like a combination of word2vec and the PMI matrix (i.e. concatenation of the 2 vector representations). Would this work, would anyone have any other suggestions, please?

Thanks in advance!

Answer 1

BERT does not provide word-level representations, but subword representations. This implies that when an unseen word is presented to BERT, it will slice it into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words. Therefore, BERT can handle out-of-vocabulary words. Some other questions and answers in this site can help you with the implementation details of BERT's subword tokenization, e.g. this, this or this.

On the other hand, word2vec is a static table of words and vectors, so it is just meant to represent words that are already in its vocabulary.

Answer 2

fastText is your friend! It is actually an extension of word2vec, where some character n-grams of the words are averaged and added to make the prediction of skip-gram better. I highly recommend reading the original paper.