Let's say I trained a Skip-Gram model (Word2Vec) for my vocabulary of size 10,000. The representation allows me to reduce the dimension from 10,000 (one-hot-encoding) to 100 (size of hidden layer of the neural network).
Now suppose I have a word in my test set which was not in my training vocabulary. What is a reasonable representation of the word in the space of dimension 100? For me, it seems that I cannot use the neural network I trained to come up with the word embeddings.
Generating word embeddings for "OOV"(out of vocabulary) words is one of the major limitations of many standard embeddings like Glove and word2vec. However, fastText circumvents this problem to some extent.
Instead of the traditional approaches which have distinct vectors for each word, they take a character n-grams level representation. For instance, a word with n= 3, will be represented by the character n-grams:
<wh, whe, her, ere, re>
and the special sequence:
< where >
Here,<>are part of the n-grams.
$$ s(w,c) = \sum_{g\varepsilon G_{_{w}}} z_{g}^{T} v_{c} $$ Here, $G$ represents the size of a dictionary of n-grams, and given a word $w$, then $G_{w}\subset \left \{ 1, ..., G \right \}$ represents the set of n-grams appearing in $w$. They associate a vector representation $z_{g}$ to each n-gram $g$ and represent a word by the sum of the vector representations of its n-grams.
This helps them tackle OOV words via knowing some representations of a subword.
For an instance, an OOV word:
sechero
The 3-gram:
<se, sec, ech, che, her, ero, ro>
since, these 3-grams may have been encountered during learning, through other known words, like:
<se - section
che - cheer
ro> - hero
Hence, it can form at least some sensible embedding, instead of returning a useless <UNK>
Fastext in fact is an extension to word2vec, with majorly the feature explained above.
During Word2Vec training, if you remember their is one hyperparaneter "min_count", which says minimum number of time a particular word should exist in corpus. Words which met this condition (along with other), considered as a part of vocabulary, else discarded.
In order to handle discarded words, we use another word representation i.e "UNK" token.
Similarly, in your case that word should be treated as "UNK".
Though you can go for further Word2Vec training for a particular word.
Another relative post on Handling UNK words
There are various ways to handle out of vocabulary words. One of the ways is definitely the above solution by Vipin. However, this is not the only way. There are implementations of using subword information to create vectors for out of vocabulary.
When you think about a business problem, its impossible to create a vocabulary with all the probable words to train the word2vec model irrespective of SkipGram or CBOW. One of the implementations I have come to appreciate is the implementation by Facebook in FastText. The implementation is based on this paper.
Here is the link to FastText.
You could also write a small network yourself to convert one hot into Vectors in your space by implementing a few papers together.
The question you need the answer to, IMHO is, how important is it for you to handle out of vocabulary words and what is its impact?
