What is a lower bound on the vocabulary size for generating word/sentence embedding using word2vec or skip thought vectors?

Question

I am working on a NLP related task. I have about 150 documents, each few pages long (5/6 pages long on average). After removing stopwords and other unnecessary symbols and digits, I have about 104,000 unique words. The task at hand probably require some kind of word embedding (such as word2vec) as simple bag-of-words type approach aren't working properly. However, I am concerned about the size of the data I have. I have looked at pre-trained word embedding (GloVec), however, due to the narrow focus of the domain (manufacturing) of our texts, I am hesitating to use these pre-trained vectors. That leaves me with training our own. However, the size of our data set concerns me. Hence I am just throwing this question out there: What should be the lower bound on the size of the vocabulary that we need in order to train a word embedding model (word2vec) that will be reasonable.

Any response would be greatly appreciated.

Thanks

Answer 1

It's not a straightforward question to answer as it is hard to compare the quality of two word2vec models with a meaningful metric. You could, of course, use the loss function, but that won't give much.

Another approach is more heuristic: take for example the frequency of each word, and remove those words that are repeated less than N times, where you can set N to for example 10 or 20. This is common practice as you need a certain number of repetitions of the same word to have some meaningful results.