1
$\begingroup$

I am working on a NLP related task. I have about 150 documents, each few pages long (5/6 pages long on average). After removing stopwords and other unnecessary symbols and digits, I have about 104,000 unique words. The task at hand probably require some kind of word embedding (such as word2vec) as simple bag-of-words type approach aren't working properly. However, I am concerned about the size of the data I have. I have looked at pre-trained word embedding (GloVec), however, due to the narrow focus of the domain (manufacturing) of our texts, I am hesitating to use these pre-trained vectors. That leaves me with training our own. However, the size of our data set concerns me. Hence I am just throwing this question out there: What should be the lower bound on the size of the vocabulary that we need in order to train a word embedding model (word2vec) that will be reasonable.

Any response would be greatly appreciated.

Thanks

$\endgroup$
1
$\begingroup$

It's not a straightforward question to answer as it is hard to compare the quality of two word2vec models with a meaningful metric. You could, of course, use the loss function, but that won't give much.

Another approach is more heuristic: take for example the frequency of each word, and remove those words that are repeated less than N times, where you can set N to for example 10 or 20. This is common practice as you need a certain number of repetitions of the same word to have some meaningful results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.