What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?

Question

When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5.

Are there any theoretical considerations for selecting a threshold for min_value? Depending on the contexts the words are seen in (and the potential variety of them), it seems that it could take more than 5 words for the model to learn a good vector representation of the word. Are there any papers that specify when the learned vectors become more static? Alternatively, can you train the model on all words, but only utilize the vectors of words that appeared in the corpus more than a specified number of times? Is frequency a better cutoff than min_count?

Answer 1

There are a few papers that have investigated the optimal value for min_count. One paper, by Mikolov et al. (2013), found that min_count=5 was a good choice for a general-purpose word2vec model. However, the optimal value for min_count may vary depending on the specific dataset and application.

It is possible to train a word2vec model on all words, but only utilize the vectors of words that appeared in the corpus more than a specified number of times. This is called "subsampling". Subsampling can be a good way to reduce the size of the vocabulary and the amount of time it takes to train the model. However, it is important to note that subsampling can also reduce the quality of the word vectors.

Whether frequency or min_count is a better cutoff depends on the specific application. If the goal is to learn good vector representations for all words, then min_count may be a better choice. However, if the goal is to reduce the size of the vocabulary or the amount of time it takes to train the model, then frequency may be a better choice.

Ultimately, the best way to choose a value for min_count is to experiment with different values and see what works best for your specific application.