1
$\begingroup$

I would like to ask you how many words should be taken as features in a ML program. For example, if I have 30000 distinct words to make a vocabulary, what would a good number be? I am currently removing stopwords, words with a few characters, numbers; applying lemmatisation; removing punctuation; analysing word frequency to keep the top 50 words. However I do not know if out of 30000 distinct words, 50 would be too low. What would be a good way to determine how many to keep or select?

$\endgroup$
1
  • 1
    $\begingroup$ This depends entirely on the amount of data that you have and the downstream applications. Common machine translation systems today, for instance, use a vocabulary of 30,000 subword units. 50 seems tremendously low—but let cross-validation be the way that you figure it out. $\endgroup$ Feb 23 at 20:59
0
$\begingroup$

It depends on many factors: number of instances, number of classes, token/type ratio, etc. A common basic technique in order to avoid having too many words is to discard the ones which appear rarely, for example by discarding all the words which appear in less than 3 documents. Selecting only the top $N$ most frequent words is unusual, even though it's a similar idea. There is a risk to end up mostly with words which appear frequently and don't have a good discriminative power.

The proper way to determine an optimal threshold, for example the minimum frequency $N$, is to run a training/testing experiment for every value of $N$ and select the value corresponding to maximum performance. Note that this experiment should use a validation set different from the final test set in order to avoid data leakage.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.