In the docs: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
it is explained that max_features is ordered by term frequency across the corpus. Why not use the idf?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The reason is probably that using the top IDF features would mean selecting the least frequent words, in particular the ones which appear only once and are very frequent. These rare words should usually be removed because they are often due to chance and anyway are unlikely to appear again, therefore these are bad features likely to cause overfitting.
In other words, it's always better for the features to be frequent so their statistical relations with other variables (especially the target) can be estimated reliably by the algorithm. Picking the top IDF features would do the opposite: take the least reliable statistical information into account.
