1
$\begingroup$

In the docs: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html

it is explained that max_features is ordered by term frequency across the corpus. Why not use the idf?

$\endgroup$

1 Answer 1

1
$\begingroup$

The reason is probably that using the top IDF features would mean selecting the least frequent words, in particular the ones which appear only once and are very frequent. These rare words should usually be removed because they are often due to chance and anyway are unlikely to appear again, therefore these are bad features likely to cause overfitting.

In other words, it's always better for the features to be frequent so their statistical relations with other variables (especially the target) can be estimated reliably by the algorithm. Picking the top IDF features would do the opposite: take the least reliable statistical information into account.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.