Recently, I used TfidfVectorizer in scikit-learn library to calculate a matrix of TF-IDF features. However, I do not know how to set some parameters such as max_features, min_df, max_df, etc.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I think these parameters are mostly used when you combine the vectorizer and a machine learning model in a pipeline. Therefore, you should tune these parameters based on the outcome of your model training.
For example, if your task is to classify input texts, you may want to tune the max_features parameter such that the number of features is not too large, but your model can still perform reasonably well. In general, these parameters are used to control the size of the vector to be fed into your machine learning model, so as to avoid overfitting and improve the efficiency of model training.
