Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am using the Sklearn logistic regression function to do a binary classification task on texts.
I did the task using three different inputs: Bag-Of-Words, TF-IDF, Doc2vec embeddings.
The question is that for the Bag-Of-Words, stronger regularization improves the performance on testing set. However, for TF-IDF and Doc2vec embedding, stronger regularization doesn't improve the performance on testing set. Actually the performance downgrades with stronger regularization.
Why this is the case?
Title question
The answer to the title question is a pretty clear-cut "no." For any fixed dataset, taking sufficiently strong regularization will make the linear model essentially constant, presumably not the best model. Like most other things, there's a bias-variance tradeoff: increasing regularization weakens the model's ability to learn the training data, which to a point will tend to prevent overfitting and improve test performance.
Your more specific context
Word embeddings are substantially different from the other two, so the optimal regularization strength is probably mostly unrelated. Between bow and tfidf, if nothing else the scale of the features will be larger in bow, so if you're not scaling, the penalties will be larger and the optimal regularization strength is likely to be smaller.
