0
$\begingroup$

I am using the Sklearn logistic regression function to do a binary classification task on texts.

I did the task using three different inputs: Bag-Of-Words, TF-IDF, Doc2vec embeddings.

The question is that for the Bag-Of-Words, stronger regularization improves the performance on testing set. However, for TF-IDF and Doc2vec embedding, stronger regularization doesn't improve the performance on testing set. Actually the performance downgrades with stronger regularization.

Why this is the case?

$\endgroup$
1
  • $\begingroup$ Please elaborate, I guess the data representation is quite different in all three cases, so that it is hard to answer your question. Provide a minimal example of data representation $\endgroup$ – Peter Jun 28 '20 at 20:24
1
$\begingroup$

Title question

The answer to the title question is a pretty clear-cut "no." For any fixed dataset, taking sufficiently strong regularization will make the linear model essentially constant, presumably not the best model. Like most other things, there's a bias-variance tradeoff: increasing regularization weakens the model's ability to learn the training data, which to a point will tend to prevent overfitting and improve test performance.

Your more specific context

Word embeddings are substantially different from the other two, so the optimal regularization strength is probably mostly unrelated. Between bow and tfidf, if nothing else the scale of the features will be larger in bow, so if you're not scaling, the penalties will be larger and the optimal regularization strength is likely to be smaller.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.