0
$\begingroup$

A classic question with an unclear answer, is it better to have an overfitted model performing better on a Cross-Validation setting, or a non-overfitted model performing worse?

In this context, higher overfitting means higher discrepancy between train and test sets.

Overfitted: Avg Test AUROC 0.82 & Avg Train AUROC 0.96
Non-overfitted: Avg Test AUROC 0.78 & Avg Train AUROC 0.81

Which model should you use?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
0
$\begingroup$

Not sure what you are refeering to with 'CV'. The overfitted model will perform really poorly with data that are in the wild. If you were willing to continue the training infinitely, you would end up with a over-fitted model having an even better score.

The non-overfitted model will behave more accordingly when facing new unseen data. This model will allow you to have higher prediction power.

I will go with the latest.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I meant the best cross-validation score. Even though seen as a technique to prevent over-fitting, the cross-validation technique can also point to an overfitted model as the best when assessing performance in test sets $\endgroup$
    – simon
    Commented Mar 29, 2023 at 13:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.