0
$\begingroup$

enter image description here

In the attached figure, the x-axis is the number features of s removed, and the y-axis is the average auc score over 10 CVs.

I want to choose the point with the highest score while avoiding overfitting points.

While looking for standards for overfitting, I found here that a 10% difference in auc score is 'overfitting' and a 1% difference is 'balanced/good fit'. Then, I realized that if the difference is more than 10% even after cross validation, overfitting can be suspected. However, what is curious is when the score difference is more than 1% and less than 10%. For example, if the difference is 5%, should you suspect overfitting? Or should we suspect overfitting when it is 9%? Or, if it is less than 10%, do we not need to suspect overfitting?

$\endgroup$
3
  • $\begingroup$ What kind of cross validation are you using and what are the sizes of the train/test data? $\endgroup$ Mar 15 at 15:23
  • $\begingroup$ I used the nested cv method to evaluate how well a model generalizes on a specific data set. The folds are 10 and 5, respectively. $\endgroup$
    – JAE
    Mar 17 at 11:48
  • $\begingroup$ and data has 8,500 rows $\endgroup$
    – JAE
    Mar 17 at 12:23

1 Answer 1

0
$\begingroup$

For example, if the difference is 5%, should you suspect overfitting? Or should we suspect overfitting when it is 9%? Or, if it is less than 10%, do we not need to suspect overfitting?

There are no hard and fast rules for when overfitting has occurred. Overfitting is more a degree than a matter of kind. That your model has much larger AUC on the training data is ultimately unsurprising since it was ostensibly constructed to make this as large as possible.

That being said, if you find that the difference is practically relevant (a question only really you can answer since you have the context on the problem) then I would recommend regularizing the model somewhat. This should shrink the gap between train and test further.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.