Ml models must strike a balance between predictive power and generalization power. Therefore, I split the data into train/test and calculate metrics on both. Often I see instructions in someone else's code like "if the overfitting is too big, then try to reduce the learning rate".

But what if the overfitting model is higher on test? I am using hyperparameter search with overfitting penalty. roc_auc_test - abs (roc_auc_train-roc_auc_test )*a, a=1/4. Something like you can overfit by 4 points for every point on the test. I also tried to run the search only by the metric on the test. And the result is that the overfitting model is still more efficient than the overfitting penalty model. For example roc_auc_train=0.7,roc_auc_test=0.7 vs roc_auc_train=0.95 roc_auc_train=0.75.

Which model should I choose? What are the pitfalls of choosing an overfitting model? Maybe it will be less stable over time or other problems.

  • $\begingroup$ If the performance on the test set is strong, how do you figure you have overfit? $\endgroup$
    – Dave
    Oct 17, 2022 at 12:57
  • $\begingroup$ In this case, the metric for train is too high. That was my question. When the metric on test is worse during overfitting, everything is obvious. Should I choose the option with the highest metric on test anyway? $\endgroup$
    – Andrew
    Oct 17, 2022 at 14:03


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.