0
$\begingroup$

When comparing the test vs train model performance to ensure no overfitting (e.g., using AUC ROC as an example), is it better to select the model with the largest test score, or the model with the smallest difference between train and test scores (smaller delta)?

For example, Which of the following two model is preferable?

Model 1 - Large test score, but large train/test delta

  • Train AUC ROC = 0.83
  • Test AUC ROC = 0.8

Model 2 - Small train/test delta, but small test score

  • Train AUC ROC = 0.79
  • Test AUC ROC = 0.78
$\endgroup$

1 Answer 1

1
$\begingroup$

When comparing models, the main objective is often to choose the one that performs well on unseen data, that is, the model that has a good generalization ability. This means you'd typically prefer the model with the better test score since it has been evaluated on data it was not trained on.

However, if there's a substantial difference between the training and test scores (as in Model 1), it might suggest overfitting. Overfitting occurs when the model learns the training data too well, including its noise and outliers, and thus performs poorly on new, unseen data.

On the other hand, a small difference between the train and test scores (as in Model 2) indicates that the model is not overfitting. However, the test score is less than Model 1, which might mean that the model is underfitting the data, not capturing enough complexity to perform as well on unseen data.

In practice, the choice between these two models would likely depend on the specific context, including the importance of prediction accuracy, the consequences of making errors, and the interpretability of the model.

If maximizing performance on unseen data is the primary concern, you might lean towards Model 1. However, the large delta might lead to unstable performance with other test sets. If consistency and stability of performance are more important, you might choose Model 2 because of its smaller delta, even if it sacrifices some accuracy.

But generally, it's ideal to aim for both a high test score and a small train/test delta. You might want to try using a validation set to tune hyperparameters, or techniques like regularization or cross-validation, to try to achieve both.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.