I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting.

When doing model selection (selecting hyperparameters), I sometimes come across these cases:

  • hyperparameter configuration 1:

    training set score: 0.6
    validation set score: 0.6
  • hyperparameter configuration 2:

    training set score: 0.9
    validation set score: 0.65

Now if you look at the raw numbers, it does look like configuration 2 generalizes better than configuration 1 but I'm a little bit worried about the large difference (0.9 to 0.65) between the scores for the training data and the validation data.

My question is: should I just consider the validation score when choosing the best model for actually using in production or does the ratio between training set and validation set scores carry some information?

I'm tempted to believe that cases where the training set and validation set score are more or less similar (as in configuration 1 above) are somehow more stable than cases where there is a large difference, even though, in absolute terms, the generalization score is better for configuration 2.

Is there any actual basis for this feeling? Could someone shed some light into this?

  • 1
    $\begingroup$ The second configuration is better but one would like to believe that you can do reduce the discrepancy with better regularization. Or maybe your test set qualitatively differs from your training set. As you know, a small training error coupled with a high test error indicates overfitting, which can be ameliorated by regularization. This is where your ratio comes in. $\endgroup$
    – Emre
    Jun 25 '17 at 20:58
  • $\begingroup$ Keyword for searching about this topic: "generalization gap" $\endgroup$
    – BrunoF
    Jan 20 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.