I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting.
When doing model selection (selecting hyperparameters), I sometimes come across these cases:
hyperparameter configuration 1:
training set score: 0.6 validation set score: 0.6
hyperparameter configuration 2:
training set score: 0.9 validation set score: 0.65
Now if you look at the raw numbers, it does look like configuration 2 generalizes better than configuration 1 but I'm a little bit worried about the large difference (0.9 to 0.65) between the scores for the training data and the validation data.
My question is: should I just consider the validation score when choosing the best model for actually using in production or does the ratio between training set and validation set scores carry some information?
I'm tempted to believe that cases where the training set and validation set score are more or less similar (as in configuration 1 above) are somehow more stable than cases where there is a large difference, even though, in absolute terms, the generalization score is better for configuration 2.
Is there any actual basis for this feeling? Could someone shed some light into this?