For lack of a better term, overfit here means a higher discrepancy between train and validation score and non-overfit means a lower discrepancy.
This "dilemma" just showed in neural network model I've recently working on. I trained the network with 10-fold cross-validation and got overfitted model (0.118 score difference):
- 0.967 accuracy for training set and
- 0.849 for validation set.
Then, I applied dropout layer with dropout rate of 0.3 after each hidden layer and got "less overfitted" model (0.057 score difference):
- 0.875 accuracy for training set and
- 0.818 for validation set
which is supposedly good since have lower discrepancy thus have better reliability for unknown data. The problem is, it has lower validation set score. My uninformed intuition says that no matter how overfitted your model is, validation set score is what matters because it indicates how well your model sees new data, so I choose the first model.
Is that a right intuition? How to go for this situation?