2
$\begingroup$

For lack of a better term, overfit here means a higher discrepancy between train and validation score and non-overfit means a lower discrepancy.

This "dilemma" just showed in neural network model I've recently working on. I trained the network with 10-fold cross-validation and got overfitted model (0.118 score difference):

  • 0.967 accuracy for training set and
  • 0.849 for validation set.

Then, I applied dropout layer with dropout rate of 0.3 after each hidden layer and got "less overfitted" model (0.057 score difference):

  • 0.875 accuracy for training set and
  • 0.818 for validation set

which is supposedly good since have lower discrepancy thus have better reliability for unknown data. The problem is, it has lower validation set score. My uninformed intuition says that no matter how overfitted your model is, validation set score is what matters because it indicates how well your model sees new data, so I choose the first model.

Is that a right intuition? How to go for this situation?

Improve this question
$\endgroup$
3
  • $\begingroup$ I find the drop in accuracy on validation a bit suspicious. Can you tell us more about your data set (number of variables, number of instance, class balance) ? How do you select your hyperparameters ? (because applying drop-out is very likely to change other parameters optima) $\endgroup$
    – Lucas Morin
    Commented Jan 6, 2020 at 9:49
  • $\begingroup$ @lcrmorin It's a really sparse dataset with 676 features with only dozens of non-zero values each row (otherwise zeros). It has 64000 instances of 4 class, which all had balanced occurence (each 16000). The model is a simple Keras implementation with all default parameters, except the optimizer is SGD with momentum=0.9 and Nesterov=True. $\endgroup$
    – kneejar
    Commented Jan 6, 2020 at 10:40
  • $\begingroup$ I suspect that not putting some work into researching better parameters than default explain why you have such a scenario. Instead of choosing between two bad models, I would suggest you to actually find one that is good. $\endgroup$
    – Lucas Morin
    Commented Jan 6, 2020 at 20:57

2 Answers 2

Reset to default
2
$\begingroup$

TLDR: I think you can do that as long as you understand why this is happening.

I think first you should be really sure your validation set is not in any way polluted by your training data. This sometimes can happen very indirect- in that case you would still be at risk. Otherwise there is nothing fundamentally wrong with using an overtrained predictor that still generalizes good enough.

Think about examples like the Titanic dataset. Its pretty small so it's not hard to learn all survivors in your training sample but still getting the general trend right.

Another point you should consider is how big your samples are. If they are small (maybe a few hundred datapoints) you could also observe statistical noise that can be quite large.

Improve this answer
$\endgroup$
3
$\begingroup$

What library are you using? Dropout is used during training to prevent overfitting.

Make sure dropout is not applied for the validation (which is standard for Keras). This could artificially decrease your validation accuracy.

Also, accuracy is a bad metric to evaluate your performance. See this answer to find out why. Try ROC-AUC to evaluate your model performance.

Improve this answer
$\endgroup$
2
  • $\begingroup$ @Pascallv Your answer had made me rethink that my result is due to wrong implementation. If what you mean is using the model with dropout for training and using the model with weights resulted from training and excluding the dropout for testing then the drop of score makes a lot of sense (since I don't think of that before). I'll be back after the 'fix'. $\endgroup$
    – kneejar
    Commented Jan 6, 2020 at 10:47
  • $\begingroup$ Yes, however, Keras, as far as I know, automatically excludes the dropout layer if you use "model.predict()" or "model.evaluate()". I don't know which library you are using but maybe before running the model again, check in the docs of your Neural Network library if dropout is excluded during testing by default $\endgroup$
    – PascalIv
    Commented Jan 6, 2020 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.