I am running a CNN, on the 1st epoch my training set accuracy is 15% and validation set is 12%, by the 51st epoch my training accuracy is 87% and validation set is 13%. What is happening? What does it mean if my validation set is lower or greater than my training set ?? (my test set gives 12%)

  • $\begingroup$ It means you training set isn't a random sample from the union of training and validation sets. Resample and try again. $\endgroup$
    – Spacedman
    Apr 17 '18 at 16:06
  • $\begingroup$ Can you show us the plots you get using the history object in Keras such that we can see what is happening. Either its what @Spacedman said, or you are over-training and thus over-fitting. $\endgroup$
    – JahKnows
    Apr 17 '18 at 16:25
  • $\begingroup$ Its the usual sign of gross overfit. $\endgroup$
    – Michael M
    Apr 17 '18 at 19:35

This is a common high-variance problem due to overfitting.

Simply put: Good training accuracy together with low dev set (or in your terminology the validation set) accuracy it means you are expressing your training data very well, whereas your model fails to perform well on unseen data (High-variance Problem).

More general: there are few other combinations of this high-/low-bias together with high-/low-variance that are explained in length in the context of Bias-Variance Trade-off by Andrew Ng in his first Machine Learning course and even more relevant to Convolutional Neural Networks in his last DeepLearning course, see this video. You also easily find many blog posts discussing about these stuffs like this blog post, or this one.

What you see here is quite alright. If fact it is highly recommended practice that you start by a model capable of describing your training set very well first (even overfit). Then if overfit meaning there is a large difference in accuracy between train/dev sets (similar to your scenario), you start penalizing the model (aka regularization). Once again you can find tons of materials on how to regularize CNN models. On the top of your search you will find Dropout, Max-Pooling. See here for example.

I also suggest to pay attention to your data split distributions. It is a very fundamental concept that often paid less attention to. Violating this assumption (aka data split mismatch) could lead to problems that could appear in any forms like large accuracy difference of model on data splits. Maybe check Ng's short video.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.