I've splitted my training set in the ratio 80:20 and have developed cnn model with a dropout of 0.5. I'm getting an accuracy of 98%. But the validation accuracy stays greater than training accuracy. Is there any problem with this?Does it leads to over fitting? If yes, then why am I getting an accuracy of 98%?The graph is given below.The red line is the validation accuracy and blue is training accuracy.
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ dataset size? label distribution? please provide as much additional detail as possible $\endgroup$– oW_Commented Jun 9, 2020 at 3:36
-
$\begingroup$ This is a 12 class classification problem and data size is 1.64 GB. $\endgroup$– user82808Commented Jun 9, 2020 at 4:16
-
1$\begingroup$ Is it possible that you have more samples from one or few classes in the validation data? $\endgroup$– kangaroo_cliffCommented Jun 9, 2020 at 4:39
-
$\begingroup$ yeah, the distribution is random. $\endgroup$– user82808Commented Jun 9, 2020 at 5:07
-
2$\begingroup$ This was a case when I worked with cats/dogs classifier. The validation split was so small that the model ( without sufficient training ) could get high accuracy. Whereas, the training data was larger and hence the model's accuracy was low here. $\endgroup$– Shubham PanchalCommented Jun 9, 2020 at 5:40
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
As @Suren has eluded to, it could be the case that your validation accuracy is higher than your training accuracy due to the distribution of the classes over the training and validation sets. If the training set contains a higher proportion of a particular class and the validation class contains examples of that particular class as well, then of course you will see validation accuracy being high.
The advice would be to balance out the classes over the training and validation and sets. You can do this data augmentation, in that you artificially increase the number of examples based not he data you already have. Have a look at this paper which evaluates the effect of different methods of class balancing using data augmentation on generalisation performance of CNNs: https://arxiv.org/pdf/1710.05381.pdf
$\begingroup$
$\endgroup$
As @shepan6 has mentioned, this could be because of your class balance in the validation set. You should print out the confusion matrix on both the training and the validation set. You might find one high error class to be underrepresented in your validation set. You might also find that your algorithm gets two classes confused frequently. For example, with MNIST, 3 and 8 are often confused since they look similar when written. Using this example, if you don't have a lot of 3's and/or 8's in your validation set, then the accuracy would be higher in the validation set than the test set.
Even though a difference of 1% with 950 samples is just 9.5 samples, the change from 97% to 98% is probably not just do to randomness. The probability of a binomial with n = 950 and p = 0.97 being greater than or equal to 950*0.98=931 is about 3.7%, so statistically this is decent sized jump. (Here I'm ignoring randomness coming from uncertainty in the training set accuracy measurement, but with 3800 samples, there isn't much to worry about, especially since the validation error goes up in general.)
Of course, you may also want to review your validation code for bugs. Maybe you simply swapped your training and validation sets.
Lastly, you may have over-tuned your algorithm to do well on the validation set, and you may want to consider collecting new validation data if possible.
