3
$\begingroup$

I have a very small dataset (40 training examples, 10 validation examples, 120 classes) for which I'm getting very high accuracies with a very simple model in Keras (batchnorm, flatten, and dense layers only).

My training accuracy is 94-95% and validation accuracy is 76-78%. I know it's overfitting and I have tried a few things. The data is not images, so I cannot augment the data. I also cannot add data because it's a specific type. I'm using two dropout layers with 0.5 levels, and the architecture is very simple so I don't think I can reduce the architecture complexity. I can paste the model if anyone likes.

My question is: Is there ever a situation where validation accuracy cannot be as high as the training accuracy? Is there a limitation based on the size of the dataset? Or is it ALWAYS possible for validation accuracies to match training accuracies and the network just needs the right parameters?

$\endgroup$
  • $\begingroup$ You have 50 examples and 120 classes, how come? Can you please elaborate? Do you have multi label classification problem? $\endgroup$ – aivanov Mar 15 '18 at 17:10
3
$\begingroup$

Yes, it is possible to have a situation in which validation accuracy cannot be as high as training accuracy. Any situation in which noise (as opposed to generalizable properties of the feature set) in the training set is more predictive of the target variable within the training set would produce this.

Consider a situation in which a random property of the training sample considered is perfectly predictive, but this is found not to be true of all other examples outside the sample. The predictive power in the training set would be perfect, but outside the training set, less than perfect. This phenomenon is broadly referred to as "overfitting".

For example, let's consider a case where you have a set of fruit data and you're trying to establish whether a given fruit is an orange or a tangerine. You have 4 features- the circumference of the fruit around the stem-medial axis, the height of the fruit across the stem, a numerical value of the hue of the fruit skin, and the first letter of the amateur baseball team whose home park is closest to the field the fruit was grown in. Let's imagine that by some baffling coincidence, the baseball team letter in the training set was perfectly predictive of whether the fruit would be a tangerine or orange.

We can imagine that this would not hold true across the country or the world, which would produce a situation where the training set accuracy would be perfect, but the validation set accuracy would not be able to approach that using the same methods.

$\endgroup$
  • $\begingroup$ Thanks so much for the reply. Maybe noise is what is producing the discrepancy between training and validation accuracies. My data is all 2D matrices, on the order of 4000 rows and 21 columns. Something I've been trying to determine that may belong in another question is if data of this kind can be augmented. When I look online, data augmentation seems to only be done with images. $\endgroup$ – Randy Jun 5 '17 at 15:21
  • $\begingroup$ You're very welcome- at stack overflow, answers are judged by upvoting- if you feel like the answer has value, please click the up-arrow next to it. If you feel like it has completely answered your question and is the best of the available answers, please click the check mark next to it to accept it as answer (though, please wait 24 hours or so at least to provide time for other answers to be provided which may be better). $\endgroup$ – Thomas Cleberg Jun 5 '17 at 15:25
  • 1
    $\begingroup$ If you think your data might require augmentation for a specific reason, there are many solutions to this type of problem, for example, if you have heavily unbalanced classes that you're trying to discriminate between, SMOTE is one method of augmenting the under represented class. (jair.org/media/953/live-953-2037-jair.pdf) $\endgroup$ – Thomas Cleberg Jun 5 '17 at 15:26
  • $\begingroup$ Thanks, Thomas. I already upvoted your answer but it won't be visible because I'm at <15 reputation. $\endgroup$ – Randy Jun 5 '17 at 15:27
  • $\begingroup$ I actually have fully balanced classes. 40 training samples of 120 classes with 10 validation samples of each class. Will look into this method, thanks. $\endgroup$ – Randy Jun 5 '17 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.