I have a very small dataset (40 training examples, 10 validation examples, 120 classes) for which I'm getting very high accuracies with a very simple model in Keras (batchnorm, flatten, and dense layers only).
My training accuracy is 94-95% and validation accuracy is 76-78%. I know it's overfitting and I have tried a few things. The data is not images, so I cannot augment the data. I also cannot add data because it's a specific type. I'm using two dropout layers with 0.5 levels, and the architecture is very simple so I don't think I can reduce the architecture complexity. I can paste the model if anyone likes.
My question is: Is there ever a situation where validation accuracy cannot be as high as the training accuracy? Is there a limitation based on the size of the dataset? Or is it ALWAYS possible for validation accuracies to match training accuracies and the network just needs the right parameters?