My accuracy changes throughout every epoc but the val_acc at the end of each epoc stays the same

Question

I am training a transfer learning CNN on 161 pictures that have been augmented into 966 photos, 4 classes. I am training on a balanced data set, so there are 52 images of each class and also in the validation and test data, I am keeping the data balanced.

I am using MobileNet as a transfer learning network, and actually only using the first 8 layers as non-trainable layers, the rest are trainable and then I added another 3 dense layers with activation functions of relu and the last one softmax so as not to encounter the vanishing gradient descent.

Currently using the adam compiler, with a learning rate of 1e-6, beta_1=0.9, beta_2=0.999 decay=0.0000001 and amsgrad=True. Loss is categorical_crossentropy and metrics are accuracy.

Now, I am running 35 epochs, and with each epoch the accuracy changes from 0 to 40% (it changes every epoch, so far acc=0.4 was the highest that I saw and it doesn't always start from 0, it changes) but at the end of every single epoch the validation accuracy is 0.2 and the following epoch starts, it seems like, randomly. Why is that and how do I fix that? It's as if each epoch starts training it from the beginning. In addition, the actual loss hasn't changed much at all, it started, first batch first epoch with a loss of 1.3866 and it fluctuated max +-0.01 (not even).

Thanks in advance!

Answer 1

Your model complexity is high enough to learn features and give you results better than 20%. Although I would suggest to increase your learning rate to 1e-3 or 1e-4.

The accuracy in epoch refers to accuracy for the batch its training on so it might fluctuate. But is val accuracy and loss is same that means the model isn't learning distiguishable features. It could mean the problem is hard.