I am using a CNN to classify medical images. I am using a four convolutional layers with ReLU activation followed by a softmax layer. I am using rmsprop as the optimizer. The problem I am facing is the network's test accuracy increases and then goes down. This continues till the maximum number of epochs is reached. I read somewhere that this could be due to the fact that the network shifts its weights in favor of one class, and then has to do so in favor of the other class. I shuffled the test data, to see if this would help, but no such luck. Is there any remedy?
Based on the image you are sharing, the training accuracy continues to increase, the validation accuracy is changing around the 50%. I think either you do not have enough data to use neural network or the network is small to capture all the information, in both cases I feel there either under fitting or over fitting problem. Can you give us some information about the database so we can help more
With the screenshot you shared, the difference between the training accuracy and the validation accuracy is huge. 90 to 50 is a big gap, which means your model is actually overfitting. It's learning to well from the training data, that it's unable to generalize it with real time or nevee seen data. My guess is the reason it's fluctuating is because of that. Try to increase your dataset, if it's an image classification, try image augmentations and try not to make the model too complex. I'm sure it'll help.