I am using a CNN to classify medical images. I am using a four convolutional layers with ReLU activation followed by a softmax layer. I am using rmsprop as the optimizer. The problem I am facing is the network's test accuracy increases and then goes down. This continues till the maximum number of epochs is reached. I read somewhere that this could be due to the fact that the network shifts its weights in favor of one class, and then has to do so in favor of the other class. I shuffled the test data, to see if this would help, but no such luck. Is there any remedy?
Based on the image you are sharing, the training accuracy continues to increase, the validation accuracy is changing around the 50%. I think either you do not have enough data to use neural network or the network is small to capture all the information, in both cases I feel there either under fitting or over fitting problem. Can you give us some information about the database so we can help more