Tensorflow oscillating Test and Train Accuracy?

Question

I have implemented a CNN with images as input and 101 classes as output. I have applied mean subtraction and normalization to the input before giving it as input to the network.

I have also experimented with multiple optimizers (ADAM & SGD) and also changed the learning rate (0.1,0.01,0,001) and also batch sizes (19,32,64,128,200). But with all of these cases I had observed a similar pattern.

I dont think the network is learning much and more importantly the accuracy oscillating between a few values.

I had initially thought it was because of the batch size but I was wrong. I have built and trained the exact same network with same inputs with caffe. And achieved a 55% accuracy(check for working of network model).

My guess is the implementation went wrong somewhere but not sure where.

Mycode

Output

This particular output is with lr=0.001 batch_size=200 and ADAM optimizer

Iter 2000, Minibatch Loss= 12560.197266, Training Accuracy= 0.01000
Testing Accuracy: 0.025
Iter 4000, Minibatch Loss= 4587.021484, Training Accuracy= 0.00500
Testing Accuracy: 0.005
Iter 6000, Minibatch Loss= 2196.316406, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 8000, Minibatch Loss= 918.784119, Training Accuracy= 0.01500
Testing Accuracy: 0.0
Iter 10000, Minibatch Loss= 315.327911, Training Accuracy= 0.01000
Testing Accuracy: 0.0
Iter 12000, Minibatch Loss= 102.744026, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 14000, Minibatch Loss= 59.417763, Training Accuracy= 0.00500
Testing Accuracy: 0.005
Iter 16000, Minibatch Loss= 33.026432, Training Accuracy= 0.00500
Testing Accuracy: 0.02
Iter 18000, Minibatch Loss= 30.361868, Training Accuracy= 0.02000
Testing Accuracy: 0.005
Iter 20000, Minibatch Loss= 33.944790, Training Accuracy= 0.01000
Testing Accuracy: 0.02
Iter 22000, Minibatch Loss= 15.029477, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 24000, Minibatch Loss= 13.751340, Training Accuracy= 0.02500
Testing Accuracy: 0.02
Iter 26000, Minibatch Loss= 9.085027, Training Accuracy= 0.01500
Testing Accuracy: 0.015
Iter 28000, Minibatch Loss= 14.407433, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 30000, Minibatch Loss= 11.090684, Training Accuracy= 0.02500
Testing Accuracy: 0.02
Iter 32000, Minibatch Loss= 9.385777, Training Accuracy= 0.02000
Testing Accuracy: 0.03
Iter 34000, Minibatch Loss= 12.902111, Training Accuracy= 0.01500
Testing Accuracy: 0.02
Iter 36000, Minibatch Loss= 5.580408, Training Accuracy= 0.02000
Testing Accuracy: 0.005
Iter 38000, Minibatch Loss= 11.641799, Training Accuracy= 0.02000
Testing Accuracy: 0.03
Iter 40000, Minibatch Loss= 6.941562, Training Accuracy= 0.01500
Testing Accuracy: 0.02
Iter 42000, Minibatch Loss= 6.781301, Training Accuracy= 0.01000
Testing Accuracy: 0.0
Iter 44000, Minibatch Loss= 6.840708, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 46000, Minibatch Loss= 5.501908, Training Accuracy= 0.01000
Testing Accuracy: 0.03
Iter 48000, Minibatch Loss= 4.667253, Training Accuracy= 0.03000
Testing Accuracy: 0.015
Iter 50000, Minibatch Loss= 4.613351, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 52000, Minibatch Loss= 7.234328, Training Accuracy= 0.02000
Testing Accuracy: 0.01
Iter 54000, Minibatch Loss= 8.120478, Training Accuracy= 0.02500
Testing Accuracy: 0.01
Iter 56000, Minibatch Loss= 6.804164, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 58000, Minibatch Loss= 5.060484, Training Accuracy= 0.03000
Testing Accuracy: 0.02
Iter 60000, Minibatch Loss= 4.937673, Training Accuracy= 0.02000
Testing Accuracy: 0.015
Iter 62000, Minibatch Loss= 8.758166, Training Accuracy= 0.01000
Testing Accuracy: 0.02
Iter 64000, Minibatch Loss= 4.616434, Training Accuracy= 0.01500
Testing Accuracy: 0.025

Answer 1

From your results I see that the network is working good: the loss function is reducing, that means that the calculated results are approaching the expected ones. But in facts the accuracy is not, so there should be some error in the way you calculate it.

The fact that the loss function start to oscillate at some point (~after 20000 iterations) is expected, and that is the reason why usually one lower the learning rate after some epochs, or use some other technique of annealing.

Answer 2

It seems that after some epochs your training oscillates. I guess the reason is that the learning rate is high. Try to set the decay parameter of Adam optimizer to a number more than one and resume training.