Constant validation loss & accuracy, training accuracy fluctuates

Question

I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my training accuracy fluctuates between 1 and 0.96875. The validation accuracy is more or less constant between [0.985, 0.986]. The base learning rate is 0.01 and decreased upto 0.00001. As far as I can tell visually training loss doesn't exactly fluctuate between two numbers as such but is in the range [0.02, 0.09] for most part except a few occasional spike.

My question is what can i infer from this? (1) Overfit? (2) Model has converged? If not should I reduce the learning rate? (3) Is the model stuck at a local minima? I am using softmax-with-loss as my loss layer.

Answer 1

Did the loss decrease up to a certain point and then start fluctuating between .02 and .09? If so you may have just reached your convergence and now it's oscillating due to the size of the learning rate being too large in relation to the size of the minima you have hit. Reduce your learning rate and see where it ends up, my guess is that you will get close to .02 this time around, unless you have truly hit a local minima, in which case you may go lower.