I'm training a cNN for binary classification. I used a batch size of 128, and the loss is decreasing and accuracy is increasing over epoches. The accuracy reached over 0.99 eventually, and the loss reached below 0.3. But after a few more epoches, the model converges to loss 0.6 and accuracy 0.5. An inspection of the model shows that it always predicts 0.5. I'm using binary cross entropy as the loss function. For each epoch, all data points are shuffled. I'm using SGD, learning rate is 0.01.

Am I hitting a local minimum with low accuracy rate but fairly good loss value? What is the suggested approach to deal with this? Also, why is it possible to have a low loss function with one single predicted class?

  • $\begingroup$ Is there any class imbalance? $\endgroup$ – ncasas Mar 1 '18 at 19:02
  • $\begingroup$ @ncasas No, approximately 50% have 1 class and the other 50% have the the other class. $\endgroup$ – xuhdev Mar 1 '18 at 19:03
  • $\begingroup$ Are you using binary cross entropy as loss function? $\endgroup$ – ncasas Mar 1 '18 at 19:32
  • $\begingroup$ @ncasas Yes, I am. $\endgroup$ – xuhdev Mar 1 '18 at 21:30
  • $\begingroup$ What kind of data shuffling are you using before batching? $\endgroup$ – ncasas Mar 2 '18 at 9:09

From your description, it might be an issue of too high learning rate. When this happens, the weights cannot reach closer to the minimum and the loss does not go down.

Accordingly, I would suggest reducing the learning rate to 0.001 and using Adam.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.