Why is my validation loss going up while my validation accuracy also goes up?

Question

Scenario: I've been training a CNN for the cifar10 dataset. I'm using tensorflow, and a CNN with 12 conv layers and 1 dense layer before a softmax dense layer. I'm using data augmentation as well with batch normalization.

After a few hundred epochs I archieved a maximum of 92.73 percent accuracy on the validation set.

My problem:

• Validation loss goes up slightly as I train more.
• While validation loss goes up, validation accuracy also goes up.

Example:

• One epoch gave me a loss of 0.295, with a validation accuracy of 90.5%. My best epoch for validation accuracy gave me 92.73% with a validation loss of 0.33.

Question:

• Why is my validation accuracy increasing while my validation loss is going up?
• Should I use a loss metric diferent to cross_entropy?

Answer 1

It is possible that accuracy and cross entropy increase at the same. For example, for a positive sample your predicted probability could go from 0.4 to 0.1 (still wrong but worse increasing the entropy loss) and for another positive sample your predicted probability could change from 0.49 to 0.51 (changes from wrong to right improving accuracy). The first case would increase your entropy loss while the second would improve accuracy without significantly changing the entropy loss.

It's a little difficult to say if cross entropy is not a good metric for your case without knowing any details. But most likely you would want to stick with it for a couple of reasons. For example, the cross entropy gives better probability estimates and has nice properties for training with gradient descent (smooth gradients, training doesn't stall for large values because the log off-sets the exp in the sigmoid activation etc.).