When I train a neural network, I observe an increasing validation loss, while at the same time, the validation accuracy is also increased.

I have read explanations related to the phenomenon, and it seems an increasing validation loss and validation accuracy signifies an overfitted model.

However, I have not really grokked the reasons why an increasing validation loss and validation accuracy signifies an overfitting.

Could you please give the explanations behind this phenomenon?


2 Answers 2


Overfitting is when your model is memorizing the training data, and therefore it is not able to generalize to unseen data, like the validation set.

This way, you see overfitting when the training loss decreases but the validation loss increases.

The fact that, while the validation loss increases, the validation accuracy also increases is a separate effect. To understand it, we should take into account that the accuracy is computed comparing the highest output of the softmax with the correct label, and it does not vary depending on the actual value of the softmax output, which is what the validation loss takes into account.


This might be going on in the learning process.

$ \begin{array} {|r|r|} \hline 0.85 &0.85 &0.85 &\color{red} {0.45} &\color{red} {0.45} \\ \hline \end{array} $

$ \begin{array} {|r|r|} \hline 0.75 &0.75 &0.75 &0.55 &\color{red} {0.45} \\ \hline \end{array} $

$ \begin{array} {|r|r|} \hline 0.65 &0.65 &0.65 &0.55 &0.55 \\ \hline \end{array} $

The accuracy is increasing but the model is less confident of the predictions.

Another important point to note is that metrics are not a part of the learning process, they are calculated based on a different function/formula while the loss is directly driving the learning.

So, if validation loss is moving up, it means that the model is indicating that it is becoming a function which better represents patterns in the training data and not patterns in the validation data. Hence the overfitting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.