Why an increasing validation loss and validation accuracy signifies overfitting?

Question

When I train a neural network, I observe an increasing validation loss, while at the same time, the validation accuracy is also increased.

I have read explanations related to the phenomenon, and it seems an increasing validation loss and validation accuracy signifies an overfitted model.

However, I have not really grokked the reasons why an increasing validation loss and validation accuracy signifies an overfitting.

Could you please give the explanations behind this phenomenon?

Answer 1

Overfitting is when your model is memorizing the training data, and therefore it is not able to generalize to unseen data, like the validation set.

This way, you see overfitting when the training loss decreases but the validation loss increases.

The fact that, while the validation loss increases, the validation accuracy also increases is a separate effect. To understand it, we should take into account that the accuracy is computed comparing the highest output of the softmax with the correct label, and it does not vary depending on the actual value of the softmax output, which is what the validation loss takes into account.

Answer 2

This might be probably going on in the Learning Process.

$ \begin{array} {|r|r|} \hline 0.85 &0.85 &0.85 &\color{red} {0.45} &\color{red} {0.45} \\ \hline \end{array} $

$ \begin{array} {|r|r|} \hline 0.75 &0.75 &0.75 &0.55 &\color{red} {0.45} \\ \hline \end{array} $

$ \begin{array} {|r|r|} \hline 0.65 &0.65 &0.65 &0.55 &0.55 \\ \hline \end{array} $

Accuracy is increasing but the Model is less confident of the predictions.

Another important point to note is that Metrics is not a part of the learning process, it is calculated based on a Threshold and a different function/formula. While Loss is directly driving the Learning.

So, if Val loss is moving up, it means that the Model is indicating that it is becoming a function which better represents the Train data pattern and not so good the Validation data pattern. Hence the Overfit.