I am developing an ANN from scratch which classifies MNIST digits.

These are the curves I get using only one hidden layer composed of 100 neurons activated by ReLU function. The output's neurons are activated by the softmax function:


Is it correct that training and validation loss are almost identical?

  • $\begingroup$ Looks like you are configuring something wrong. Perhaps your training and validation set are the same, or maybe your computation of the metrics is accidentally reusing values from the other set. Please share your code. $\endgroup$ Jul 5 at 10:03

2 Answers 2


Informal explanation:

We usually expect the validation loss to be higher than the training loss due to overfitting. This occurs because the model fits random noise in the training samples, it "memorizes" them.

However, to memorize the samples the model needs a sufficient capacity (lets say number of parameters), that allows it to memorize the samples. In your example of a single-layer MLP, we have a very low model capacity, so we wouldn't expect overfitting to occur, which your experiments confirm.

If you're interested in a more rigorous explanation of this phenomenon, you should look into the "Bias-variance tradeoff".


Summary :

If validation loss >> training loss you can call it overfitting
If validation loss  > training loss you can call it some overfitting.
If validation loss  < training loss you can call it some underfitting.
If validation loss << training loss you can call it underfitting.
If validation loss == training loss perfect fit

Following are the three cases of model fit:

  1. Underfitting

This is the only case where loss > validation loss, but only slightly, if loss is far higher than validation loss, please post your code and data so that we can have a look at

  1. Overfitting

loss << validation loss

This means that your model is fitting very nicely the training data but not at all the validation data, in other words it's not generalizing correctly to unseen data

  1. Perfect fitting

loss == validation loss

If both values end up to be roughly the same and also if the values are converging (plot the loss over time) then chances are very high that you are doing it right

  • $\begingroup$ Copy and pasted from another answer $\endgroup$
    – tail
    Nov 3, 2022 at 16:35
  • $\begingroup$ This explanation of "underfitting" is incorrect. Underfitting refers to the case where the model can't fit the data, so both the loss and validation loss would be high, see ibm.com/topics/underfitting . "perfect fitting" is generally not used in this way either $\endgroup$ Feb 10 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.