Training and validation loss are almost the same (perfect fit?)

Question

I am developing an ANN from scratch which classifies MNIST digits.

These are the curves I get using only one hidden layer composed of 100 neurons activated by ReLU function. The output's neurons are activated by the softmax function:

Is it correct that training and validation loss are almost identical?

Answer 1

Informal explanation:

We usually expect the validation loss to be higher than the training loss due to overfitting. This occurs because the model fits random noise in the training samples, it "memorizes" them.

However, to memorize the samples the model needs a sufficient capacity (lets say number of parameters), that allows it to memorize the samples. In your example of a single-layer MLP, we have a very low model capacity, so we wouldn't expect overfitting to occur, which your experiments confirm.

If you're interested in a more rigorous explanation of this phenomenon, you should look into the "Bias-variance tradeoff".

Answer 2

Summary :

If validation loss >> training loss you can call it overfitting
If validation loss  > training loss you can call it some overfitting.
If validation loss  < training loss you can call it some underfitting.
If validation loss << training loss you can call it underfitting.
If validation loss == training loss perfect fit

Following are the three cases of model fit:

1. Underfitting

This is the only case where loss > validation loss, but only slightly, if loss is far higher than validation loss, please post your code and data so that we can have a look at

2. Overfitting

loss << validation loss

This means that your model is fitting very nicely the training data but not at all the validation data, in other words it's not generalizing correctly to unseen data

3. Perfect fitting

loss == validation loss

If both values end up to be roughly the same and also if the values are converging (plot the loss over time) then chances are very high that you are doing it right