# Does overfitting depend only on validation loss or both training and validation loss?

There are several scenarios that can occur while training and validating:

1. Both training loss and validation loss are decreasing, with the training loss lower than the validation loss.
2. Both training loss and validation loss are decreasing, with the training loss higher than the validation loss.
3. Training loss decreasing, but validation loss is increasing.

I am aware that overfitting occurs in scenario 3, but does overfitting occur in scenario 1? If so, does this mean that overfitting only occurs when either scenario 1 or scenario 3 occur? Otherwise, if overfitting only occurs in scenario 3, does this mean that overfitting only occurs when validation loss is increasing?

• Scenario 1 is a perfectly valid situation, where you are gradually polishing your model. What matters is that the validation loss is decreasing. Typically, the training loss is lower than the validation loss even if you are doing things right... In general, "overfitting" does not have a rigorous, mathematical definition, to the best of my knowledge. Feb 2, 2021 at 5:57
• Completely agree to @stans, there's not a very rigorous definition but as long as your validation loss decreases you are going to be ok. This is already discussed a bit in datascience.stackexchange.com/questions/15242/… and datascience.stackexchange.com/questions/32306/… Feb 2, 2021 at 8:26
• @DavidMasip the second answer in the first question that you linked states "If your training loss goes under you validation loss, you are overfitting, even if validation is still dropping". Is this misleading? Feb 2, 2021 at 9:22
• to me it is, yeah. I only consider overfitting when the validation loss increases, but the definition is not very well defined and some people may have a looser definition of it Feb 2, 2021 at 9:23
• @DavidMasip Is it accurate to say that the point where validation loss stops decreasing and starts increasing is where both the bias and the variance of the estimator in question are minimized? Feb 2, 2021 at 9:25