1
$\begingroup$

There are several scenarios that can occur while training and validating:

  1. Both training loss and validation loss are decreasing, with the training loss lower than the validation loss.
  2. Both training loss and validation loss are decreasing, with the training loss higher than the validation loss.
  3. Training loss decreasing, but validation loss is increasing.

I am aware that overfitting occurs in scenario 3, but does overfitting occur in scenario 1? If so, does this mean that overfitting only occurs when either scenario 1 or scenario 3 occur? Otherwise, if overfitting only occurs in scenario 3, does this mean that overfitting only occurs when validation loss is increasing?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Scenario 1 is a perfectly valid situation, where you are gradually polishing your model. What matters is that the validation loss is decreasing. Typically, the training loss is lower than the validation loss even if you are doing things right... In general, "overfitting" does not have a rigorous, mathematical definition, to the best of my knowledge. $\endgroup$ – stans Feb 2 at 5:57
  • 1
    $\begingroup$ Completely agree to @stans, there's not a very rigorous definition but as long as your validation loss decreases you are going to be ok. This is already discussed a bit in datascience.stackexchange.com/questions/15242/… and datascience.stackexchange.com/questions/32306/… $\endgroup$ – David Masip Feb 2 at 8:26
  • $\begingroup$ @DavidMasip the second answer in the first question that you linked states "If your training loss goes under you validation loss, you are overfitting, even if validation is still dropping". Is this misleading? $\endgroup$ – mhdadk Feb 2 at 9:22
  • 1
    $\begingroup$ to me it is, yeah. I only consider overfitting when the validation loss increases, but the definition is not very well defined and some people may have a looser definition of it $\endgroup$ – David Masip Feb 2 at 9:23
  • $\begingroup$ @DavidMasip Is it accurate to say that the point where validation loss stops decreasing and starts increasing is where both the bias and the variance of the estimator in question are minimized? $\endgroup$ – mhdadk Feb 2 at 9:25
1
$\begingroup$

In my opinion, only case 3 should be considered overfitting. As @stans has mentioned, there is not a very rigorous definition of overfitting so other people might think differently.

I wouldn't say the point where the validation loss stops decreasing is where bias and variance are minimized since there is a trade-off between bias and variance:

  • A constant model will have very low variance, but very high bias.
  • An overfitting model will have very low bias, but very high variance.

The point where the validation loss starts increasing can be considered optimal in terms of the sum of squared bias and variance, that is, an optimum of the generalization error.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.