1
$\begingroup$

I am always getting lost when dealing with these terms. Especially being asked questions about the relationship such as underfitting-high bias (low variance) or overfitting-high variance (low bias). Here is my argument:

  1. From wiki:

In statistics, **overfitting is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably".1 An overfitted model is a statistical model that contains more parameters than can be justified by the data.2 The essence of overfitting is to have unknowingly extracted some of the residual variation (i.e. the noise) as if that variation represented underlying model structure.[3]:45

Underfitting occurs when a statistical model cannot adequately capture the underlying structure of the data. An under-fitted model is a model where some parameters or terms that would appear in a correctly specified model are missing.2

Based on this definition, both under-fitting and over-fitting are biased. I really could not tell which one has a higher bias. Furthermore, "too closely in training data" but "fail in test data" does not necessarily mean high variance.

  1. From Stanford CS229 Notes

High Bias ←→ Underfitting High Variance ←→ Overfitting Large σ^2 ←→ Noisy data

If we define underfitting and overfitting directly based on High Bias and High Variance. My question is: if the true model f=0 with σ^2 = 100, I use method A: complexed NN + xgboost-tree + random forest, method B: simplified binary tree with one leaf = 0.1 Which one is overfitting? Which one is underfitting?

$\endgroup$

4 Answers 4

2
$\begingroup$

You can look into the following figure to get an graphical intuition. Visit the source for detailed illustration.

Source : https://www.kaggle.com/getting-started/166897

$\endgroup$
2
  • $\begingroup$ Thanks for this information. I am OK with the conclusions based on the examples they provide: Overfitting has high variance and underfitting has high bias. $\endgroup$
    – rain keyu
    Aug 17, 2021 at 1:49
  • $\begingroup$ But what I could not follow is how could we claim these conclusions are ALWAYS true? Overfitting ALWAYS has high variance and underfitting ALWAYS has high bias. $\endgroup$
    – rain keyu
    Aug 17, 2021 at 1:50
1
$\begingroup$

"An overfitted model is a statistical model that contains more parameters than can be justified by the data"

This is an idea that is well past it's ``best before date''. In the early days of computational statistics, the most common way of controlling the complexity of a model was to limit the number of parameters (e.g. feature selection for linear models). But that hasn't been true for a long time. The early 1970s saw the introduction of ridge-regression, which introduced the idea of regularisation to control the capacity of a model. It adds a penalty term to the training criterion that penalises large magnitudes of weights. This is mathematically equivalent to placing an upper bound on the squared norm of the weight vector. This implements a simple form of "structural risk minimisation" (c.f. SVM) - if we increase the bound slightly, the model can do anything that it could do before, plus a few other things. So the regularisation parameter forms a set of nested model classes of increasing complexity. This means we can have over-parameterised model that don't over-fit, and indeed that is pretty much what modern machine learning algorithms are all about.

So one thing that would reduce the confusion is not to comflate over-fitting (fitting the data too closely) with over-parameterisation (having more parameters than strictly necessary to represent the underlying structure of the data).

When we "fit" a model, we generally mean we adjust the parameters of the model so that it's output more closely resembles the calibration data according to some criterion that measures the data "misfit". So over-fitting basically means reducing the value "data-misfit" function too much. How much is "too much"? If it makes generalisation performance worse, that is "too much".

If you can make generalisation performance better by using a more complex model (or training it for longer) then your model is currently "underfitting" the data.

Over/under-fitting is not defined in terms of bias or variance, it is defined in terms of the value of the training error (the data misfit) and the generalsiation properties of the model. Bias and variance are useful terms for understanding the consequences of over- and under-fitting. The diagrams help though.

$\endgroup$
2
  • $\begingroup$ Thanks for your input. I agree that overfitting is the situation that makes generalization performance worse such as "MSE". So we may have a perfect 0 MSE in the train set but with a very high MSE in the test set. If this is the case, how could we say that this high MSE (overfitting) is coming from high variance, not from high bias? This is always the missing puzzle in my mind. $\endgroup$
    – rain keyu
    Aug 18, 2021 at 6:49
  • $\begingroup$ @rainkeyu I'd view it more as variance being the consequence of having a model with excess capacity, rather than the cause. Variance is a property of the estimator, whereas it is a particular solution that is over-fit. $\endgroup$ Aug 18, 2021 at 7:19
0
$\begingroup$

I'll try to make it as simple as possible. Underfitting is when you have high bias and high variance in your model. So the model learns nothing from the training data (low training score aka high bias) and predicts poorly on the test data (low variance). You get underfitting when your model is too simple for the data or the data is too complex for your model to understand.

Here is an example of underfitting:-

enter image description here

As we can see both train and test scores are poor which means the model learns nothing from the data and performs/predicts nothing on the test set.

Techniques to reduce underfitting :

  1. Increase model complexity

  2. Increase number of features, performing feature engineering

  3. Remove noise from the data.

  4. Increase the number of epochs or increase the duration of training to get better results.

Overfitting is when you have low bias and high variance. So the model learns everything from the training dataset (high train score aka low bias) but is not able to perform good on the test set (low test score aka high variance) You get overfitting when your model is too complex for the data or your data is too simple for the model.

Here is an example of overfitting:-

enter image description here

As we can see, the training loss decreases initially (low bias) but the test/validation loss, after decreasing to a certain point starts gradually increasing. Also apparent is the large gap between train and test lines.

Techniques to reduce overfitting :

  1. Increase training data.

  2. Reduce model complexity.

  3. Early stopping during the training phase (have an eye over the loss over the training period as soon as loss begins to increase stop training).

  4. Ridge Regularization and Lasso Regularization

  5. Use dropout for neural networks to tackle overfitting.

Hope that clears the confusion!

$\endgroup$
7
  • $\begingroup$ Thanks for your comments. If so, why do people usually say that a decision tree with low-rank is underfitting, and it has high bias but low variance. $\endgroup$
    – rain keyu
    Aug 15, 2021 at 19:21
  • $\begingroup$ Btw, if you use loss (like mean square error) to define "underfitting" and "overfitting", please check the equation in Stanford CS229 Notes. That loss = irreducible error + bias^2 + model variance. An increase of loss in the test dataset could come from either high bias or high model variance. $\endgroup$
    – rain keyu
    Aug 15, 2021 at 20:07
  • $\begingroup$ @rainkeyu if by low rank you mean a shallow tree then yes, low rank trees are prone to underfitting as (not always the case though) if one doesn't tune the max_depth properly. $\endgroup$
    – spectre
    Aug 16, 2021 at 5:34
  • $\begingroup$ Yes increase in loss can be due to high bias or variance. You can also choose another metric if you are not comfortable with loss such as f1_score. It is much more suitable for imbalanced and multiclass datasets. PS: yes you are right, underfitting means high bias and low variance. It was a mistake, corrected it. $\endgroup$
    – spectre
    Aug 16, 2021 at 5:41
  • $\begingroup$ "Also apparent is the large gap between train and test lines." this does not necessarily indicate over-fitting. Some classifiers perform best when they get zero training set error. $\endgroup$ Aug 16, 2021 at 14:38
0
$\begingroup$

Personally I find Victor Lavrenko's explanation of underfitting and overfitting the most intuitive and concise definition:

This definition is very useful for at least these two points:

  1. This is not always an easy task to measure model's complexity (check) as presented in most of the diagrams that explain this concept

  2. You avoid the pitfall of comparing the same model's metrics in train and test set as described here:

...We can identify if a machine learning model has overfit by first evaluating the model on the training dataset and then evaluating the same model on a holdout test dataset. If the performance of the model on the training dataset is significantly better than the performance on the test dataset, then the model may have overfit the training dataset....

This situation is not clearly defined since there is no a "standard" difference between train and test error that can guarantee that your model is overfitting, so there is no such thing as significantly better on the test set (as far as I know)

But of course I'm not saying that you should no care about training vs testing error.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.