Why underfitting is called high bias and overfitting is called high variance?

I have been using terms like underfitting/overfitting and bias-variance tradeoff for quite some while in data science discussions and I understand that underfitting is associated with high bias and over fitting is associated with high variance. But what is the reason of such association or in terms of a model what is high bias and high variance, How can one understand it intuitively?

3 Answers

Let us assume our model to be described by $$y = f(x) +\epsilon$$, with $$E[\epsilon]=0, \sigma_{\epsilon}\neq 0$$. Let furthermore $$\hat{f}(x)$$ be our regression function, i.e. the function whose parameters are the ones that minimise the loss (whatever this loss is). Given a new observation $$x_0$$, the expected error of the model is $$E[(y-\hat{f}(x))^2|x=x_0].$$ This expression can be reduced (by means of more or less tedious algebra) to $$E[(y-\hat{f}(x))^2|x=x_0] = \sigma_{\epsilon}^2 + (E[\hat{f}(x_0)]-f(x_0))^2 + E[\hat{f}(x_0)-E[\hat{f}(x_0)]]^2$$ where the second term is the difference between the expected value of our estimator $$\hat{f}$$ and its true value (therefore the bias of the estimator) and the last term is the definition of variance.

Now for the sake of the example consider a very complex model (say, a polynomial with many parameters or similar) which you are fitting against the training data. Because of the presence of these many parameters, they can be adapted very closely to the training data to even the average out (because there is many of them); as a consequence the bias term is reduced drastically. On the other hand, though, it is generally the case that whenever you have many parameters their least square estimations come with high variance: as already mentioned, since they have been deeply adapted to the training data, they might not generalise well on new unseen data. Since we have many parameters (complex model) a small error in each of them sums up to a big error in the overall prediction.

The converse situation may happen when one has a model that is very static (imagine very few parameters): their variances do not sum up very much (because there is few of them) but the trade-off is that their estimation of the mean might not correspond closely to the true value of the regressor.

In the literature one refers to the former behaviour as overfit, to the latter as underfit. In the description I have given you can see that they may be related to the complexity of the model but need not necessarily be, namely you may as well have particularly complex models that do not necessarily overfit (because of the way they are constructed, one above all is random forest) and simple model that do not necessarily underfit (for instance linear regressions when the data are actually linear).

Check out the answer provided by Brando Miranda in the following quora question:

"High variance means that your estimator (or learning algorithm) varies a lot depending on the data that you give it."

"Underfitting is the “opposite problem”. Underfitting usually arises because you want your algorithm to be somewhat stable, so you are trying to restrict your algorithm to much in some way. This might make it more robust to noise but if you restrict it too much it might miss legitimate information that your data is telling you. This usually results in bad train and test errors. Usually underfitting is also caused by biasing you model too much."

https://www.quora.com/What-is-meant-by-high-variance-low-bias-and-why-does-this-lead-to-overfitting-in-machine-learning

How can one understand it intuitively?

Underfitting is called "Simplifying assumption" (Model is HIGHLY BIASED towards its assumption). your model will think linear hyperplane is good enough to classify your data which may not be true. consider you are shown a picture of cat 1000 times, Now you are blindfolded, No matter Whatever you are shown the 1001th time, probability that you will say cat is very high(You are HIGHLY BIASED that the next picture is also gonna be a cat). Its because you believe its gonna be a cat anyway. Here you are simplifying assumptions

In statistics, Variance informally means how far your data is spread out. Overfitting is you memorise 10 qns for your exam and on the next day exam, only one question has been asked in the question paper from that 10 you read. Now you will answer that one qn correctly just like in the book, but you have no idea what the remaining questions are(Question are HIGHLY VARIED from what you read). In overfitting, model will memorise the entire train data such that it will give high accuracy on train but will suck in test. Hope its helps