Why underfitting is called high bias and overfitting is called high variance?

Question

I have been using terms like underfitting/overfitting and bias-variance tradeoff for quite some while in data science discussions and I understand that underfitting is associated with high bias and over fitting is associated with high variance. But what is the reason of such association or in terms of a model what is high bias and high variance, How can one understand it intuitively?

Answer 1

How can one understand it intuitively?

Underfitting is called "Simplifying assumption" (Model is HIGHLY BIASED towards its assumption). your model will think linear hyperplane is good enough to classify your data which may not be true. consider you are shown a picture of cat 1000 times, Now you are blindfolded, No matter Whatever you are shown the 1001th time, probability that you will say cat is very high(You are HIGHLY BIASED that the next picture is also gonna be a cat). Its because you believe its gonna be a cat anyway. Here you are simplifying assumptions

In statistics, Variance informally means how far your data is spread out. Overfitting is you memorise 10 qns for your exam and on the next day exam, only one question has been asked in the question paper from that 10 you read. Now you will answer that one qn correctly just like in the book, but you have no idea what the remaining questions are(Question are HIGHLY VARIED from what you read). In overfitting, model will memorise the entire train data such that it will give high accuracy on train but will suck in test. Hope its helps

Answer 2

Let us assume our model to be described by $y = f(x) +\epsilon$, with $E[\epsilon]=0, \sigma_{\epsilon}\neq 0$. Let furthermore $\hat{f}(x)$ be our regression function, i.e. the function whose parameters are the ones that minimise the loss (whatever this loss is). Given a new observation $x_0$, the expected error of the model is $$ E[(y-\hat{f}(x))^2|x=x_0]. $$ This expression can be reduced (by means of more or less tedious algebra) to $$ E[(y-\hat{f}(x))^2|x=x_0] = \sigma_{\epsilon}^2 + (E[\hat{f}(x_0)]-f(x_0))^2 + E[\hat{f}(x_0)-E[\hat{f}(x_0)]]^2 $$ where the second term is the difference between the expected value of our estimator $\hat{f}$ and its true value (therefore the bias of the estimator) and the last term is the definition of variance.

Now for the sake of the example consider a very complex model (say, a polynomial with many parameters or similar) which you are fitting against the training data. Because of the presence of these many parameters, they can be adapted very closely to the training data to even the average out (because there is many of them); as a consequence the bias term is reduced drastically. On the other hand, though, it is generally the case that whenever you have many parameters their least square estimations come with high variance: as already mentioned, since they have been deeply adapted to the training data, they might not generalise well on new unseen data. Since we have many parameters (complex model) a small error in each of them sums up to a big error in the overall prediction.

The converse situation may happen when one has a model that is very static (imagine very few parameters): their variances do not sum up very much (because there is few of them) but the trade-off is that their estimation of the mean might not correspond closely to the true value of the regressor.

In the literature one refers to the former behaviour as overfit, to the latter as underfit. In the description I have given you can see that they may be related to the complexity of the model but need not necessarily be, namely you may as well have particularly complex models that do not necessarily overfit (because of the way they are constructed, one above all is random forest) and simple model that do not necessarily underfit (for instance linear regressions when the data are actually linear).

Answer 3

Check out the answer provided by Brando Miranda in the following Quora question:

"High variance means that your estimator (or learning algorithm) varies a lot depending on the data that you give it."

"Underfitting is the “opposite problem”. Underfitting usually arises because you want your algorithm to be somewhat stable, so you are trying to restrict your algorithm too much in some way. This might make it more robust to noise but if you restrict it too much it might miss legitimate information that your data is telling you. This usually results in bad training and test errors. Usually underfitting is also caused by biasing your model too much."

https://www.quora.com/What-is-meant-by-high-variance-low-bias-and-why-does-this-lead-to-overfitting-in-machine-learning

Answer 4

A model based on simple assumptions (biased) will probably fit the data badly (under-fitting) whereas a more complex, flexible model that can vary more may fit the training data so well (over-fitting) that it becomes less good at predicting new data.

Answer 5

Let's say the problem is predicting whether you will pass or fail in subject C based on your grades in subject A and subject B. Suppose you had a model which takes inputs $x$ and outputs predictions $y$. For each $x$, there is a true target $t$ (i.e. what the "correct" prediction is). So $x$ are the grades in course A and course B, and $y$ and $t$ are binary, indicating pass or fail.

Suppose you train your model on a dataset $D$. The output of your model $y$ for any given $x$ will differ based on what $D$ you train it on. (i.e. sampling all the students with student id's ending in 5 vs. all the students with student id's ending in 0). In this sense, $y$ is a random variable, where the randomness comes from the choice of the dataset $D$. If you overfit, you will memorize the peculiar aspects of the dataset that do not generalize. So if you are provided with different $D$'s, and trained your model on all of them, for a fixed $x$, your prediction $y$ will vary a lot depending on which $D$ you trained your model on (since the model remembers all the details about each $D$). The variability of $y$ is due to overfitting.

Next consider the case where you have a very basic model, that just takes the average of the two courses A and B and if it's above some threshold, predicts the student will pass subject C. Suppose course A was actually English, course B was Differential Geometry, and course C was Linear Algebra, and the optimal prediction given $x$ is to predict $y^*$. One would expect students did well in course B could also do well in course C. You can think of $y^*$ in this scenario as having lots to do with the grades in course B.

But your model, being as simplistic as it is, on average, predicts $E[y|x]$, since it routinely fails to capture the importance of subject B and the unimportance of subject A for predicting subject C.Your model is biased towards predicting $E[y|x]$ rather than $y^*$, since it is underfitting (i.e. failing to capture the relevant structure of the data that helps it make good predictions on average).