7
$\begingroup$

I'm reading "An Introduction to Statistical Learning: With Applications in R".

In the Paragraph 2.2.2 The Bias-Variance Trade-Off, the authors say:

enter image description here

I'm not able to understand why the bias tends to initially decrease faster than the variance increases. Can you help me ? Both rigorous and intuitive explanations are greatly appreciated

$\endgroup$

1 Answer 1

7
$\begingroup$

One way to look at this is through the idea of under-/overfitting

First off, here is a sketch of the generally observed relationship between bias and variance, in the context of model size/comlpexity:

bias-variance trade-off

Say you have a model which is learning quite well, but your test accuracy seems to be pretty low: 80%. The model is essentially not doing a great job of mapping input features to outputs. We have a high bias. But for a wide variety of input (assuming a good test set), we consistently obtain this 20% error; we have a low variance. We are underfitting

Now we decide to use a bigger model (e.g. a deep neural network), which is able to capture more details of the feature space and so maps inputs to outputs more accurately. We now have an improved test accuracy: 95%. At the same time, we notice that several runs of the model produce different results; sometime we have 4% error, and sometimes 6%. We have introduced a higher amount of variance. We are perhaps somewhere around the optimum model complexity shown on the graph above.

You say ok... let's create a monolithic neural network. It totally nails training and ends with a perfect accuracy: 100%. However, the test accuracy now drops to 90%! So we have zero bias, but a large variance. We are overfitting. The model is almost as good as a look-up table for training data, but doesn't generalise at all when it sees new samples. Intuitively, that 10% error corresponds to a difference in distribution between the training and test sets used $\rightarrow$ the model knows the training distribution in extreme detail, some of which do not apply to the test set (i.e. the reality).

In summary: The bias tends to decrease faster than the variance increases, because you can likely still make a more competitive model for your dataset; the model is underfitting. It is like the low-hanging fruit that you can easily get - so an incremental improvement on the red curve above gives a big decrease in bias (increase in performance). Obviously that pattern cannot go on indefinitely, with each increment in model complexity, you get a lower increase in performance; i.e. you have diminishing returns. Furthermore, as you begin to overfit, the model becomes less able to generalise and so exhibits larger errors on unseen data; variance is creeping in.


For some more intuition between bias/variance in machine learning, I'd recommend this talk by Andrew Ng. There is also a text summary of the talk, for a quicker overview.

For a brief but more mathematical explaination, head over to this post of Cross-Validated. The second answer there is very recent and is perhaps better than the (old) accepted answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.