Linear machine learning algorithms “often” have high bias/low variance?

Question

In this blog, which explains the meaning of bias and variance in machine learning, there's a line under the heading "Bias-Variance Trade-Off" which says:

Parametric or linear machine learning algorithms often have a high bias but a low variance.

I know that there's an "often" in the first sentence, but how can it be true?
If a linear ML algorithm has high bias how can we expect it to have low variance?

Answer 1

The bias is error of the model in the training stage. If your goal is, let's say, a 96% of accuracy, and you get 90%, you have a bias of 6%. The variance is the error difference between training and validation, so, taking the same example, if you get a validation accuracy of 80%, you have 10% variance.

A linear ML algo won't work well on non-linear cases, which are most of the real cases. Then, often, a linear model will perform bad on the training data (high bias) and equally bad on the validation set (low variance). In this scenario, the model is underfitting the data, it is too simple.

As an example:

• Desired accuracy: 95%
• Training accuracy: 65%
• Validation accuracy: 62%

Answer 2

The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more likely to produce bias-type errors rather than variance-type. Another way of thinking about it is that the way that most linear models will give you broadly correct predictions overall, instead of specifically accurate or wildly inaccurate predictions. Obviously that's a super simplistic reading, but I've found an intuitive way to approach. Essentially - the way that linear models are constructed will lead them to predict more generally, instead of reflecting every type of twist and turn in the data. This is partially a function of the models themselves, and partially due to linear models being the first line of attack with smaller datasets.

If you think about the formula, one way to interpret is that the intercept generalizes the model's predictions. That being said, with enough variables and certain types of data, it's very possible to overtrain a linear model (which will tip it towards variance-type errors).

I like to think of the bias/variance tradeoff question as being one with three factors, instead of just the two in the title. I'll use the short definitions below (for more information, I've found wikipedia's article on the topic to be relatively clear):

1. Bias - causing the types of errors frequently thought of as "underfitting," high bias might generalize well at the cost of capturing important nuances in the data
   2. Variance - causing the types of errors frequently thought of as "overfitting," high variance will capture most nuances in the training data including noise that will reduce its likelihood of generalizing well to future like datasets
   3. Data/Data Quality - While not usually included in the classic understanding of the tradeoff, data type and quality is integral to understanding why a certain model is tipping towards either the bias or variance sides of the tradeoff. This also includes any kinds of time effects or other quirks that in essence mean that a training/test split will not properly represent the model's future performance, especially as regards linear models.