In this blog, which explains the meaning of bias and variance in machine learning, there's a line under the heading "Bias-Variance Trade-Off" which says:

Parametric or linear machine learning algorithms often have a high bias but a low variance.

I know that there's an "often" in the first sentence, but how can it be true?
If a linear ML algorithm has high bias how can we expect it to have low variance?


The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more likely to produce bias-type errors rather than variance-type. Another way of thinking about it is that the way that most linear models will give you broadly correct predictions overall, instead of specifically accurate or wildly inaccurate predictions. Obviously that's a super simplistic reading, but I've found an intuitive way to approach. Essentially - the way that linear models are constructed will lead them to predict more generally, instead of reflecting every type of twist and turn in the data. This is partially a function of the models themselves, and partially due to linear models being the first line of attack with smaller datasets.

If you think about the formula, one way to interpret is that the intercept generalizes the model's predictions. That being said, with enough variables and certain types of data, it's very possible to overtrain a linear model (which will tip it towards variance-type errors).

I like to think of the bias/variance tradeoff question as being one with three factors, instead of just the two in the title. I'll use the short definitions below (for more information, I've found wikipedia's article on the topic to be relatively clear):

  1. Bias - causing the types of errors frequently thought of as "underfitting," high bias might generalize well at the cost of not-capturing (losing out on) important nuances in the data
    1. Variance - causing the types of errors frequently thought of as "overfitting," high variance will capture most nuances in the training data including noise that will reduce its likelihood of generalizing well to future like datasets
    2. Data/Data Quality - While not usually included in the classic understanding of the tradeoff, data type and quality is integral to understanding why a certain model is tipping towards either the bias or variance sides of the tradeoff. This also includes any kinds of time effects or other quirks that in essence mean that a training/test split will not properly represent the model's future performance, especially as regards linear models.
| improve this answer | |

The bias is error of the model in the training stage. If your goal is, let's say, a 96% of accuracy, and you get 90%, you have a bias of 6%. The variance is the error difference between training and validation, so, taking the same example, if you get a validation accuracy of 80%, you have 10% variance.

A linear ML algo won't work well on non-linear cases, which are most of the real cases. Then, often, a linear model will perform bad on the training data (high bias) and equally bad on the validation set (low variance). In this scenario, the model is underfitting the data, it is too simple.

As an example:

  • Desired accuracy: 95%
  • Training accuracy: 65%
  • Validation accuracy: 62%
| improve this answer | |
  • 2
    $\begingroup$ I am afraid your definitions of bias and variance are wrong. These quantities can only be calculated if the true underlying data generation process is known. Which is never the case in practice. $\endgroup$ – Michael M Dec 15 '18 at 7:11
  • 1
    $\begingroup$ Well, you are right in some sense. From a statistical point of view, there is a mathematical formula for both bias and variance, related to the expectation of the difference between the estimates and the true values... But in the ML field, is quite often to talk about the bias and the variance in that way $\endgroup$ – ignatius Dec 17 '18 at 8:37

Sorry, but I think the statement

Parametric or linear machine learning algorithms often have a high bias but a low variance.

is not true.

Ordinary least squares (OLS) is - under certain assumptions, i.e the Gauss-Markov-Theorem - unbiased. It is known as the „best linear unbiased estimator“ (BLUE). This property is preserved if we add non-linear parameters (e.g. polynomials) to the regression equation. So non-linearity in the data is not the issue here.

Using Lasso or other things like boosting, reduces variance, but introduces bias. So I think the statement must be just the other way round.

Parametric or linear machine learning algorithms often have a high variance but a low bias.

| improve this answer | |
  • $\begingroup$ That's the wrong meaning of "bias" here. From wikipedia's bias-variance tradeoff article, "The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting)." $\endgroup$ – Ben Reiniger Nov 18 '19 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.