Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In this blog, which explains the meaning of bias and variance in machine learning, there's a line under the heading "Bias-Variance Trade-Off" which says:
Parametric or linear machine learning algorithms often have a high bias but a low variance.
I know that there's an "often" in the first sentence, but how can it be true?
If a linear ML algorithm has high bias how can we expect it to have low variance?
The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more likely to produce bias-type errors rather than variance-type. Another way of thinking about it is that the way that most linear models will give you broadly correct predictions overall, instead of specifically accurate or wildly inaccurate predictions. Obviously that's a super simplistic reading, but I've found an intuitive way to approach. Essentially - the way that linear models are constructed will lead them to predict more generally, instead of reflecting every type of twist and turn in the data. This is partially a function of the models themselves, and partially due to linear models being the first line of attack with smaller datasets.
If you think about the formula, one way to interpret is that the intercept generalizes the model's predictions. That being said, with enough variables and certain types of data, it's very possible to overtrain a linear model (which will tip it towards variance-type errors).
I like to think of the bias/variance tradeoff question as being one with three factors, instead of just the two in the title. I'll use the short definitions below (for more information, I've found wikipedia's article on the topic to be relatively clear):
The bias is error of the model in the training stage. If your goal is, let's say, a 96% of accuracy, and you get 90%, you have a bias of 6%. The variance is the error difference between training and validation, so, taking the same example, if you get a validation accuracy of 80%, you have 10% variance.
A linear ML algo won't work well on non-linear cases, which are most of the real cases. Then, often, a linear model will perform bad on the training data (high bias) and equally bad on the validation set (low variance). In this scenario, the model is underfitting the data, it is too simple.
As an example:
Sorry, but I think the statement
Parametric or linear machine learning algorithms often have a high bias but a low variance.
is not true.
Ordinary least squares (OLS) is - under certain assumptions, i.e the Gauss-Markov-Theorem - unbiased. It is known as the „best linear unbiased estimator“ (BLUE). This property is preserved if we add non-linear parameters (e.g. polynomials) to the regression equation. So non-linearity in the data is not the issue here.
Using Lasso or other things like boosting, reduces variance, but introduces bias. So I think the statement must be just the other way round.
Parametric or linear machine learning algorithms often have a high variance but a low bias.
