2
$\begingroup$

I've been trying to find an answer to this question for a long time.

What are the relationships/differences between Bias, Variance and Residuals?

I think I do understand Bias, Variance and Residuals as separate concepts. Correct me if I'm wrong -

Bias is the difference between the average expected results from different runs of the model and the true values from data.

Variance is the variability in the expected results (predictions) of a given data point between different runs of the model.

Residual is the difference between the expected results from a model and the true values from data.
y - y^
Residual seems somewhat similar to Bias. But is that my misunderstanding?

Please explain to me in plain words and also simple equations the relationships or differences between these three.

Thanks in advance.

$\endgroup$
3
$\begingroup$

I think your confusion arises from mixing two different kinds of terms together here. Bias and variance are general concepts which can be measured and quantified in a number of different ways. A residual is a specific measurement of the differences between a predicted value and a true value.

Bias, loosely speaking, is how far away the average prediction is from the actual average. One way to measure it is in the difference of the means. You could also use difference of medians, difference in range, or several other calculations. To a get a complete picture of the bias of a model, you will want to look at several different measures.

Variance, when used informally in data science, is a property of single sets (whether predictions or true values). The variance of a model, loosely speaking, is how far from the average prediction a randomly selected prediction will be. It's very often assessed using cross-validation. You construct multiple models using slightly different training sets but the same algorithm and tuning parameters. You then calculate an evaluation metric for each model, and calculate the standard deviation of this evaluation over all your models. This gives you a sense of the "stability" of a given algorithm/parameter set when exposed to different training and testing sets.

(N.B. This can be confusing because there is a specific definition of "variance" used in statistics, $v = \sigma^2$. In data science, it's usually used more informally.)

Residuals are a specific quantity associated with a single prediction/true value set pair. You've got the right definition there. This makes it not a general concept, but instead a measurement that you can use to assess either bias or variance. They're also frequently used in fitting regression models and otherwise performing gradient descent-style optimization. The mean or median of a residual set can be a way to assess bias, while the standard deviation of a residual set can be used to assess a variance.

$\endgroup$
  • $\begingroup$ Thank you so much for the clarification. To confirm, Bias is the difference between the average of all predictions and the average of true values of a single training model. (No multiple samples/training sets here). Variance is the difference between each prediction and the average of all predictions over several training models. Residual is the difference between the prediction and true value of a specific data point. The average of residuals against the average of true values can be used to determine Bias? $\endgroup$ – Minu Mar 23 '16 at 15:42
  • $\begingroup$ You're on the right track. Average residuals are a direct measure of Bias, as a perfect (completely unbiased) model would have an average residual of 0. Bias is not so much about the difference between the average of all predictions and the average of all true values. It's a fuzzier concept, more about how far off your predictions as a whole are from your true values. $\endgroup$ – Data Science Dojo Mar 23 '16 at 22:31
4
$\begingroup$

A graphical explanation how I understood bias and variance from Andrew Ngs course:

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.