6
$\begingroup$

I am familiar with terms high bias and high variance and their effect on the model.

Basically your model has high variance when it is too complex and sensitive too even outliers.

But recently I was asked the meaning of term Variance in machine learning model in one of the interview?

I would like to know what exactly Variance means in ML Model and how does it get introduce in your model? I would really appreciate if someone could explain this with an example.

$\endgroup$
  • $\begingroup$ Variance in statistics is the same as variance in ML. That’s because ML is a rebranding of statistics. $\endgroup$ – Jon Aug 23 '18 at 16:32
6
$\begingroup$

It is pretty much what you said. Formally you can say:

Variance, in the context of Machine Learning, is a type of error that occurs due to a model's sensitivity to small fluctuations in the training set.

High variance would cause an algorithm to model the noise in the training set. This is most commonly referred to as overfitting.

When discussing variance in Machine Learning, we also refer to bias.

Bias, in the context of Machine Learning, is a type of error that occurs due to erroneous assumptions in the learning algorithm.

High bias would cause an algorithm to miss relevant relations between the input features and the target outputs. This is sometimes referred to as underfitting.

Relationship between bias and variance:

In most cases, attempting to minimize one of these two errors, would lead to increasing the other. Thus the two are usually seen as a trade-off.

Cause of high bias/variance in ML:

The most common factor that determines the bias/variance of a model is its capacity (think of this as how complex the model is).

  • Low capacity models (e.g. linear regression), might miss relevant relations between the features and targets, causing them to have high bias. This is evident in the left figure above.

  • On the other hand, high capacity models (e.g. high-degree polynomial regression, neural networks with many parameters) might model some of the noise, along with any relevant relations in the training set, causing them to have high variance, as seen in the right figure above.

How to reduce the variance in a model?

The easiest and most common way of reducing the variance in a ML model is by applying techniques that limit its effective capacity, i.e. regularization.

The most common forms of regularization are parameter norm penalties, which limit the parameter updates during the training phase; early stopping, which cuts the training short; pruning for tree-based algorithms; dropout for neural networks, etc.

Can a model have both low bias and low variance?

Yes. Likewise a model can have both high bias and high variance, as is illustrated in the figure below.

How can we achieve both low bias and low variance?

In practice the most methodology is:

  1. Select an algorithm with a high enough capacity to sufficiently model the problem. In this stage we want to minimize the bias, so we aren't concerned about the variance yet.
  2. Regularize the model above, to minimize its variance.
$\endgroup$
2
$\begingroup$

Variance is the change in prediction accuracy of ML model between training data and test data.

Simply what it means is that if a ML model is predicting with an accuracy of "x" on training data and its prediction accuracy on test data is "y" then

Variance = x - y

$\endgroup$
1
$\begingroup$

Variance is the variability of model prediction for a given data point or a value which tells us spread of our data. Model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before. As a result, such models perform very well on training data but has high error rates on test data.

Error due to variance

Error due to variance is the amount by which the prediction, over one training set, differs from the expected value over all the training sets. In machine learning, different training data sets will result in a different estimation. But ideally it should not vary too much between training sets. However, if a method has high variance then small changes in the training data can result in large changes in results.

https://www.coursera.org/lecture/machine-learning/diagnosing-bias-vs-variance-yCAup

https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229

$\endgroup$

protected by Community Aug 22 at 11:44

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.