Meaning of variance in machine learning models

Question

I know that high variance cause overfitting, and high variance is that the model is sensitive to outliers.

But can I say Variance is that when the predicted points are too prolonged lead to high variance (overfitting) and vice versa.

Answer 1

Variance actually measures the variability of the model prediction (say, for simplification, for a particular sample instance) if we would retrain the model multiple times (on different subsets of the data).

To gain an intuitive feeling of variance, suppose you have 100 training examples in your dataset (150 examples in total where 50 examples are reserved for a 50:50 split between validation and test sets).

Pick a specific example from this set of 150 examples, say ${x}_{i}$. You are required to classify this in any one of the classes: ${k}_{1}, {k}_{2}, ..., {k}_{n}$.

Now randomly pick 100 samples from your original set of 150 samples (excluding your ${x}_{i}$). Now suppose you choose a hypothesis that has many higher degree terms, for instance: ${a}_{1}{x}^8 + {a}_{2}{x}^{7} + ... {a}_{0} = 0$ and do not regularise it, it will fit the model nicely and you get a low error on your training set.

You now run this model on ${x}_{i}$ and it predicts wrongly. You repeat the above process of having 100 samples five times and each time you get a different result, say ${k}_{a}, {k}_{b}, {k}_{c}, {k}_{d}$, and ${k}_{e}$.

This variability in your prediction (coupled with the low error on your training set) implies your model is overfitting the data it is given. That's why it is not able to follow the general pattern in your data and is being affected by outliers more than it should.

I guess that answers your question about declaring a model as highly variant when

the predicted points are too prolonged

Answer 2

What is variance?

Variance is the variability of model prediction for a given data point or a value which tells us spread of our data. Model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before. As a result, such models perform very well on training data but has high error rates on test data.

Error due to variance

Error due to variance is the amount by which the prediction, over one training set, differs from the expected value over all the training sets. In machine learning, different training data sets will result in a different estimation. But ideally it should not vary too much between training sets. However, if a method has high variance then small changes in the training data can result in large changes in results.

https://www.coursera.org/lecture/machine-learning/diagnosing-bias-vs-variance-yCAup

https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229

Answer 3

Variance is the mean of squared deviations from the mean. Analyzing variance tests, the hypothesis that the means of two or more populations are equal.

Bias versus variance trade-off

Every model has both bias and variance error components in addition to white noise. Bias and variance are inversely related to each other; while trying to reduce one component, the other component of the model will increase. The ideal model will have both low bias and low variance

An example of a high bias model is logistic or linear regression, in which the fit of the model is merely a straight line and may have a high error component due to the fact that a linear model could not approximate underlying data well.

An example of a high variance model is a decision tree, in which the model may create too much wiggly curve as a fit, in which even a small change in training data will cause a drastic change in the fit of the curve.

The above information is from the book Statistics for Machine Learning which i used to refer

Answer 4

Its a bias variance trade-off problem:

• When increase model complexity, variance is increased and bias is reduced
• When regularize the model, bias is increased and variance is reduced.

Mathematically

High Bias:

• No matter how much data we feed the model, the model cannot represent the underlying relationship and has high systematic errors

• Poor fit

• Poor generalization

High Variance:

• Require data to improve

• Can simplify the model with fewer or less complex features

To approach the issue we can graph our model's performance based on varying criteria during the analysis process to visualizing behavior that may not have been apparent from the results alone.

Learning Curves - Scikit Learn

Here is learning curve for a decision tree example.

See that the maximum depth 3 is the one that represents the best learning curve.

Note that in this example the maximum depth 3 is the one that represents the best learning curve.

Even with the addition of more training points the score of the training curve reduces your score a little, but gradually tends to decrease this reduction demonstrating stability even with the addition of a high number of training points.

Already in the case of the test curve, with the addition of more training points the score tends to increase significantly at the beginning, denoting the best fit of the model, however even if it converges in high numbers of training points the score of the curve also achieves a stability in the improvement in a certain number of training points.

Having more training points benefits the model until a certain point in time that the scores of your training and test curves stabilize, not requiring further training points increment.

A excelent reference is in the link bellow: Andrew Ng Guide