It is pretty much what you said. Formally you can say:
Variance, in the context of Machine Learning, is a type of error that occurs due to a model's sensitivity to small fluctuations in the training set.
High variance would cause an algorithm to model the noise in the training set. This is most commonly referred to as overfitting.
When discussing variance in Machine Learning, we also refer to bias.
Bias, in the context of Machine Learning, is a type of error that occurs due to erroneous assumptions in the learning algorithm.
High bias would cause an algorithm to miss relevant relations between the input features and the target outputs. This is sometimes referred to as underfitting.
These terms can be decomposed from the expected error of the trained model, given different samples drawn from a training distribution. See here for a brief mathematical explanation of where the terms come from, and how to formally measure variance in the model.
Relationship between bias and variance:
In most cases, attempting to minimize one of these two errors, would lead to increasing the other. Thus the two are usually seen as a trade-off.
Cause of high bias/variance in ML:
The most common factor that determines the bias/variance of a model is its capacity (think of this as how complex the model is).
Low capacity models (e.g. linear regression), might miss relevant relations between the features and targets, causing them to have high bias. This is evident in the left figure above.
On the other hand, high capacity models (e.g. high-degree polynomial regression, neural networks with many parameters) might model some of the noise, along with any relevant relations in the training set, causing them to have high variance, as seen in the right figure above.
How to reduce the variance in a model?
The easiest and most common way of reducing the variance in a ML model is by applying techniques that limit its effective capacity, i.e. regularization.
The most common forms of regularization are parameter norm penalties, which limit the parameter updates during the training phase; early stopping, which cuts the training short; pruning for tree-based algorithms; dropout for neural networks, etc.
Can a model have both low bias and low variance?
Yes. Likewise a model can have both high bias and high variance, as is illustrated in the figure below.
How can we achieve both low bias and low variance?
In practice the most methodology is:
- Select an algorithm with a high enough capacity to sufficiently model the problem. In this stage we want to minimize the bias, so we aren't concerned about the variance yet.
- Regularize the model above, to minimize its variance.