In basic statistics, variance is a measure the variability of the data about its mean. In machine learning, variance is a measure of learning the training data too well/capturing the noise in the data/oversensitivity to the small local fluctuations of the data.

Since machine learning is mostly refurbished statistics, how does these two concepts relate to one another

  • 2
    $\begingroup$ Variance is a mathematical concept; it is the expected value of the squared error between a random variable and its mean (its first moment). Through the bias-variance decomposition in which we expand the expected squared error between the target, y, and the estimator f* of y (our model, which is a random variable!) you can derive an expression that by definition is defined in terms of the bias of f*, variance of f* and irreducible error. Think of variance and mean as expressions in terms of random variables. Data or "a model" are like any other random quantity with some probability dist. $\endgroup$ – aranglol Jul 8 at 19:10
  • 2
    $\begingroup$ Also note: the concept of overfitting/high variance and the tradeoff between underfitting/high bias is a direct reference to the decomposition of the mean squared error between y and our estimator/model f* into bias, variance and irreducible error. My overall point is that there is no difference between the ML definition or the statistical one because these are just specific cases of the exact same mathematical definition. $\endgroup$ – aranglol Jul 8 at 19:14

Variance in machine learning appears in many places, but in all cases it is of course simply an application of the mathematical definition. For instance, people often worry about the variance of stochastic gradients $\mathbb{V}[\nabla \mathcal{L}]$ (especially when using score function/likelihood ratio gradient estimators), which refers to how much the gradient (treated as a random variable) varies, either over time OR as a function of the minibatch. If you took the same model, with many different minibatches, ideally you'd hope that the variance of the resulting gradient estimate would be low (sometimes true with large enough batch sizes).

For the specific case you are talking about

variance is a measure of learning the training data too well/capturing the noise in the data/oversensitivity to the small local fluctuations of the data

This is the variance in model performance with respect to changes in the training data.

Here's a simple way to look at this. Let $\mathcal{D}=(\mathcal{X},\mathcal{Y})$ be the space of possible datasets and $P_D$ some distribution on that space (so $D\sim P_D$ where $D=(X,Y)\in\mathcal{D} $ is a random training dataset). A specific training example is $(x,y)\in D$ with $x\in {X}$ and $y\in {Y}$. We write $x,y\sim D$ to mean $D\sim P_D$ and $x,y\sim\mathcal{U}(D)$ where $\mathcal{U}$ denotes the discrete uniform distribution. Now, let $f_\theta : X \rightarrow Y $ be a learned function parametrized by $\theta\in \Theta$. Suppose $\mathcal{L}: \Theta\times \mathcal{D}\rightarrow \mathbb{R}$ is some loss function defined on the space of learned functions. Note that it is dependent on the dataset! For example, one possible definition of $\mathcal{L}$ could be $$ \widetilde{\mathcal{L}}(\theta,D) = \frac{1}{|D|}\sum_{(x,y)\in D} L_s(f_\theta(x), y) $$ where $L_s : Y\times Y\rightarrow\mathbb{R}$ is some loss function for specific examples. This is one possible form, but a specific one is not necessary.

What is the variance then? We can consider $\mathcal{L}$ as a (random) function of $D$ by first assuming that we can compute $\theta$ given $D$, presumably by some stochastic training algorithm $A$, so that $\theta \sim A(D)$. Presumably, this looks something like stochastic gradient descent. Then our expected loss is $$ \mu_\mathcal{L} = \mathbb{E}_{D\sim P_D}\left[ \mathbb{E}_{\theta \sim A(D)}\left[ \mathcal{L}(\theta, D) \right] \right] $$ meaning the variance in the loss as a function of the training set is given by $$ \mathbb{V}[\mathcal{L}] = \mathbb{E}_{D\sim P_D}\left[ \mathbb{E}_{\theta \sim A(D)}\left[ \left(\mathcal{L}(\theta, D) - \mu_\mathcal{L}\right)^2 \right] \right] $$ Essentially, for a fixed (random) training procedure, we compute the variance in the model performance as the training set varies.

This is pretty oversimplified and honestly non-standard, so if you are interested in the formalities of this kind of thing, look into Probably Approximately Correct (PAC) Learning Theory and Empirical Risk Minimization, where this is formulated properly (meaning, rigorously). :)

The comments above discuss (very appropriately) an important special case of this, namely the Bias-Variance Decomposition for squared error. A common simplification for this case is to assume that $P_D$ samples from a fixed set of paired values $(X\in\mathcal{X}, Y\in\mathcal{Y}=\mathbb{R})$, related by $$y = h(x) + \xi, \;\;\; x\in X, \, y\in \mathbb{R},\, \xi\sim\eta(\mu_\xi,\sigma_\xi^2)$$ where $\mathbb{E}[\xi]=\mu_\xi$, $\mathbb{V}[\xi]=\sigma_\xi^2$, and $\xi$ independent from $x,y$. We still therefore denote $D\sim P_D$ to mean sampling a random training set (meaning, draw $n=|D|$ random points $x_i\sim X$ and then perform $y_i=h(x_i) + \xi_i$). Note that $\theta$ (and $f_\theta$) a random variable, dependent on the random variable $D$, computed by some (potentially stochastic) procedure like $$\theta^*(D) \leftarrow \arg\min_\theta \mathcal{L}(\theta,D) = \arg\min_\theta \frac{1}{|D|}\sum_{(x,y)\in D} (y - f_\theta(x))^p + R(\theta) , $$ where $p\geq 1$ and $R$ is a regularizer, for instance.

Our expected squared error on a single fixed pair $(x,y)$ is then \begin{align} \mathbb{E}_{D\sim P_D}\left[ (y - f_\theta(x))^2 \right] &= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 + 2h(x)\xi + \xi^2 \right]\\ &\;\;\;\;\;\;\;- 2 \left( \mathbb{E}_{D\sim P_D}\left[ h(x)f_\theta(x) \right] + \mathbb{E}_{D\sim P_D}\left[ \xi f_\theta(x) \right] \right) \\ &\;\;\;\;\;\;\;+ \mathbb{E}_{D\sim P_D}\left[ f_\theta(x)^2 \right] \\ &= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 - 2h(x) f_\theta(x) + f_\theta(x)^2 \right] \\ &\;\;\;\;\;\;\;+ \underbrace{\sigma_\xi^2}_{\mathcal{V}_\xi} + \underbrace{\mu^2_\xi + 2\mu_\xi\mathbb{E}_{D\sim P_D}\left[ f_\theta(x) - h(x) \right]}_{\mathcal{B}_\xi} \\ &= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 - 2h(x) f_\theta(x) + f_\theta(x)^2 \right] + \mathcal{V}_\xi + \mathcal{B}_\xi \\ &= \underbrace{\mathbb{E}_{D\sim P_D}[h(x)]^2}_{h(x)^2} + \underbrace{\mathbb{V}_{D\sim P_D}[h(x)]}_0 - 2h(x)\mathbb{E}_{D\sim P_D}[f_\theta(x)] \\ &\;\;\;\;\;\;\;+ \mathbb{E}_{D\sim P_D}[f_\theta(x)]^2 + \underbrace{\mathbb{V}_{D\sim P_D}[f_\theta(x)]}_{\,\mathcal{V}_f} + \mathcal{V}_\xi + \mathcal{B}_\xi \\ &= \underbrace{\left( h(x) - \mathbb{E}_{D\sim P_D}[{f_\theta(x)}] \right)^2}_\mathcal{B_f}+ \mathcal{V}_f + \mathcal{V}_\xi + \mathcal{B}_\xi \\ &= \mathcal{V}_f + \mathcal{B}_f + \mathcal{V}_\xi + \mathcal{B}_\xi \end{align} where we used $\mathbb{V}[r] = \mathbb{E}[r^2] - \mathbb{E}[r]^2$ for any random variable $r$, $\mathbb{E}[uv]=\mathbb{E}[u]\mathbb{E}[v]$ for independent $u$ and $v$, and the fact that $h(x)$ is deterministic (meaning $\mathbb{E}[h(x)]=h(x)$ and $\mathbb{V}[h(x)]=0$), since $x$ is fixed and we are varying the epxectation over $D$ (which only affects $\theta$). Note that the "classical" bias-variance decomposition (as on Wikipedia) is a special case when $\mu_\xi = 0$.

In any case, we get that the expected error of our regressor on a fixed input (where we consider the expectation over possible datasets) is a sum of four terms (two variance and two bias):

  1. The bias $\mathcal{B}_\xi$ and variance $\mathcal{V}_\xi$ caused by the inherent noise $\xi$ in the relation between $x$ and $y$.

  2. The bias $\mathcal{B}_f$ which is how far the average prediction of $f_\theta$ is from the true value.

  3. The variance $\mathcal{V}_f$ which is how "noisy" the output of $f_\theta$ is. Note this does not depend on the true value, nor on the noise in the true relation between $x$ and $y$. It is the inherent noise in the learning processes, measuring how much a change in the training set alters the resulting model.

So anyway, I got a little sidetracked, but all of these derivations were merely just to show that the variance in ML is in fact literally a simple mathematical variance. :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.