3
$\begingroup$

My understanding of high variance is that the targets are spread widely around. The output values are "all over the place".

In a binary classification model, there can only be 2 outcomes. I am at a loss when visualizing what high variance mean in a binary classification model.

$\endgroup$
3
$\begingroup$

You have correctly intuited that variance isn't as useful a concept in this case. Statisticians typically look at the binomial deviance instead (see here for a thorough technical development).

If you do want to think about variance, we can recognize that many binary classifiers output an estimate of the label $\hat{Y}$. It helps here to think of a randomized classifier which estimates $P(Y=1|X=x) = E[Y|X=x]$ (call this estimate $\hat{p}(x)$) and generates $\hat{Y} = 1$ with probability $\hat{p}(x)$, $\hat{Y} = 0$ otherwise. We have $$var(\hat{Y}|X=x) = E[(\hat{Y} - E[\hat{Y}|X=x])^2|X=x].$$ This will be smaller when the classifier is more certain about the classification once it sees the covariates. I.e., if $\hat{p}$ is always either 1 or 0, then the variance is 0 (but of course the bias will be huge). If there are equal numbers of each class in the training data and the classifier is able to learn nothing from the covariates, then $\hat{p} = \frac{1}{2}$ for any $x$ and so the variance is $\frac{1}{4}$--a coin flip, as we'd expect.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. If variance is not so relevant in binary classification problems, does it follow that priority should be given to achieve low bias when making trade-off between bias and variance in binary classification models? $\endgroup$ – user781486 Aug 29 '19 at 0:57
  • $\begingroup$ No--I'm not saying variance isn't important, I'm just saying that the traditional definition of variance doesn't apply well with binary outcomes and non-randomized classifiers, so we use the deviance instead. You still need to do the balancing act of bias vs. "variance," you just think of "variance" in terms of the deviance. $\endgroup$ – Sheridan Grant Aug 29 '19 at 4:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.