I have a hard time intuitively understand the Bayer error in the context of supervised learning. We have an input X and an output Y. We want to find the function f(X) = Y.

I feel like I don't understand why we model X as a random variable in the first place. I think we construct the function f not based on the distribution of X but rather on actual values of X. Why should we care if X is stochastic or not?

Example I: Let's say X = [1,2,3], Y = [1,2,3] and f(x) = x. When X=1 => Y=1, X=2 => Y=2, X=3 => Y=3, I will never make an error.

Example II: Why should it be different if X = [picture of dog, picture of cat, picture of house] and Y = [dog, cat, house]. I can still find a function which does the mapping. It is obviously more complex but doable.

Where did the Bayes error get lost in my examples?

I am looking for an intuitive explanation of the Bayes error preferably in the context of image classification.


1 Answer 1


As an MNIST dataset, see the mispredicted images on this figure below, you notice for some samples, we as human don't know for sure what is the true label.

enter image description here

Maybe there are other factors that are not visible to us to predict the correct output.

Assume I want to predict customer will leave our company, maybe there are tons of useful features to start with, yet he may simply have some other problems that we are not aware of, and it may affect his decision in leaving the company.

  • $\begingroup$ I think what you are saying is that our task is not to find a function which maps from X to Y but rather X +e to Y where e is some random event/noise. The introduction of e causes our data input to become a random variable. In your example e is the noise in the handwritten digits. Do I understand that correctly? $\endgroup$
    – siva
    Commented May 6, 2018 at 11:58
  • $\begingroup$ We want X to be a random variable, not for the sake of the stochasticity introduced with the noise (imperfect data), but if we compute P(X), we would disentangle the factors that explain the variations observed in the data, maybe y is predictable by some factors, (not all of them), at the end of the day, we want to maximize the likelihood of the data, so we need data points that drawn from the true distribution to have high probability. $\endgroup$ Commented May 6, 2018 at 12:07
  • $\begingroup$ hm not really satisfied yet. But I think the variations in the data are only there because of noise. And where exactly is the Bayes error? $\endgroup$
    – siva
    Commented May 6, 2018 at 12:56
  • $\begingroup$ No, variations exist because of the so-called latent variables. variations are not limited to the digit class, but the angle that the digit is drawn, the stroke width, and also abstract stylistic properties. $\endgroup$ Commented May 6, 2018 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.