In theory we would always like to predict qualitative responses using the Bayes classifier. But for real data, we do not know the conditional distribution of Y given X, and so computing the Bayes classifier is impossible. Therefore, the Bayes classifier serves as an unattainable gold standard against which to compare other methods.

  • In a real life situation why we are not able to calculate the Bayes Classifier ?
  • Why can't we calculate the conditional distribution of Y given X ?
  • Can some one explanation with an example?

Any help is appreciated.

  • $\begingroup$ It seems as if you have cited the first part from "An Introduction to Statistical Learning: with Applications". Please mark it as a citation (prefix it with >) and add the reference. $\endgroup$ Jan 26, 2016 at 18:33

1 Answer 1


You can't calculate the conditional distribution because that distribution is defined over the entire population and assuming a probabilistic model which you do not know.

In real life you have only a sample of the population. More often than not, in real life you do not have the structure of the model also. You do not know if it is a linear model, a polynomial or another type.

The probabilistic approach is to assume a model, consider some assumptions for that model and fit the model's parameters using data. That is not the Bayes classifier, it's only an approximation.

The trick is 'all models are wrong, some models are useful'. There are many approaches to learn this approximation. You can estimate the joint probability (generative classifiers), you can estimate directly the conditional probability (discriminative classifiers) or you can use a different approach (like SVM, for example) and twist it somehow to approximate the conditional probability. But all of them are approximations. The real one, THE Bayes classifier will remain unknown.

[Later edit]

Looking at the Wikipedia page for Bayes classifier it states that: "The Bayes classifier is a useful benchmark in statistical classification.". My opinion is that this statement is confusing, if not dead wrong. Probably this is related with the wrong usage of expression "Bayes classifier" to designate the Naive Bayes classifier. If one search on google for "Bayes classifier", he will see a lot of materials related with Naive Bayes. Personally I do not agree with the statement from wikipedia. Bayes classifier is that state of nature that we do not know but we aim to approximate as well as possible.

  • $\begingroup$ Wikipedia being wrong? Quelle surprise!! $\endgroup$ Mar 17, 2023 at 5:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.