I'm trying to understand how optimal bayes classifier works and I was wondering if, given that the function we try to maximize when making a new prediction does not depend on the instance we are trying to classify, is it correct to state that the optimal bayes classifier would always predict the most probable class no matter what the input is?


I've been studying the subject on "Machine Learning, Tom Mitchell, McGraw Hill, 1997" where is stated that the prediction for a new instance is the class $v_j$ for which the function $\underset{v_j \in V}{\operatorname{arg max}}\sum_{h_i \in H}{P}(v_j|h_i){P}(h_i|D) $
Where $V$ is the set of all possible classes, $H$ is the space of the hypothesis and $D$ is the train dataset.

  • $\begingroup$ Hi Niccolò, welcome to DS stack exchange. Please include a reference for your statement, this would help clear the particular misunderstanding that it seems you're having. Anyway, the statement is wrong. The output does depend on the input. $\endgroup$
    – KishKash
    May 16, 2023 at 10:00
  • $\begingroup$ @KishKash thank you for the suggestion, I don't know if I'm misinterpreting the way the classification happens but to me it seems like all the values involved in the maximization problem depends on dataset, hypotesis space and class space $\endgroup$ May 16, 2023 at 10:24

1 Answer 1


What you seem to miss is that each hypothesis is itself a function that maps input to outputs. According to this post, a hypothesis is "an instance or specific candidate model that maps inputs to outputs and can be evaluated and used to make predictions."

Therefore, the $\arg \max$ in the optimization problem above is over different functions mapping inputs to outputs, taking into account the entire training set.

Once the optimal hypothesis within the hypothesis space is found, it has the ability to classify individual instances to their correct classes.

  • $\begingroup$ The books says that the sum in the argmax I wrote gives the probability ${P}(v_j|D)$ and that the optimal classification is the $v_j$ value which is a class value itself. So I don't agree with the fact that the problem is trying to obtain the best classification function. The book also differentiate the bayes optimal classifier from the MAP problem which is the one that gives the optimal hypotheses as output. $\endgroup$ May 16, 2023 at 20:09
  • $\begingroup$ Pretty much the same expression that you quoted also appears in wikipedia. It seems like some notation oversight. The text supports the fact that the sum in the expression is intended to result in an average (or precisely, ensemble) hypothesis rather than a single classification. $\endgroup$
    – KishKash
    May 18, 2023 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.