For conditionally independent features $f_i$, Naive Bayes Classification gives me the classifier

$Classifier(f) := \arg \max_{k} P(C=k) · ∏^n_{i=1} P(f_i|C=k)$

for classes $k$. I understand that for Gaussian Naive Bayes, I can assume normally distributed features, yielding

$Classifier(f) := \arg \max_k P(C=k) · ∏^n_{i=1} \frac{1}{\sqrt{(2πσ_{k,i})}} e^{( -\frac{(f_i - μ_{k,i})^2}{2σ_{k,i})}}$

where $μ_{k,i}$ is the mean of class $k$ and feature $f_i$ (and similiar for variance $σ_{k,i}$).

But where is the "learning step" in this whole procedure?


1 Answer 1


I'm assuming you're asking about the intuition behind Naive Bayes (NB). For the sake of clarity I'm considering only categorical features. Gaussian NB is simply an application of NB to numerical features (assumed to be normally distributed).

During training every $p(f_i|C_k)$ is calculated by counting how often the feature value $f_i$ is associated with class $C_k$ among all the other possible features values associated with $C_k$:

  • This is done by counting how often $f_i$ and $C_k$ appear together across all the instances. That's how NB generalizes: the fact that a feature appears with a particular class is just an example, but the fact it appears proportionally more often with class A than class B forms a pattern.
  • The probability $p(f_i|C_k)$ represents how important $f_i$ is within class $C_k$.

When predicting the class for a new instance:

  • NB "weighs all the pros and cons" for this instance to be predicted as $C_k$ by combining all the $p(f_i|C_k)$ corresponding to this instance (in the sense that some of the probabilities $p(f_i|C_k)$ are low and some are high, so their product reflects the combination of "pros" and "cons" indications).
  • But even if $p(f_i|A) > p(f_i|B)$, it doesn't imply that $f_i$ is a strong indication of class $A$, because maybe this is due to class $A$ being less frequent than class $B$. This is taken into account in the prior $p(C_k)$, which gives more importance to a frequent class than a rare one (this is the basis of the Bayes theorem).

These last two points show how NB uses the "knowledge" of the trained model to make a prediction about any unknown instance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.