SkLearn Categorical Naive Bayes Vs Mathematical theory of Naive Bayes

Question

The Naive Bayes classification based on the following formula

$P(C_i|X) = {P(X|C_i)P(C_i) \over P(X)} ... i)$

$P(X|C_i)$ is the posterior probability of $X$ conditioned on $C_i$, $P(X)$ prior probability of $X$, $C_i$ represents the class.

Now if we have a dataset following:

Age    Income    Buy_computer

Senior   fair        Yes
Junior   fair        Yes
Young    poor        No
Senior   poor        Yes
Junior   fair        No
Young    poor        No

Now if we get a new data (Age = young, Income= fair) we need to find out in which class this data should belong. ... example 1)

We can use eq i) to find out the class

I have also learned Categorical Naive Bayes

As per the documentation,

The probability of category t in feature i given class c is estimated as:

P(Xi=t|y=c ;alpha) = (Ntic + alpha)/(Nc + alpha ni) ...ii)

As per the example 1) we can convert equation ii) as

P(Age = young, Income= fair| Buy_computer=?)

and then apply equation i) to it to find out the class of P(Age=young, Income=fair)

However, I cannot understand how the right hand side of eq ii) is related to equation i)

Equation i) also does not have any alpha parameter, how could the parameter alpha influence the answer?

Thank you.

Answer 1

Now if we get a new data (Age = young, Income= fair) we need to find out in which class this data should belong. ... example 1)

If a sample doesn't have a label you can't include it in the train/test set,not sure if that's what you mean here but I want to clarify that just in case. That being said, you could try to predict it's lable after the model has been trained and tested.

However, I cannot understand how the right hand side of eq ii) is related to equation i)

I think there are two things causing confusion here.

1. The notation you're using for equation $i)$ and $ii)$ is inconsistent. My guess is you're citing these from different sources. I'll use the notation used by sklearn for my answer.
2. You can't apply equation $i)$ to equation $ii)$. Equation $i)$ is a statement of Bayes' Theorem, while $ii)$ is an assumption we make about the liklihood $P(x_i|y)$ for categorically distributed datasets. How $i)$ and $ii)$ are related will be come more clear if we go through a quick derivation of the math behind CategoricalNB

Suppose we have a feature set $X$ with label $y$. We'd like to train a model to calculate proability of output $y$ given this feature set $X$ as this will allow the model to predict unlabeled data. According to Bayes's Theorem, the probability of $y$ given $X$ (denoted $P(y|X)$) is: \begin{eqnarray*} P(y|X) &=& \frac{P(X|y)P(y)}{P(X)} \\ &=& \frac{P(x_1,..x_a|y)P(y)}{P(x_1,..x_a)} \end{eqnarray*} where in the second line we've expanded the $X$ into it's individual features $x_i$. $P(y)$ and $P(x_1,..x_a)$ can be calculated from the training data, but how the hell do we calculate $P(x_1,..x_a|y)$? To do this we assume the features are mutually independent in which case we have: \begin{eqnarray*} P(x_1,..x_a|y) &=& P(x_1|y)P(x_2|y)..P(x_a|y) \\ &=& \Pi_{i=1}^{i=a} P(x_i|y) \end{eqnarray*} The assumption of mutual independence is what puts the Naive in Naive Bayesian Model, i.e. if a Bayesian model is described as Naive it means it is based on an assumption of mutual independence between features. So the problem of calculating $P(y|X)$ has been reduced to calculating $P(x_i|y)$, what seperates all of the Naive Bayesian models is the methodology they use to calculate $P(x_i|y)$. For the methodology behind CategoricalNB we make the further assumption that each feature $x_i$ has a categorical distribution given by: \begin{equation} P(x_i|y, \alpha) = \frac{N_{tic}+\alpha}{N_c+\alpha n_i} \end{equation} where $N_{tic}$ is the number of times category $t$ appears in feature $x_i$ when $y=c$, and $N_c$ it the number of times $y=c$. $\alpha$ is a hyperparameter introduced to reduce overfitting on the train set and $n_i$ is the number of catergories in the feature $x_i$.

So to summarize:

• Equation $i)$ is a statement of Bayes' Theorem, which is the cornerstone of every Bayesian model (that's why they're call Bayesian models after all)
• Equation $ii)$ is an assumption made about the liklihood $P(x_i|y)$. This assumption. along with the assumption of mutually independent features are what underpins the methodology behind sklearns CategoricalNB
• $\alpha$ is a hyperparameter used to reduce overfitting. You can't calculate $\alpha$ with some kind of pen and paper calculation, it can only be calculate via hyperparameter fine tuning

Hope this helps