1
$\begingroup$

I am studying Naive Bayes classification method from Data Mining Concept and Technique by Han, Kamber, Pei.

There is an example of how to find out the class probability using Naive Bayes classifier.

The dataset that they have consideredDataset

The derivation is enter image description here

enter image description here

I like to implement it using SkLearn. So I read documentation of SkLearn for categorical Naive Bayes. enter image description here

The main formula is

$$P(x_i = t \mid y = c \: ;\, \alpha) = \frac{ N_{tic} + \alpha}{N_{c} + \alpha n_i},$$

I like to compare it with the formula of the book.

$$P(x_i = t \mid y = c \: ;\, \alpha)$$ can be written as P(age = youth| buy_computer =yes) because as per the SkLearn definition t is a category for feature i. So, youth is a category for feature age.

The right hand side of the formula is $$\frac{ N_{tic} + \alpha}{N_{c} + \alpha n_i},$$

N_tic is the number of times category t appears in the samples x_i. Here from the table we can see that youth appears 2 times when buy_computers = yes. N_c is the number of samples with class c. So there are 9 yes for buy_computer columns.

$$P(x_i = t \mid y = c \: ;\, \alpha) = \frac{ N_{tic} + \alpha}{N_{c} + \alpha n_i},$$ = (2+1)/(9+1*3) = 1/4 = 0.25 (by default value of alpha =1.0) $$n_i$$ number of available category in feature i. Age has 3 categories.

If we consider the mathematics of the book P(Age = 'Youth'| buy_computer = yes) = 0.222 which is not equal to 0.25.

Secondly, if I follow the equation of Scikit Learn how could I implement the last portion of mathematical derivation from the book?

this portion

$\endgroup$
2
  • 2
    $\begingroup$ The difference between the value calculated by the book and scikit-learn is caused by the fact that you are using a value for the smoothing parameter of 1 (applying smoothing), whereas the book does not use any smoothing in it's calculation (i.e. an alpha of 0). $\endgroup$
    – Oxbowerce
    Commented Sep 24, 2022 at 17:01
  • $\begingroup$ Ok. Thank you. I would also like to know my 2nd question. How does Sklearn calculate P(X|buy_computer= yes) = P(age=youth|buy_computer=yes)*P(income=medium|buy_computer=yes)*P(student=yes|buy_computer=yes)*P(credit_rating=fair|buy_computer=yes)? $\endgroup$
    – Encipher
    Commented Sep 24, 2022 at 18:08

1 Answer 1

0
$\begingroup$

Secondly, if I follow the equation of Scikit Learn how could I implement the last portion of mathematical derivation from the book?

The calculations your referencing are doing 2 different things

  1. The first half of the calculation is training the model by calculating the likilihood $P(X|C_i)$ of each class $C_i$. The authors do this calculation by using the labeled data provided in Table 8.1 referenced in your question.
  2. The second half of the calculation is using the trained model to predict a label for the unlabeled tuple $X=(youth, medium, yes,fair)$. Note that this tuple doesn't appear in Table 8.1, so it is indeed unlabeled.

Doing this in Scikit Learn would like something like

from sklearn.naive_bayes import CategoricalNB
clf = CategoricalNB()

## the fit() method trains the model.
clf.fit(X_train, y_train)

## the predict() method predicts labels for unlabeled data
clf.predict(X_test)

In this snippet of code I'm assuming you've already split the feature dataset in train and test sets using train_test_split()

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.