2
$\begingroup$

Im trying to understand why naive is needed in Naive Bayes and everyone says Naive Bayes assumes the input features (predictors) are not correlated hence they are not dependent on each other .

i want to understand what will happen if features are dependent i.e Non Naive (the opposite part of being naive )

if we have a sentence "You won lottery for 1million" and we need to classify it as spam and not spam using naive bayes .

p(y|x)=p(x|y).p(y)

In the likelihood partwe will model the probability as p(x|y)

here x="You won lottery for 1million" and y=spam or not spam

p('You won lottery for 1million'|y=spam)

p('You won lottery for 1million'|y=notspam)

what is the correct way of writting this probability and finding its value without considering independence of event among X ?

should it be written as

**to find probaiblity of spam given feature are depenent**
p('You |won, lottery, for, 1million,spam) * 
p('won| lottery, for, 1million,spam) *
p('lottery| for, 1million,spam)*
p(for| 1million,spam)*
p( 1million|spam)

**to find probaiblity of not spam given feature are depenent**
p('You |won, lottery, for, 1million,notspam) * 
p('won| lottery, for, 1million,notspam) *
p('lottery| for, 1million,notspam)*
p(for| 1million,notspam)*
p( 1million|notspam)

is this correct way of finding the probability of the X considering its events are dependent on each other ? Should the spam/notspam also be included in the dependence part ?

What is the probelm in finding the above 2 probability and why is it so hard that naive has to pitch in and make the features as independent inorder to calculate probability .

$\endgroup$
4
$\begingroup$

I do not think your formulation is correct. What you have described are just conditional distributions for each word in the sentence but not the joint conditional distribution, given a specific class.

In your case, we have by Bayes rule:

$$ Pr(spam | X) \propto Pr(X | spam) \times Pr(spam) = Pr(you, won, lottery, for, 1million | spam) \times Pr(spam).$$

Now, $\hat Pr(spam) = \frac{\text{# of spam occurences in train}}{\text{total observations in train}}$, the MLE that uses none of the features.

In Naive Bayes, we make the assumption that:

$$Pr(you, won, lottery, for, 1million | spam) \times Pr(spam) = Pr(you | spam) * Pr(won | spam) * Pr(lottery | spam) * Pr(for|spam) * Pr(1million | spam) * Pr(spam).$$ That is, we assumed that each of the features are conditionally independent given a class. If we do not wish to make this assumption, we simply leave $Pr(you, won, lottery, for, 1million | spam)$ as is in the above expression and posit some sort of multivariate/joint conditional distribution.

The reason why we make this assumption is due to the difficulty in determining what the conditional joint distribution of the features is given a particular class. Is the joint distribution a multivariate Poisson? A complex mixture of many? Some other multivariate counting distribution? What is the dependence structure, and how should we represent it? What if we have a thousand features? How do we represent the dependence structure now in a reasonable way? Is the dependence structure so complicated that representing it reasonably is more annoying and/or time-consuming than just assuming independence?

These questions are not trivial and are very difficult, if not impossible, to answer. Thus, we make the independence assumption to simplify the problem. We may not know the joint conditional distribution but perhaps we can derive and/or visualize the marginal conditional distributions for each feature. This, of course, comes at the cost of assuming independence between features which is highly unlikely to be true (especially for word frequencies) but nevertheless occasionally effective for some problems (kind of).

| improve this answer | |
$\endgroup$
  • $\begingroup$ @so the non naive is calculalted using multivariate/joint conditional distribution ,can you tell (it need not be precise just an overview ) on how we can get this probability Pr(you,won,lottery,for,1million|spam) for my example ? In naive we just take count (you)/total words (spam) , likewise how do you do for non naive using multivariate/joint conditional distribution ? . it need not be prices but please give a overview of calculating it based on some assumption ,so that i can get a full clear idea of it . $\endgroup$ – Aj_MLstater Feb 19 at 17:47
  • $\begingroup$ Another example Predit the class label for instance (A=1,B=2,C=2) using naive Bayes .Let C1 be class 1 and C2 be class 2. For C1, by the assumption of Naive Bayesian , we have P(A=1,B=2,C=2∣C1)=P(A=1∣C1)⋅P(B=2∣C1)⋅P(C=2∣C1) Take P(A=1∣C1) as an example. There are 4 training records of C1, among which there are 2 records with A=1. Therefore, P(A=1∣C1)=24. Similarly, you can calculate P(B=2∣C1) and P(C=2∣C1).It is similar to calculate P(A=1,B=2,C=2∣C2). how can we ignore independence assumption and calculate the NON Naive part of this using the multivariate/joint conditional distribution $\endgroup$ – Aj_MLstater Feb 19 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.