why naive is needed in Naive Bayes ,what happens if naive is not included in Bayes theorem?

Question

Im trying to understand why naive is needed in Naive Bayes and everyone says Naive Bayes assumes the input features (predictors) are not correlated hence they are not dependent on each other .

i want to understand what will happen if features are dependent i.e Non Naive (the opposite part of being naive )

if we have a sentence "You won lottery for 1million" and we need to classify it as spam and not spam using naive bayes .

p(y|x)=p(x|y).p(y)

In the likelihood partwe will model the probability as p(x|y)

here x="You won lottery for 1million" and y=spam or not spam

p('You won lottery for 1million'|y=spam)

p('You won lottery for 1million'|y=notspam)

what is the correct way of writting this probability and finding its value without considering independence of event among X ?

should it be written as

**to find probaiblity of spam given feature are depenent**
p('You |won, lottery, for, 1million,spam) * 
p('won| lottery, for, 1million,spam) *
p('lottery| for, 1million,spam)*
p(for| 1million,spam)*
p( 1million|spam)

**to find probaiblity of not spam given feature are depenent**
p('You |won, lottery, for, 1million,notspam) * 
p('won| lottery, for, 1million,notspam) *
p('lottery| for, 1million,notspam)*
p(for| 1million,notspam)*
p( 1million|notspam)

is this correct way of finding the probability of the X considering its events are dependent on each other ? Should the spam/notspam also be included in the dependence part ?

What is the probelm in finding the above 2 probability and why is it so hard that naive has to pitch in and make the features as independent inorder to calculate probability .

Answer 1

I do not think your formulation is correct. What you have described are just conditional distributions for each word in the sentence but not the joint conditional distribution, given a specific class.

In your case, we have by Bayes rule:

$$ Pr(spam | X) \propto Pr(X | spam) \times Pr(spam) = Pr(you, won, lottery, for, 1million | spam) \times Pr(spam).$$

Now, $\hat Pr(spam) = \frac{\text{# of spam occurences in train}}{\text{total observations in train}}$, the MLE that uses none of the features.

In Naive Bayes, we make the assumption that:

$$Pr(you, won, lottery, for, 1million | spam) \times Pr(spam) = Pr(you | spam) * Pr(won | spam) * Pr(lottery | spam) * Pr(for|spam) * Pr(1million | spam) * Pr(spam).$$ That is, we assumed that each of the features are conditionally independent given a class. If we do not wish to make this assumption, we simply leave $Pr(you, won, lottery, for, 1million | spam)$ as is in the above expression and posit some sort of multivariate/joint conditional distribution.

The reason why we make this assumption is due to the difficulty in determining what the conditional joint distribution of the features is given a particular class. Is the joint distribution a multivariate Poisson? A complex mixture of many? Some other multivariate counting distribution? What is the dependence structure, and how should we represent it? What if we have a thousand features? How do we represent the dependence structure now in a reasonable way? Is the dependence structure so complicated that representing it reasonably is more annoying and/or time-consuming than just assuming independence?

These questions are not trivial and are very difficult, if not impossible, to answer. Thus, we make the independence assumption to simplify the problem. We may not know the joint conditional distribution but perhaps we can derive and/or visualize the marginal conditional distributions for each feature. This, of course, comes at the cost of assuming independence between features which is highly unlikely to be true (especially for word frequencies) but nevertheless occasionally effective for some problems (kind of).