Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have two concept related questions related to Naïve Bayes.
Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?
It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.
Regards, Akshit Bhatia
Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?
Imagine a classifier for sentiment analysis. For a strongly positive word like $w=great$, the conditional probability $p(w|pos)$ is going to be quite high whereas $p(w|neg)$ is going to be quite low, so the posterior $p(pos|d)$ for a document $d$ containing this word is likely to be much higher than $p(neg|d)$.
Now what happens with a neutral word $w=today$? Neither $p(w|pos)$ or $p(w|neg)$ is going to be much higher than the other. So all other things being equal, the difference between the two posterior probabilities is not going to depend much on this word compared to other more relevant words, for instance "great".
It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.
In practice the independence assumption almost never holds with real data. For example words in a text actually depend on each other, that's how they make sense together in a sentence. This is true for entities like "Hong Kong" but also for virtually any sentence. For instance "I love chocolate but you hate it" doesn't mean that same as "You love chocolate but I hate it", or "it chocolate you hate love but" which means nothing. NB will treat all these variants the same way: basically the model assumes independence for the purpose of making things simpler and easier to compute, and it turns out that it works quite well in general, despite the massive simplification.
