3
$\begingroup$

I have two concept related questions related to Naïve Bayes.

Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?

It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.

Regards, Akshit Bhatia

$\endgroup$
1
  • 1
    $\begingroup$ Ask the decision tree question separately, it's not related to NB $\endgroup$
    – Erwan
    Feb 1 '20 at 14:12
1
$\begingroup$

Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?

Imagine a classifier for sentiment analysis. For a strongly positive word like $w=great$, the conditional probability $p(w|pos)$ is going to be quite high whereas $p(w|neg)$ is going to be quite low, so the posterior $p(pos|d)$ for a document $d$ containing this word is likely to be much higher than $p(neg|d)$.

Now what happens with a neutral word $w=today$? Neither $p(w|pos)$ or $p(w|neg)$ is going to be much higher than the other. So all other things being equal, the difference between the two posterior probabilities is not going to depend much on this word compared to other more relevant words, for instance "great".

It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.

In practice the independence assumption almost never holds with real data. For example words in a text actually depend on each other, that's how they make sense together in a sentence. This is true for entities like "Hong Kong" but also for virtually any sentence. For instance "I love chocolate but you hate it" doesn't mean that same as "You love chocolate but I hate it", or "it chocolate you hate love but" which means nothing. NB will treat all these variants the same way: basically the model assumes independence for the purpose of making things simpler and easier to compute, and it turns out that it works quite well in general, despite the massive simplification.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.