Really confused with characteristics of Naive Bayes classifiers?

Question

Naive Bayes classifiers have the following characteristics-:

1. They are robust to isolated noise points because such points are averaged out when estimating contiditional probabilities from data. Naive Bayes classifiers can also handle missing values by ignoring the example during model building and classification.

2. They are robust to irrelevant attributes. If X_i is an irrelevant attributet then P(X_i/Y) becomes almost uniformly distributed. The class conditional probability for X_i has no impact on overall computation of posterior probability.

I barely understand anything said here. The book doesn't even provide examples and most of the resources available online are exact photocopies of this book. None of those materials dive deep into these things and actually explain this.

Can you guys help me out here to explain what this means via examples? I will be really glad. I have been banging my head against the wall to get this concept for a long time. I will be glad with some recommended reading that I need to do as well.

Answer 1

Before explaining I just want to point out that these points are only about the advantages of NB classification, there are also disadvantages (in particular NB is very prone to overfitting).

They are robust to isolated noise points because such points are averaged out when estimating conditional probabilities from data.

An "isolated noise point" has features values which differ a lot from the majority of the points. Since by definition there are very few such points, their values play a very small role in the conditional probability across all the points.

In my opinion this argument is a bit questionable, because isolated points can also cause a NB model to overfit due to rare features values (this applies to Bernouilli NB, probably not to Gaussian NB).

Naive Bayes classifiers can also handle missing values by ignoring the example during model building and classification.

For a particular feature $x_i$, if some of the instances do not have a value for this feature it's still possible to calculate the conditional probabilities $p(x_i|Y)$ using the other instances. Interestingly, the model can ignore different instances for different features, which makes NB a bit more robust (i.e. kind of flexible) than other methods.

They are robust to irrelevant attributes. If X_i is an irrelevant attribute then P(X_i/Y) becomes almost uniformly distributed. The class conditional probability for X_i has no impact on overall computation of posterior probability.

An "irrelevant feature" is a feature which does not help predicting the class, which means that $p(x_i,Y) \approx p(x_i)p(Y)$ (the variables are close to independent). This is equivalent to $p(x_i|Y) \approx p(x_i)$, therefore the probability for this feature will be identical for every possible class $Y=y_k$ so it gives the same weight to every class.

Note: I think saying that "$P(X_i|Y)$ becomes almost uniformly distributed" is ambiguous at least, because normally $p(a|b)$ means the distribution of varying value $a$ given fixed value $b$. In my opinion it should be: $P(X_i|Y)$ becomes almost identical to $P(X_i)$.