Improving the performace of the Naive Bayes classifier by decorrelating the data

Question

I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $P(a_1, a_2 | c_1) = P(a_1 | c_1)P(a_2 | c_1)$ which is not necessarily true.

What if we applied a transformation into the space using decorrelation which will result in $Cov(a_1, a_2) = 0$ for all features. We have that if $a_1$, $a_2$ independent, then $Cov(a_1, a_2) = 0$. Although the converse does not strictly follow (e.g. consider the two dependent random variables $X \sim Norm(0, 1)$, and $X^2$ whose covariance is 0), the new features will be closer to being independent.

Answer 1

As you pointed out, a null covariance does not guarantee that variables are independent. You can have strongly dependent variables which show a covariance equal to 0. See a famous plot taken from Wikipedia page on Pearson correlation coefficient

A standard way to uncorrelate variables is to perform principal component analysis on the dataset, and use all components which are, by definition, uncorrelated. But please note that the Naive Bayes assumption is conditional independence, so independence for each class. In the general case, PCA projection would not be the same depending on the selected class.

Regarding the performance of Naive Bayes, I see no reason that a simple transformation should improve the model. This is supported by some reasonably well-received answers on similar questions, for instance this one on Stack Overflow, or this other one on Cross Validated.

However, I have read several reports of people getting increased accuracy when performing PCA prior to Naive Bayes. See here for instance, for a few exchanges on the subject, within a community of which trustworthiness is unknown to me; or search "PCA naive bayes" through your web search engine.

So in the end, I have no mathematical evidence to support this, but my feeling would be that the model performance variation before/after transformation could depend on the problem, especially the directions of principal components for each class.

Perhaps, if you perform some tests, you could share the result here. Otherwise, I think you could get answers with more mathematical background on Cross Validated.