# Independence/correlation test between features (not feautre-label)

I'm doing a Naive Bayes prediction model where I've 666 features to select from.

I've tried the SelectKBest chi2 test but it's a features-label test whereas what I'm trying to do is to find out the independency and also correlation feature-features. As the assumption to use Naive Bayes is that variables are all independent to each other, I have to remove features that are dependent/correlated to each other.

What are the other independence tests that I should look into apart from $chi^2$? (I have continuous features)

This package is very useful, this does the feature engg. for you, gives you the important variables. sample code would be something like this.

    set.seed(123)
boruta.input <- Boruta(target variable~., data = training_data, doTrace = 2)
print(boruta.input)

#plot a graph for better understanding
plot(boruta.input, xlab = "", xaxt = "n")
lz<-lapply(1:ncol(boruta.input$ImpHistory),function(i) boruta.input$ImpHistory[is.finite(boruta.input$ImpHistory[,i]),i]) names(lz) <- colnames(boruta.input$ImpHistory)
Labels <- sort(sapply(lz,median))
axis(side = 1,las=2,labels = names(Labels),
at = 1:ncol(boruta.input\$ImpHistory), cex.axis = 0.7)


By using this you can get all the important features but the downfall is it takes time if you have more data In my case my data consists 40 features and 200,000 records it took almost 2 hours but the results were good.

For better understanding you can go through this link

# --

I think you can use this test too, in the description it tells that it can be applied on numerical data too.

If you have data which is Nominal data, you can use G-test

Attaching one more link along with this, which consists of tests for different kinds of features. Might be helpful in the future Link

I have a question, why aren't you applying PCA(or any DR Techniques) on the data-set? (assuming that you din't try any DR Techniques)

If you apply DR Techniques, I think you might get better Components which would aid you in getting better results/better understanding of data.

As you mentioned that you were using NB Classifier, why did you choose that, is there any specific reason? As you know that there are many other Classifiers which might outrun NB Classifier. In my case I used NB Classifier for twitter sentiment mining, as those are short sentences and NB Classifier best suited for my analysis.