1
$\begingroup$

I'm doing a Naive Bayes prediction model where I've 666 features to select from.

I've tried the SelectKBest chi2 test but it's a features-label test whereas what I'm trying to do is to find out the independency and also correlation feature-features. As the assumption to use Naive Bayes is that variables are all independent to each other, I have to remove features that are dependent/correlated to each other.

What are the other independence tests that I should look into apart from $chi^2$? (I have continuous features)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This package is very useful, this does the feature engg. for you, gives you the important variables. sample code would be something like this.

    set.seed(123)
    boruta.input <- Boruta(target variable~., data = training_data, doTrace = 2)
    print(boruta.input)

    #plot a graph for better understanding
    plot(boruta.input, xlab = "", xaxt = "n")
    lz<-lapply(1:ncol(boruta.input$ImpHistory),function(i)
    boruta.input$ImpHistory[is.finite(boruta.input$ImpHistory[,i]),i])
    names(lz) <- colnames(boruta.input$ImpHistory)
    Labels <- sort(sapply(lz,median))
    axis(side = 1,las=2,labels = names(Labels),
    at = 1:ncol(boruta.input$ImpHistory), cex.axis = 0.7)

By using this you can get all the important features but the downfall is it takes time if you have more data In my case my data consists 40 features and 200,000 records it took almost 2 hours but the results were good.

For better understanding you can go through this link

--

I think you can use this test too, in the description it tells that it can be applied on numerical data too.

If you have data which is Nominal data, you can use G-test

Attaching one more link along with this, which consists of tests for different kinds of features. Might be helpful in the future Link

I have a question, why aren't you applying PCA(or any DR Techniques) on the data-set? (assuming that you din't try any DR Techniques)

If you apply DR Techniques, I think you might get better Components which would aid you in getting better results/better understanding of data.

As you mentioned that you were using NB Classifier, why did you choose that, is there any specific reason? As you know that there are many other Classifiers which might outrun NB Classifier. In my case I used NB Classifier for twitter sentiment mining, as those are short sentences and NB Classifier best suited for my analysis.

Hope my answer is helpful!

NB: Naive Bayes

DR: Dimensionality Reduction

Improve this answer
$\endgroup$
3
  • $\begingroup$ hey thanks for your fast reply. I am using recursive features elimination currently after chi2 test. but I did try PCA but it doesnt seems to boost the accuracy alot. My data is time series telco data (upload/download bytes, duration etc.) , I apply features engineering to extract daily characteristic which results in 666 features. Hence, I'm looking for a robust features selection method to choose significant features that can captures the trend of the data to predict telco equipment faulty. $\endgroup$
    – jynn
    Commented Nov 10, 2017 at 9:01
  • $\begingroup$ cause I'm thinking to use independence test to remove first round of features then only apply feature selection method. there're too many features selection method out there, the best one i get is random forest regressor recursive feature elimination. I even thought of using bayesian optimization to tune the parameter but I dont think is neccessary. Please advise me on what I should do here $\endgroup$
    – jynn
    Commented Nov 10, 2017 at 9:20
  • $\begingroup$ Thanks alot! this is helpful for me! will definitely try it out! $\endgroup$
    – jynn
    Commented Nov 10, 2017 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.