Sorry for vague heading for the question. My question is that, is there any way to compare features (or attributes) used in machine learning algorithm? I have used Naive Bayesian classifier for binary classification which consists of total 6 features. I want to compare this features with one another and also list out features in priority order of their significance in models(how important that feature is). For example we use information gain or gini index to decide how effective feature is in CART. How can I compare this in naive bayes? I have read about confusion matrix and f measure but they are good at defining goodness of models and not features.
$\begingroup$
$\endgroup$
5
-
1$\begingroup$ So just to clarify, you want to compare the importance of each variable on an algorithm by algorithm basis? $\endgroup$– plumbus_bouquetCommented Feb 6, 2018 at 19:39
-
1$\begingroup$ Feature importance and feature selection are well studied concepts in ML. The general idea is that you evaluate the discriminative power of a feature. Most implementations even support those directly as a method to the class. Please note however that methods that use an independence assumption (NB) might value feature very different from methods that don't (CART). Also, this link: stats.stackexchange.com/questions/6478/… $\endgroup$– S van BalenCommented Feb 6, 2018 at 20:23
-
$\begingroup$ @SvanBalen Yes, i know feature importance varies from algorithm to algorithm. That's exactly what I want to explain by displaying graph analysis of importance. And yes I know I can use information gain in CART(Let me edit this in the main question). But what about Naive Bayes? $\endgroup$– ScorpionkCommented Feb 7, 2018 at 12:52
-
$\begingroup$ @plumbus_bouquet Yes you got it right. $\endgroup$– ScorpionkCommented Feb 7, 2018 at 12:52
-
$\begingroup$ Good point: Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. In fact the feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn), the variability of those probabilities should express the importance of those features. $\endgroup$– S van BalenCommented Feb 7, 2018 at 14:08
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. The feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn for instance), the variability of those probabilities with respect to the classes should express the importance of those features.
answered Feb 8, 2018 at 7:31