2
$\begingroup$

Sorry for vague heading for the question. My question is that, is there any way to compare features (or attributes) used in machine learning algorithm? I have used Naive Bayesian classifier for binary classification which consists of total 6 features. I want to compare this features with one another and also list out features in priority order of their significance in models(how important that feature is). For example we use information gain or gini index to decide how effective feature is in CART. How can I compare this in naive bayes? I have read about confusion matrix and f measure but they are good at defining goodness of models and not features.

$\endgroup$
  • 1
    $\begingroup$ So just to clarify, you want to compare the importance of each variable on an algorithm by algorithm basis? $\endgroup$ – plumbus_bouquet Feb 6 '18 at 19:39
  • 1
    $\begingroup$ Feature importance and feature selection are well studied concepts in ML. The general idea is that you evaluate the discriminative power of a feature. Most implementations even support those directly as a method to the class. Please note however that methods that use an independence assumption (NB) might value feature very different from methods that don't (CART). Also, this link: stats.stackexchange.com/questions/6478/… $\endgroup$ – S van Balen Feb 6 '18 at 20:23
  • $\begingroup$ @SvanBalen Yes, i know feature importance varies from algorithm to algorithm. That's exactly what I want to explain by displaying graph analysis of importance. And yes I know I can use information gain in CART(Let me edit this in the main question). But what about Naive Bayes? $\endgroup$ – Scorpionk Feb 7 '18 at 12:52
  • $\begingroup$ @plumbus_bouquet Yes you got it right. $\endgroup$ – Scorpionk Feb 7 '18 at 12:52
  • $\begingroup$ Good point: Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. In fact the feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn), the variability of those probabilities should express the importance of those features. $\endgroup$ – S van Balen Feb 7 '18 at 14:08
0
$\begingroup$

Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. The feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn for instance), the variability of those probabilities with respect to the classes should express the importance of those features.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.