Contributions of each feature in classification?

Question

I have some features and I am using Weka to classify my instances.

For example I have:

Number of adj number of adverb number of punctuation

in my feature set. However, I would like to know the contribution of each feature in the feature set. So what metrics or parameters are helpful to get the contribution of features?

Answer 1

This is called feature ranking, which is closely related to feature selection.

• feature ranking = determining the importance of any individual feature
• feature selection = selecting a subset of relevant features for use in model construction.

So if you are able to ranked features, you can use it to select features, and if you can select a subset of useful features, you've done at least a partial ranking by removing the useless ones.

This Wikipedia page and this Quora post should give some ideas. The distinction filter methods vs. wrapper based methods vs. embedded methods is the most common one.

---

One straightforward approximate way is to use feature importance with forests of trees:

Other common ways:

• recursive feature elimination.
• stepwise regression (or LARS Lasso).

If you use scikit-learn, check out module-sklearn.feature_selection. I'd guess Weka has some similar functions.