2
$\begingroup$

I have a dataset with 28 attributes and 7 class values. I want to know if its possible to find out the most important attribute(s) for deciding the class value, for each class.

For example an answer could be: Attribute 2 is most important for class 1, Attribute 6 is most important for class 2 etc. Or an even more informed answer could be: Attribute 2 being below 0.5 is most important for class 1, Attribute 6 being above 0.75 is most important for class 2 etc

My initial approach to this was to build a decision tree on the data and find the node that had the largest information gain/gain ratio for each class and that would be the most determining factor for that class. The problem with this is that the decision tree implementations I have found don't give the information gain/gain ratio for each node and as this is time bound I don't have the time to implement my own version. My current thought is to create multiple datasets which are all one class vs the rest and then perform attribute selection (eg. information gain) on them to find the most important attribute. Is this the right direction to go down or is their a better option?

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Covariance matrix for each attribute and the output. $\endgroup$
    – Green Falcon
    Commented Oct 16, 2018 at 12:47
  • $\begingroup$ @Media could you expand on this please? I thought a covariance matrix would only let me know the correlation between attributes. If you have a solution you could add it as an answer $\endgroup$
    – Nate
    Commented Oct 16, 2018 at 13:15
  • $\begingroup$ Important in what way? That best split your data (given that you were talking about decision trees and splits)? There are ML algorithms that return feature importances, for ex. random forests. $\endgroup$
    – user2974951
    Commented Oct 16, 2018 at 13:37
  • $\begingroup$ Important in the way that I want to know the attributes that have the biggest say in determining each class. I believe this means the best split of the data. As far as I know random forest doesn't return feature importance for each class, or am I wrong about this? $\endgroup$
    – Nate
    Commented Oct 16, 2018 at 13:52
  • $\begingroup$ @Nate, "My current thought is to create multiple datasets which are all one class vs the rest and then perform attribute selection (eg. information gain) on them to find the most important attribute." ---> if you're happy with that definition of "most important" then yes. Your work is to define what you mean by "most important", but your approach (classification ONE vs ALL_OTHERS) is a good one. $\endgroup$
    – f.g.
    Commented Oct 16, 2018 at 17:33

4 Answers 4

Reset to default
1
$\begingroup$

If you must split the dataset for each class then I suggest you try PCA. Principle Component Analysis is basically used for dimensionality reduction in the sense that it gives you the subset of attributes which represent the distribution of data best. You could use that for all classes and in that way get the attributes which affect the class data distribution the best.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Could also 'rank' the importance of the attributes by looking at each attributes coefficient? $\endgroup$
    – Nate
    Commented Oct 16, 2018 at 16:12
  • $\begingroup$ But does PCA retain knowledge about the original features and how they map to the reduced features? $\endgroup$
    – rjurney
    Commented Feb 18, 2020 at 4:44
1
$\begingroup$

Let's say that in this way. The easiest way to ascertain the relation among different features and the outputs is to use covariance matrix. You can even visualise the data for each class. Take a look at the following image.

enter image description here

Suppose that the vertical axis is the output and the horizontal axis is for one of the features. As you can see, having knowledge about the features informs us of the changes in the output. Now, consider the following illustration.

enter image description here

In this figure, you can see that considering this typical feature does not inform you of the changes in the output.

Another approach can be using PCA which finds the appropriate features itself. What it does is finding a linear combination of important features which are more relevant to the output.

Improve this answer
$\endgroup$
0
$\begingroup$

Or an even more informed answer could be: Attribute 2 being below 0.5 is most important for class 1, Attribute 6 being above 0.75 is most important for class 2 etc

A way of doing this is by discretizing your continuous variables into histograms. Every bin of a histogram can thereafter be considered as a separate variable and its importance can be easily found using standard decision tree implementations such as the one in sklearn that provides a _feature_importances attribute. This will offer you insight on the important regions of each variable.

This is demonstrated in Figure 9 in this paper.

Welcome to the site, Nate!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ah yes good idea, the only thing is that wouldn't the _feature_importances attribute tell me the importance of the feetures for the whole model, rather than the importance per class. $\endgroup$
    – Nate
    Commented Oct 16, 2018 at 14:23
  • $\begingroup$ when a feature is important to classify a sample into class A, it means that it is also important for not classifying the same sample in class B. I mean that you shouldn't ask "which feature is important to classify the sample in class A and which one to classify it in class B" BUT "which feature is important to classify the sample correctly". This is feature importance. $\endgroup$
    – pcko1
    Commented Oct 16, 2018 at 14:35
0
$\begingroup$

It is difficult to get the answer to "which one is the most important" for every class, normally, the distintion between is "between classes" not "for a specific class".

I use the feature importance from xgboost, this method allows to measure which feature participates in more trees of the boosted forest xgboost has. There is even, the possibility of plotting these importances resulting in very nice data to show.

Feature importance in XGBoost

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.