4
$\begingroup$

What is the standard or what method do you use to select part of features.

For ex: Using random forest, I got the following feature importances:

a : 25.4884726
b : 17.2736393
c : 12.3493490
d : 8.9383737
e : 8.1083837
f : 6.8272717
g : 4.1999999
...
...
...

For final prediction, you need to select only few features from the above. How do you decide the following:

  1. No. of features that you choose?
  2. Does the score matter when you choose?
Improve this question
$\endgroup$
2
  • $\begingroup$ Feature importance is selected by the entropy values in Decision trees. Not sure for RFs. Entropy averaging? $\endgroup$
    – Dawny33
    Commented Jul 12, 2016 at 6:31
  • 2
    $\begingroup$ Go from the top and do cross-validation to observe the score. In doubt, select fewer features. $\endgroup$
    – Gere
    Commented Jul 13, 2016 at 6:36

2 Answers 2

Reset to default
3
$\begingroup$

This is an important problem, since many feature selection methods return feature scores/importances rather than a finite feature set. I currently know three approaches:

  • choose the k best attributes (fixed number defined)
  • choose the best k * 100% of attributes (relative number of features defined)
  • make a cutoff at the biggest difference in feature scores: all features are ordered according to their score and a split is made at the largest difference between one score and the next lower (biggest loss in importance)

You find a nice implementation doing this in R with the FSelector package.

I think, there are some research works tackling exactly this problem and may suggest better approaches, but I hadn't had the time to go deeper into this.

Improve this answer
$\endgroup$
-2
$\begingroup$

  1. No. of features that you choose?

    Its depends on the number of classes you have to predict, let's explain this with example: consider we have 5 classes(labels) in our dataset and we choose only one feature, so tree has only one parent node and two leaf node which will accommodate only two classes and hence the accuracy decrease sharply, similarly if we choose only two features then we can only accommodate three classes and so one, so you have to consider more number of features as of classes and there after you have to test it with accuracy till there is no change in accuracy by adding extra features in the model.

  2. Does the score(Variable Importance) matter when you choose?

    Yes, the score matter when deciding the features that you choose, since its depends on the Variable Importance of a feature is computed as the average decrease in model accuracy on the out of bag samples when the values of the respective feature are randomly permuted, so if you choose only the lower score variables for features then the accuracy sharply decreases.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Conter-example to your first part : I can predict 4 classes with one feature, let's say the age : baby : [0, 3[, child : [3, 13[, teenager : [13, 18[, adults = [18 and above[. This is because the tree might definitely have more than one parent node and two leaves. $\endgroup$
    – Igor OA
    Commented Jul 11, 2016 at 8:06
  • $\begingroup$ @IgorOA Thanks for pointing it out, you are totally right, I missed it. I have tried creating a decision tree based on your comment but unable to accommodate same features on multiple node in tree, have a look at my github repo R code and fancy decision tree are attached with random sample dataset. $\endgroup$
    – krishna Prasad
    Commented Jul 12, 2016 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.