12
$\begingroup$

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many levels (on the order of 100-1000). Given that one-hot encoding creates a dummy variable for each level, the feature importances are for each level and not each feature (column). What is a good way to aggregate these feature importances?

I thought about summing or getting the average importance for all levels of a feature (probably the former will be biased towards those features with more levels). Are there any references on this issue?

What else can one do to decrease the number of features? I am aware of group lasso, could not find anything easy to use for scikit-learn.

Improve this question
$\endgroup$
2
  • $\begingroup$ Can anyone answer the question of whether summing the variable importance of each level of the categorical variable makes sense? $\endgroup$
    – see24
    Jul 30, 2018 at 15:35
  • $\begingroup$ @see24 No you can't just sum them: stats.stackexchange.com/questions/314567/… $\endgroup$
    – Dan
    Apr 24, 2019 at 14:35

1 Answer 1

Reset to default
5
$\begingroup$

It depends on how you're one-hot encoding them. Many automated solutions for that will name all the converted booleans with a pattern so that a categorical variable called "letter" with values A-Z would end up like:

letter_A, letter_B, letter_C, letter_D,....

If after you've figured out feature importance you've got an array of feature and the associated weight/importance, I would analyze the array and perhaps sum up the feature importance weights for anything starting with "letter%".

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ Isn't the sum giving an advantage to those features with more levels? $\endgroup$
    – user90772
    Apr 6, 2017 at 14:34
  • $\begingroup$ Hmm, good point. Maybe sum it up then divide by the number of levels/one-hot encoded variables to get an "average" importance. $\endgroup$
    – CalZ
    Apr 6, 2017 at 15:11
  • 3
    $\begingroup$ I thought about this some more and it depends on how the importance is scored. In some cases, the value for each feature is a relative weight where the whole set totals to 1. In that case, I think it would make sense to sum up the one-hot features. If the score for the feature was more like a regression coefficient and not weighted relative to the net effect, then averaging would probably be better. $\endgroup$
    – CalZ
    Apr 6, 2017 at 23:42
  • $\begingroup$ Thank you for the reply. Given that I am quite new to the area, I thought that this was a standard thing for people in data science but either it is not what I should be doing to assess feature importance of a column or this post did not get enough views. In any case, thank you! $\endgroup$
    – user90772
    Apr 7, 2017 at 12:12
  • 1
    $\begingroup$ Many people advocate looking at the model's internals as a black box and evaluating the performance instead. In certain cases (e.g. neural networks) this is because you can't really examine it deeply. For some where you can easily get a view of which features are important (e.g. linear regression), you can easily be mislead (see: stats.stackexchange.com/questions/105114/…). I think that's why people sometimes shy away from looking at individual feature importance. $\endgroup$
    – CalZ
    Apr 7, 2017 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.