I have a dataset with 20 features(columns that is). I create a few models pairs with a subset of these parameters.

For example: If I have 6 columns (named A, B, C, D, E, F) with 10k lines of data, one of model in the pair will have (A, B, C) with all 10k lines of data and the other (A, B, C) with maybe say 6k lines of data. Yet another pair will have the parameters (B, C, E), however, the number of lines is always fixed at 10k and 6k.

I am limited to using a sci-kit learn's decision tree and this cannot be changed. I use the following function to calculate my features:

model = clf_gini.best_estimator_

My question is: **Is there any formal, programmatic way to compare feature importance across model pairs using sci-kit, other than intuition? Also, does it make sense to compare to models trained on some similar some different parameters? **

Please let me know if the question is ambiguous, I will clarify.

  • $\begingroup$ Isn't is possible to compare the Gini index of the features in both models directly in terms of e.g. Gini(A1) < Gini(A2)? What exactly are you aiming at? Generally it does make sense to compare differently trained models, since you can usually not be sure that you have the best configuration for your algorithm. $\endgroup$ – André Sep 3 '18 at 10:06

This may not be the answer you are hoping for (I can't leave a comment because my reputation is too low) but I believe feature importance in Sci-Kit Learn is derived from the average depth at which each feature appears. I don't know of any built-in functions in Sci-Kit Learn but perhaps you could scale the feature importance using the depth of the tree and the number of features?

This article may help with developing a comparative measure of feature importance: https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3.

  • $\begingroup$ Well, does it make sense to compare to models trained on some similar some different parameters? $\endgroup$ – Jishan Sep 1 '18 at 23:46
  • $\begingroup$ My guess is that it does make sense. In my experience, features tend to be reliably important across different datasets and in combination with other features, or they do not. The part I do not know is how to make direct comparisons and this is where I hoped the link might help you get started. I am sorry if this is not a great answer; I really just wanted to leave a comment, rather than an answer, but my reputation is too low. $\endgroup$ – from keras import michael Sep 2 '18 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.