After applying PCA to reduce the number of features, I am using a DecisionTreeClassifier for a ML problem

enter image description here Additionally I want to compute the feature_importances_. However, with each iteration of the DecisionTreeClassifier, the feature_importances_ change.

Iteration #1

enter image description here

Iteration #2

enter image description here

Why would it change? I thought the initial split was made on a feature to "produce the purest subsets (weighted by their size)". Acting on the same training set, why would that change?

Thanks in advance for any help.


From sklearn.tree.DecisionTreeClassifier help:

The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

Also, you might want to have a look at my critique on feature importance.

  • 1
    $\begingroup$ Ah, in the future I need to read to the very end of the documentation ;) Thank you very much! $\endgroup$ – Chris Tennant Jan 8 '19 at 19:01
  • $\begingroup$ Even if you keep random_state, the features are getting changed for every iterations. $\endgroup$ – Swaminathan Sekar Sep 29 '20 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.