1
$\begingroup$

Decision Trees split based on which feature and which cut-off value creates the largest mean decrease in impurity (assuming hyperparameter split="best", criterion="gini"). Now take for example, you have two identical columns in your dataset. For each split, both these columns will have an equal mean impurity decrease. How does the algorithm choose which feature to use?

I have tested this out with the Titanic dataset, and have found that both features have a non-zero feature importance, so they both have been used in at least one split:

X["Duplicate"] = X["Pclass"]

DecisionTreeClassifier().fit(X, y)
DecisionTreeClassifier().feature_importances_

Inspecting the tree, I cannot find any specific patterns that indicate why one feature is chosen over another. Is it by random choice?

And is there any way to make it so that only one of these duplicate features is used in the algorithm? (without simply removing one of the duplicates)

Note: This question specifically refers to the sklearn implementation of the decision tree algorithm.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
0
$\begingroup$

From the docstring for the random_state parameter:

The features are always randomly permuted at each split, even if splitter is set to "best". [...] The best found split may vary across different runs, even if max_features=n_features. That is the case, if the improvement of the criterion is identical for several splits and one split has to be selected at random.

So

Is it by random choice?

Yep.

And is there any way to make it so that only one of these duplicate features is used in the algorithm? (without simply removing one of the duplicates)

Not readily. This is one reason to consider removing highly correlated features even for a tree-based model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I see, I missed this clear explanation in the documentation. Is this also the reason why it's recommended to remove correlated features before using RFECV? Since the feature importance is roughly distributed evenly between highly correlated features, it sometimes keeps both correlated features and hurts model performance. Of course the "sometimes" is a bit more nuanced than that, but is this the basic reasoning. $\endgroup$
    – AvanishM
    Commented Feb 10 at 14:13
  • $\begingroup$ @AvanishM that's another question, but at the very least yes, there are things to think about for correlated features in feature selection. $\endgroup$
    – Ben Reiniger ♦
    Commented Feb 10 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.