Decision Trees split based on which feature and which cut-off value creates the largest mean decrease in impurity (assuming hyperparameter split="best", criterion="gini"). Now take for example, you have two identical columns in your dataset. For each split, both these columns will have an equal mean impurity decrease. How does the algorithm choose which feature to use?
I have tested this out with the Titanic dataset, and have found that both features have a non-zero feature importance, so they both have been used in at least one split:
X["Duplicate"] = X["Pclass"]
Inspecting the tree, I cannot find any specific patterns that indicate why one feature is chosen over another. Is it by random choice?
And is there any way to make it so that only one of these duplicate features is used in the algorithm? (without simply removing one of the duplicates)
Note: This question specifically refers to the sklearn implementation of the decision tree algorithm.