Decision Tree : why do unused features impact predictions?

Question

I am currently building a tree, with 10 features but setting max_depth = 2 in sklearn.tree.DecisionTreeClassifier.

Since only tree features are explicitly used to make predictions I wondered about dropping uneeded columns.

Counter-intuitively, the absence of just 1 of the 7 variables not highlighted by the tree is enough to change the accuracy of the predictions, albeit marginally.

Looking around I found that

Variables that are not used in any split can still affect the decision tree [...] It is possible for a variable to be used in a split, but the subtree that contained that split might have been pruned.

However, max_depth does not technically prune according to one answer on this website.

So what could be the reason?

Answer 1

My guess is multicollinearity: some of the predictors are correlated and removing one may change the relationships between the remainining predictors and the response variable. Also while calculating the split of each node the algorithm use metrics to decide the feature to split, e.g. Gini impurity (Gini Impurity) or information gain (analyticsvidhya: Simple Ways to Split a Decision Tree, towardsdatascience: How Decision Trees Split Nodes); for this calculation all the features contributed to the calculation.

Answer 2

I don't understand Memristor's answer since the removed feature was not even included in the splits of the DecisionTree, so it's interaction with other features shouldn't change anything. Also, the metrics are only calculated using one feature, not all the features. The model iterates over all features and identifies which is the best by using this metric. (Sorry didn't have the reputation to write this in a comment)

For your question, assuming the only hyperparameter you have set is max_depth, then one possible reason I can think of is that two features have tied metric scores. Then each time you have called the model, it may have chosen a different feature between these two ties. For further explanation refer to this other question.

However, a better way to understand the differences is simply to visualise the decision tree as it's only 3 splits:

import graphviz
from sklearn import tree

model = tree.DecisionTreeClassifier(max_depth=2)
model.fit(X,y)

out = tree.export_graphviz(model)
graphviz.Source(out)

This way you can check if the variance from removing an unused feature is caused by the model changing, or maybe RNG in the accuracy metric you're using.