Unexpected results from scikit learn regression decision tree

Apologies for this newbie question. I have a scikit learn DecisionTreeRegressor with muti-variable output. If the output is in the format [ output_var1, output_var2 ], where each variable is a continuous number not an integer, why the result is [1, 1] instead of [1.5, 1.5] ? What needs to be changed in this model to get [1.5, 1.5] ?

from sklearn.tree import DecisionTreeRegressor

X = [ [1,1], [2,2], [3,3] ]
y = [ [1,1], [2,2], [3,3] ]

print('X:' , X)
print('-----------------------------')
print('y:' , y)
print('-----------------------------')

regr = DecisionTreeRegressor()
regr.fit(X, y)

X_test = [ [1.5, 1.5] ]
print('X_test:' , X_test)
print('-----------------------------')

y_result = regr.predict(X_test)
print('y_result:' , y_result )


Result:

X: [[1, 1], [2, 2], [3, 3]]
-----------------------------
y: [[1, 1], [2, 2], [3, 3]]
-----------------------------
X_test: [[1.5, 1.5]]
-----------------------------
y_result: [[1. 1.]]

• You should visualize your tree, then it will be clear why. But in short, sklearn's decision trees will split into groups "as much as it makes sense", and for each group it will return the mean value of that group. Jun 28, 2019 at 12:36
• but isn't the mean 1.5? Jun 28, 2019 at 12:43
• Since it is outputting "1." and not "1" it is not outputting an integer but a float that happens to be the value "1". Jun 28, 2019 at 13:17
• @Pedro I saw that, but why is it 1 instead of 1.5? Jun 28, 2019 at 13:51
• I don't understand why you're expecting 1.5? The tree presumably perfectly fits on the training data, and so the only outputs that are possible are those in the training y. Jun 28, 2019 at 14:52

This is the visualization for the inducted tree from your data:

As you can see, it can never predict [[1.5, 1.5]]. Your data is just too pure, the inducted tree will fit it perfectly, having all leaf nodes with only one sample. If a leaf node has more than one sample in it, then the predicted value is the mean of the y values for those samples.

Furthermore, trees are not the best model to induct the identity function ($$y = f(x) = x$$). Trees work by creating rules and dividing the feature space orthogonally (i.e. only vertical or horizontal lines in the feature space).

In other words, your model didn't learn that f(1.5, 1.5) = (1.5, 1.5) because a) there's not enough data, b) trees are not good at doing that anyway.

• Why did the DecisionTreeRegressor build the nodes with 1.5 and 2.5? Jul 4, 2019 at 19:36
• @ps0604 the first thing the tree-building algorithm does is checking splits. It could split [[1], [2, 3]] or [[1, 2], [3]]. Both provide the same entropy; in this case it used the first, but it can be considered random. For splitting the partitions [1] and [2, 3], it uses a rule with the mean between the highest of the lower node and the lower of the higher node, so (1 + 2) / 2. The same priciple applies to splitting the data [[2], [3]] in the intermediary node. Jul 4, 2019 at 20:23

The data is being overfitting by the model, the model is memorizing the training data. Given the very few training samples, the decision tree can not learn the general pattern. If the model is given more training data with greater variation, it can learn the general pattern.

One approach is not to change the model but change the data. You can simulate a lot of random data and get the approximately correct results:

from sklearn.tree import DecisionTreeRegressor
import numpy as np

nums = np.random.random(size=100_000)*3
X = list(zip(nums, nums))
y = list(zip(nums, nums))
regr = DecisionTreeRegressor()
regr.fit(X, y)
X_test = [ [1.5, 1.5] ]
y_result = regr.predict(X_test)
assert np.isclose(y_result, X_test, atol=.001)


I afraid that Decision Tree does not suitable. I recommend read about linear regression.

https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py