4
$\begingroup$

Apologies for this newbie question. I have a scikit learn DecisionTreeRegressor with muti-variable output. If the output is in the format [ output_var1, output_var2 ], where each variable is a continuous number not an integer, why the result is [1, 1] instead of [1.5, 1.5] ? What needs to be changed in this model to get [1.5, 1.5] ?

from sklearn.tree import DecisionTreeRegressor

X = [ [1,1], [2,2], [3,3] ]
y = [ [1,1], [2,2], [3,3] ]

print('X:' , X)
print('-----------------------------')
print('y:' , y)
print('-----------------------------')

regr = DecisionTreeRegressor()
regr.fit(X, y)

X_test = [ [1.5, 1.5] ]
print('X_test:' , X_test)
print('-----------------------------')

y_result = regr.predict(X_test)
print('y_result:' , y_result )

Result:

X: [[1, 1], [2, 2], [3, 3]]
-----------------------------
y: [[1, 1], [2, 2], [3, 3]]
-----------------------------
X_test: [[1.5, 1.5]]
-----------------------------
y_result: [[1. 1.]]
Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ You should visualize your tree, then it will be clear why. But in short, sklearn's decision trees will split into groups "as much as it makes sense", and for each group it will return the mean value of that group. $\endgroup$
    – Mephy
    Commented Jun 28, 2019 at 12:36
  • $\begingroup$ but isn't the mean 1.5? $\endgroup$
    – ps0604
    Commented Jun 28, 2019 at 12:43
  • 1
    $\begingroup$ Since it is outputting "1." and not "1" it is not outputting an integer but a float that happens to be the value "1". $\endgroup$
    – Pedro Henrique Monforte
    Commented Jun 28, 2019 at 13:17
  • $\begingroup$ @Pedro I saw that, but why is it 1 instead of 1.5? $\endgroup$
    – ps0604
    Commented Jun 28, 2019 at 13:51
  • $\begingroup$ I don't understand why you're expecting 1.5? The tree presumably perfectly fits on the training data, and so the only outputs that are possible are those in the training y. $\endgroup$
    – Ben Reiniger ♦
    Commented Jun 28, 2019 at 14:52

3 Answers 3

Reset to default
2
+50
$\begingroup$

This is the visualization for the inducted tree from your data:

Regressor

As you can see, it can never predict [[1.5, 1.5]]. Your data is just too pure, the inducted tree will fit it perfectly, having all leaf nodes with only one sample. If a leaf node has more than one sample in it, then the predicted value is the mean of the y values for those samples.

Furthermore, trees are not the best model to induct the identity function ($y = f(x) = x$). Trees work by creating rules and dividing the feature space orthogonally (i.e. only vertical or horizontal lines in the feature space).

In other words, your model didn't learn that f(1.5, 1.5) = (1.5, 1.5) because a) there's not enough data, b) trees are not good at doing that anyway.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Why did the DecisionTreeRegressor build the nodes with 1.5 and 2.5? $\endgroup$
    – ps0604
    Commented Jul 4, 2019 at 19:36
  • $\begingroup$ @ps0604 the first thing the tree-building algorithm does is checking splits. It could split [[1], [2, 3]] or [[1, 2], [3]]. Both provide the same entropy; in this case it used the first, but it can be considered random. For splitting the partitions [1] and [2, 3], it uses a rule with the mean between the highest of the lower node and the lower of the higher node, so (1 + 2) / 2. The same priciple applies to splitting the data [[2], [3]] in the intermediary node. $\endgroup$
    – Mephy
    Commented Jul 4, 2019 at 20:23
2
$\begingroup$

The data is being overfitting by the model, the model is memorizing the training data. Given the very few training samples, the decision tree can not learn the general pattern. If the model is given more training data with greater variation, it can learn the general pattern.

One approach is not to change the model but change the data. You can simulate a lot of random data and get the approximately correct results:

from sklearn.tree import DecisionTreeRegressor
import numpy as np

nums = np.random.random(size=100_000)*3
X = list(zip(nums, nums))
y = list(zip(nums, nums))
regr = DecisionTreeRegressor()
regr.fit(X, y)
X_test = [ [1.5, 1.5] ]
y_result = regr.predict(X_test)
assert np.isclose(y_result, X_test, atol=.001)
Improve this answer
$\endgroup$
0
$\begingroup$

I afraid that Decision Tree does not suitable. I recommend read about linear regression.

https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.