16
$\begingroup$

So, our data set this week has 14 attributes and each column has very different values. One column has values below 1 while another column has values that go from three to four whole digits.

We learned normalization last week and it seems like you're supposed to normalize data when they have very different values. For decision trees, is the case the same?

I'm not sure about this but would normalization affect the resulting decision tree from the same data set? It doesn't seem like it should but...

Improve this question
$\endgroup$

1 Answer 1

Reset to default
24
$\begingroup$

Most common types of decision trees you encounter are not affected by any monotonic transformation. So, as long as you preserve orde, the decision trees are the same (obviously by the same tree here I understand the same decision structure, not the same values for each test in each node of the tree).

The reason why it happens is because how usual impurity functions works. In order to find the best split it searches on each dimension (attribute) a split point which is basically an if clause which groups target values corresponding to instances which has test value less than split value, and on the right the values greater than equal. This happens for numerical attributes (which I think is your case because I do not know how to normalize a nominal attribute). Now you might note that the criteria is less than or greater than. Which means that the real information from the attributes in order to find the split (and the whole tree) is only the order of the values. Which means that as long as you transform your attributes in such a way that the original ordering is reserved, you will get the same tree.

Not all models are insensitive to such kind of transformation. For example linear regression models give the same results if you multiply an attribute with something different than zero. You will get different regression coefficients, but the predicted value will be the same. This is not the case when you take a log of that transformation. So for linear regression, for example, normalizing is useless since it will provide the same result.

However this is not the case with a penalized linear regression, like ridge regression. In penalized linear regressions a constraint is applied to coefficients. The idea is that the constraint is applied to the sum of a function of coefficients. Now if you inflate an attribute, the coefficient will be deflated, which means that in the end the penalization for that coefficient it will be artificially modified. In such kind of situation, you normalize attributes in order that each coefficient to be constraint 'fairly'.

Hope it helps

Improve this answer
$\endgroup$
2
  • $\begingroup$ The tree building would not change since the ordering would not change. What might change is the performance of the model (as indicated by this example). I think the only possible explanation for this is that the new (test, unseen data) might go into different splits in normalized compared to unormalized case. $\endgroup$
    – Antonios Sarikas
    Commented Dec 27, 2023 at 18:42
  • $\begingroup$ The example you provided states that the performance might change. All of that is due to numerical reasons, as you noted. This is favored also by np.float32 sklearn applies. In the example case the test data does not go exactly into the same leaf nodes. For other cases the decision tree can have a slightly different structure after numerical transformation. Same effect as changing the seed. This mean the performance is changed? If your fit depends on random seed or numerical computation than the large the variation the larger the trouble you are in. $\endgroup$
    – rapaio
    Commented Feb 28 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.