0
$\begingroup$

While I am aware that tree-based algorithms (e.g., DT, RF, XGBoost) are 'immune' to multi-collinearity, how do they handle linearly combined features? For example, is there is any additional value or harm in including the three feature: a, b and a+b in the model?

$\endgroup$
2
$\begingroup$

If the sum of the two feature makes sense on the domain semantically, it might be a good idea.

But while trees can handle redundant features pretty well, increasing the number of features without adding any extra "value" or "information" can lead to lower performance in certain situations. For example, if there is no added value and you aggressively sample or restrict the number of features in each tree the two related feature will take the place of a useful features when they are both selected.

If you think the sum is a better feature than the partitions you might consider addig the sum only, or the sum and one of the components.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.