6
$\begingroup$

I have a conceptual question. My understanding is, that Random Forest can be applied even when features are (highly) correlated. This is because with bagging, the influence of few highly correlated features is moderated, since each feature only occurs in some of the trees which are finally used to build the overall model.

My question: With boosting, usually even smaller trees (basically "stunps") are used. Is it a problem to have many (highly) correlated features in a bagging approach?

$\endgroup$
9
$\begingroup$

Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables.

In general, tree based models are not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares. You can easily add a variable twice without numeric problem. Note however that most interpretability tools like split importance or partial dependence plots are affected by multicollinearity. So be careful with them in such cases.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ OKay, thanks. But with bagging, there is some kind of ensamble learning going on. My understanding is that bagging kind of does the trick when it comes to highly correlated independent variables... $\endgroup$ – Peter Mar 29 at 19:05
  • $\begingroup$ Bagging is about rows, not about columns. So if you e.g. bag a linear regression and there are two identical columns, every model will have collinearity issue. $\endgroup$ – Michael M Mar 29 at 21:40
  • $\begingroup$ I understood that the max_features parameter in sklearn RF refers to the number of column sampled per tree split. How does this affect the model when having high correlated features? $\endgroup$ – Carlos Mougan Apr 1 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.