1
$\begingroup$

Let's say you have 6 variables.

A Random-Forest regression using the first 5 variables has an R^2 of 0.1. Another regression using just the 6th variable yields R^2 of 0.3. All of the first 5 variables are uncorrelated with variable 6 (<0.1 correlation in absolute value).

Why would a regression with all 6 variables have an R^2 of 0.31 or 0.29, i.e. adding first 5 variables to the 6th variable only negligibly increases performance by 0.01 or even decreases it?

Notice that all models are tuned via random search cross-validation tuning depth, number of features, splits and number of trees.

$\endgroup$
2
$\begingroup$

Even if your variables are linearly uncorrelated, they may still be non linearly correlated. Just using the correlation coefficient to say the data is unrelated is not enough for non linear regression.

Still, adding variables to a model may not improve the end result if the variables cannot explain anything new from the target data. This seems to be such a case.

$\endgroup$
  • $\begingroup$ Agreed, already trying to test for a nonlinear relationship. Do you have a favorite test? $\endgroup$ – John Doe Dec 19 '18 at 18:04
  • $\begingroup$ There are many other tests possible, Spearman rank test would be the first one I would try. $\endgroup$ – Matthieu Brucher Dec 19 '18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.