13
$\begingroup$

Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?

$\endgroup$
9
$\begingroup$

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how much data you have and whether any small difference between correlated variables tell you something about the outcome, after all.

The first two you can tell before you do any model, the final one not. So, it may be very reasonable to remove variables based on the combination of the first two considerations (i.e. even if the extra variables may in principle contain some useful information, you would not be able to tell given the strength of correlation and how much data you have) before you do any modelling/feature engineering. The final point can really only be assessed after doing some modelling.

$\endgroup$
3
$\begingroup$

Weird that nobody else mentioned interpretability.

If all you are concerned with is performance, then it makes no sense to remove two correlated variables, unless correlation=1 or -1, in which case one of the variables is redundant.

But if are concerned about interpretability then it might make sense to remove one of the variables, even if the correlation is mild. This is particularly true for linear models. One of the assumptions of the linear regression is lack of perfect multicollinearity in the predictors.

If A is correlated with B, then you cannot interpret the coefficients of neither A nor B. To see why, imagine the extreme case when A=B (perfect correlation). Then, the model y=100*A+50*B is the same as the model y=5*A+10*B or y=-2000*A+4000*B. There are multiple equilibra in the possible solutions to the least square minimzation problem therefore you cannot "trust" neither.

Similar things can happen with other models. For example, if A is very correlated with B, then if the decision tree chooses A double the times as B, then you cannot say that A is more important than B. If you retrain the model, the opposite could have happened.

$\endgroup$
2
$\begingroup$

You should consider checking VIF(Variance Inflation Factor). Try removing features with higher VIF. Generally, it is preferred that VIF is below 10.

$\endgroup$
1
1
$\begingroup$

It doesn't matter. But for efficiency before feature engineering.

$\endgroup$
1
$\begingroup$

Determine the covariance, and do your initial work with the highest set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.