3
$\begingroup$

I'm running a linear regression model as baseline for a specific estimation problem. Based on the resulting R-squared, regressor coefficients and their respective p-values, I can conclude that specific independent variables can be removed from the model.

  • What is the induced risk of removing these variables from the feature set?
  • Can other models—that are better at modelling nonlinear relationships—suffer from this decision?
  • How can I be sure that I am not loosing valuable nonlinear information without running "nonlinear" regressors?
$\endgroup$
  • 4
    $\begingroup$ @KorkiBuziek This is a standard question of variable selection and there is a huge literature on it. High p-value and/or low variable importance does not imply lack of utility. stats.stackexchange.com/questions/215154/… $\endgroup$ – horaceT Mar 13 '18 at 16:58
2
$\begingroup$

Variable selection in linear regression is based on partial correlations, not zero order correlations. The partial correlation is the one resulting after a conditioning variable effect is removed from both X and Y.

While a variable may be non highly correlated with Y, it may be highly correlated with an X variable in such a way that the X variable after being partialled out of the conditioning effect becomes highly significant.

This effect is known as suppression among other names. HTH

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.