Often, after training a linear regression model on data, some variables/features would have high P-value, which means they are not statistically significant. Although there are automated methods like variable selection, such as Step-wise, LASSO, etc. I would like to understand the best practice for manually selecting variables.

One thing I can think of is to check for collinearity and perhaps discard or combine variables that shows high correlation. One can also compare models using the F test, or perhaps use domain knowledge. Aside from that, what other best practices are there?

I feel simply discarding variables that shows high P-value is too simplistic. And there is no guarantee that the remaining variables will all have high P-value after you have discarded some variables.

  • $\begingroup$ Good models can have variables with high, “insignificant” p-values. You don’t need to get rid of them. What is the goal of analysis? $\endgroup$ – Dave May 29 '20 at 21:57
  • $\begingroup$ I think for prediction purposes, you don't need to get rid of them. But what about for inference? $\endgroup$ – Paul May 30 '20 at 1:20
  • $\begingroup$ Usually you have a hypothesis in mind when you run the regression, not just throw a bunch of variables at a problem and see what sticks. So what is your hypothesis? $\endgroup$ – Dave May 30 '20 at 1:25
  • $\begingroup$ Causal model or a model for prediction? $\endgroup$ – Peter May 30 '20 at 17:40

In general, there is no clear and easy way for deciding which features to include in a model.

That being said, there are different strategies you can use to process features in an efficient way:

  1. Domain knowledge

One of the most important aspects when determining important features is the knowledge of the specific domain related to your dataset. This might mean reading past research papers that have explored similar topics or asking key stakeholders to determine what they believe the most important factors are for predicting the target variable.

So, these methods depend on your knowledge from the domain.

  1. Wrapper methods

These methods determine the optimal subset of features using different combinations of features to train models and then calculating performance. Every subset is used to train models and then evaluated on a test set. Sp, these methods could end up being very computationally intensive, though they are highly effective in determining the optimal subset. It is not suggested to use these methods with large feature sets because they are computationally expensive.

For this case you can use Recursive Feature elimination (RFE)

  1. Filter methods

In these methods, feature selection methods carried out as a pre-processing step before even running a model.

They work by observing characteristics of how features are related to one another. Different metrics can be used to determine which features will get eliminated and which will remain. They also return a feature ranking that tells you how independent variables are ordered in relation to one another. They will remove the variables that are considered redundant.

One example of this method is "VarianceThreshold" from sklearn

  1. Embedded methods

These methods are included within the actual formulation of the machine learning algorithm. The most common type of embedded method is regularization such as lasso.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.