How to identify the appropriate predictors in regression model

Question

I have a big data and want to perform some of prediction models(such as Regression,Decision Tree,..) on it in R but I have a problem. As there are too many variables, Is there any way to identify which variables should i use as predictors in the model to predict a specific variable? Any help would be greatly appreciated.

Answer 1

I assume by too many variables you mean the case where number of variables is greater than the number of observations ($p>n$) or close to that. You will need to either reduce the number of variables before feeding them in, use some kind of automatic feature selection methods as @M.Mashayekhi has pointed out or utilize learning models that are robust to high feature dimension. Keep in mind that feature selection/creation, in my opinion, is the most important part of model building. There is no point in using a complex machine learning algorithm if all your input variables are useless.

It is always a good idea to first reduce the number of input variables by business intuition or context. By that I mean answering the questions: "Do I really need all of these variables to be in the model?", "What variables just wouldn't make sense to include in the model as a predictor?" "Are there variables that has to be in the model even if it is not the strongest predictor, simply for business reasons? If these questions are not easy to answer, or you have already reduced the number of variables but are still in the case of too many variables, then you can try the following.

Principal Component Analysis (PCA)

One suggestion would be to use PCA to reduce your feature space then run learning algorithms like regression, decision trees on the reduced feature space. You can do that in R using pca. Pick the first however many principal components where the next PC has a decline in marginal variance explained (Since each addition principal component always increases variance explained).

Regression with Lasso ($\mathcal{L1}$) Regularization

An alternative would be to let the model do the feature selection for you. A good starter would be a regression with the lasso ($\mathcal{L1}$) penalty, this shrinks the estimated coefficients toward zero. It does that by shrinking large effect coefficients more slowly toward zero than small effect ones. This effectively removes small effect coefficients from the model by setting them equal to zero, hence, auto feature selection. The challenge is to pick a good shrinkage parameter $\lambda$, which governs how much you are shrinking the coefficients. You can do that with cross validation. glmnet in R let's you run lasso regressions, and glmnet.cv let's you pick $\lambda$ using CV. Also note that by shrinking coefficients, lasso regression reduces prediction errors by balancing between variance reduction and bias increase. You will have to decide which is more important, inference or making good predictions, since unlike OLS regression, estimates by lasso regression are no longer unbiased.

Random Forest

Random Forest is an ensemble tree learning algorithm (implemented in randomForest in R) which is robust to many useless variables. Particularly, it works well in the case of $p>n$. RF is very popular because of its robustness and simple tuning. The downside is this is what's called a Black Box algorithm. Meaning that there is no good way to interpret the relationship between the input and dependent variables. You can however use something like the "variable importance plot" to give you a rough idea of what variables were most useful during the splits on average. This is fine if you are only concerned about making good predictions.

Other techniques

There are certainly other techniques that goes well with high dimensional data like Support Vector Machines and Boosted Regression Trees (Difference between RF and Boosted Trees), but they might usually require a lot more tuning than the ones mentioned above.

Answer 2

Check the correlation of the variables or features, the ones with the least correlation can be discarded and the ones with highest must be considered to perform prediction. However, least and highest is subjective to your requirement