Chose among highly correlated variables

Question

I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest.

Below you can find the correlation among all (independent and dependent) variables. We can quickly observe that only "GRE Score", "TOEFL Score" and "CGPA" considerably affect the "Chance of Admit" variable. So, it makes sense to eliminate all other variables from the predictive model.

Now among the "GRE Score", "TOEFL Score" and "CGPA" variables, we can see that they are all highly corellated (this also makes sense in real life as you always expect a good student to get good grades in these tests). I cannot decide which variables to keep for my final model. Could I keep all of them ? or how do I decide which one to exclude ?

Any help is appreciated.

Answer 1

I agree with @peter's detailed answer on points #1 through #5 and would like to supplement that with some more details:

1. Perform a PCA or MFA of the correlated variables and check how many predictors from this step explain all the correlation. For example, highly correlated variables might cause the first component of PCA to explain 95% of the variances in the data. Then, you can simply use this first component in the model.

2. Random forests can also be used for feature selection by looking at the feature importances of the variable. However, correlated variables can cause misleading feature importances. You can use permutation feature importance (https://scikit-learn.org/stable/modules/permutation_importance.html)

3. You might be overfitting when using all the features. Therefore, it is essential to use a validation set to check for overfitting.

4. You can also drop useless features by adding a feature with random values and dropping any feature that has lower feature importance than that feature.

For this topic, I highly recommend the book: Feature Engineering and Selection: A Practical Approach for Predictive Models; Open-sourced here: http://www.feat.engineering/ And also available on Amazon.

Answer 2

A few hints/ideas regarding your task:

1. I would not kick out other explanatory variables $x$ too early. When I look at your correlation chart, I suspect that all variables (but serial no) have some impact on $y$. If you kick out these variables, you may throw away important information.
2. In case some $x$ are highly correlated, you may face the problem of multicollinearity. This can be a real problem with some methods, e.g. linear regression and related approaches. Other methods, such as Random Forest are not so prone to overfitting with highly correlated features, since the algorithm only takes into account few features in each forest, so that highly influencial or correlated features do not play a big role in each single tree.
3. I suggest you check correlation first and possibly also the variance inflation factor from linear regression. This should give you a good idea on if the features are too strongly correlated to be considered jointly, e.g. in linear regression. Linear regression could also be a good baseline model here.
4. In order to perform feature selection, you could use Lasso/Ridge regression, which works like an "automated" way of selecting features based on importance for prediction.
5. In order to improve the model fit, you could also check generalised additive models (GAM), which is a linear family, but allows you to add highly non-linear representations of $x$ to the model without much need for tuning and without long training times.

In order to get a good start with ML, I suggest you have a look at "Introduction to Statistical Learning". There are R and Python labs for the book. This will give you a very good start and a sound understanding of the most important problems regarding ML.

For future reading (if you are interested): If you would work on a causal model (for statistical inference), your model would suffer from endogeneity. In this case you would need to take a different approach to solve the problem (instrumental variable or IV estimation). But since you work on a predictive model, you have no problems here. Wooldridge ("Introductory Econometrics") is a good starting point for anyone who is interested in causal modeling.