# Chose among highly correlated variables

I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest.

Below you can find the correlation among all (independent and dependent) variables. We can quickly observe that only "GRE Score", "TOEFL Score" and "CGPA" considerably affect the "Chance of Admit" variable. So, it makes sense to eliminate all other variables from the predictive model.

Now among the "GRE Score", "TOEFL Score" and "CGPA" variables, we can see that they are all highly corellated (this also makes sense in real life as you always expect a good student to get good grades in these tests). I cannot decide which variables to keep for my final model. Could I keep all of them ? or how do I decide which one to exclude ?

Any help is appreciated.

• What kind of predictive model are you building? Some models are sensitive to "double counting" highly correlated variables, but others are implicitly feature-selective and will only keep variables that add new information, and some are latent variable models that can summarize correlated variables into one new feature. Correlated input variables may or may not be a problem for your downstream task, depending on what you plan to do. – Nuclear Wang Jan 3 at 19:30
• So far I have tried Recurrent neural networks, multilinear regression and k-NN regressor.. I will also try SVR and decision trees.. However, I am still new to machine learning and I do not know much about how each specific model treats highly correlated variables.. Could you give me more info ? – batman Jan 3 at 23:11

I agree with @peter's detailed answer on points #1 through #5 and would like to supplement that with some more details:

1. Perform a PCA or MFA of the correlated variables and check how many predictors from this step explain all the correlation. For example, highly correlated variables might cause the first component of PCA to explain 95% of the variances in the data. Then, you can simply use this first component in the model.

2. Random forests can also be used for feature selection by looking at the feature importances of the variable. However, correlated variables can cause misleading feature importances. You can use permutation feature importance (https://scikit-learn.org/stable/modules/permutation_importance.html)

3. You might be overfitting when using all the features. Therefore, it is essential to use a validation set to check for overfitting.

4. You can also drop useless features by adding a feature with random values and dropping any feature that has lower feature importance than that feature.

For this topic, I highly recommend the book: Feature Engineering and Selection: A Practical Approach for Predictive Models; Open-sourced here: http://www.feat.engineering/ And also available on Amazon.

1. I would not kick out other explanatory variables $$x$$ too early. When I look at your correlation chart, I suspect that all variables (but serial no) have some impact on $$y$$. If you kick out these variables, you may throw away important information.
2. In case some $$x$$ are highly correlated, you may face the problem of multicollinearity. This can be a real problem with some methods, e.g. linear regression and related approaches. Other methods, such as Random Forest are not so prone to overfitting with highly correlated features, since the algorithm only takes into account few features in each forest, so that highly influencial or correlated features do not play a big role in each single tree.
5. In order to improve the model fit, you could also check generalised additive models (GAM), which is a linear family, but allows you to add highly non-linear representations of $$x$$ to the model without much need for tuning and without long training times.