I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest.

Below you can find the correlation among all (independent and dependent) variables. We can quickly observe that only "GRE Score", "TOEFL Score" and "CGPA" considerably affect the "Chance of Admit" variable. So, it makes sense to eliminate all other variables from the predictive model.

Now among the "GRE Score", "TOEFL Score" and "CGPA" variables, we can see that they are all highly corellated (this also makes sense in real life as you always expect a good student to get good grades in these tests). I cannot decide which variables to keep for my final model. Could I keep all of them ? or how do I decide which one to exclude ?

Any help is appreciated.

enter image description here

  • 3
    $\begingroup$ What kind of predictive model are you building? Some models are sensitive to "double counting" highly correlated variables, but others are implicitly feature-selective and will only keep variables that add new information, and some are latent variable models that can summarize correlated variables into one new feature. Correlated input variables may or may not be a problem for your downstream task, depending on what you plan to do. $\endgroup$ Commented Jan 3, 2020 at 19:30
  • $\begingroup$ So far I have tried Recurrent neural networks, multilinear regression and k-NN regressor.. I will also try SVR and decision trees.. However, I am still new to machine learning and I do not know much about how each specific model treats highly correlated variables.. Could you give me more info ? $\endgroup$
    – batman
    Commented Jan 3, 2020 at 23:11

2 Answers 2


I agree with @peter's detailed answer on points #1 through #5 and would like to supplement that with some more details:

  1. Perform a PCA or MFA of the correlated variables and check how many predictors from this step explain all the correlation. For example, highly correlated variables might cause the first component of PCA to explain 95% of the variances in the data. Then, you can simply use this first component in the model.

  2. Random forests can also be used for feature selection by looking at the feature importances of the variable. However, correlated variables can cause misleading feature importances. You can use permutation feature importance (https://scikit-learn.org/stable/modules/permutation_importance.html)

  3. You might be overfitting when using all the features. Therefore, it is essential to use a validation set to check for overfitting.

  4. You can also drop useless features by adding a feature with random values and dropping any feature that has lower feature importance than that feature.

For this topic, I highly recommend the book: Feature Engineering and Selection: A Practical Approach for Predictive Models; Open-sourced here: http://www.feat.engineering/ And also available on Amazon.


A few hints/ideas regarding your task:

  1. I would not kick out other explanatory variables $x$ too early. When I look at your correlation chart, I suspect that all variables (but serial no) have some impact on $y$. If you kick out these variables, you may throw away important information.
  2. In case some $x$ are highly correlated, you may face the problem of multicollinearity. This can be a real problem with some methods, e.g. linear regression and related approaches. Other methods, such as Random Forest are not so prone to overfitting with highly correlated features, since the algorithm only takes into account few features in each forest, so that highly influencial or correlated features do not play a big role in each single tree.
  3. I suggest you check correlation first and possibly also the variance inflation factor from linear regression. This should give you a good idea on if the features are too strongly correlated to be considered jointly, e.g. in linear regression. Linear regression could also be a good baseline model here.
  4. In order to perform feature selection, you could use Lasso/Ridge regression, which works like an "automated" way of selecting features based on importance for prediction.
  5. In order to improve the model fit, you could also check generalised additive models (GAM), which is a linear family, but allows you to add highly non-linear representations of $x$ to the model without much need for tuning and without long training times.

In order to get a good start with ML, I suggest you have a look at "Introduction to Statistical Learning". There are R and Python labs for the book. This will give you a very good start and a sound understanding of the most important problems regarding ML.

For future reading (if you are interested): If you would work on a causal model (for statistical inference), your model would suffer from endogeneity. In this case you would need to take a different approach to solve the problem (instrumental variable or IV estimation). But since you work on a predictive model, you have no problems here. Wooldridge ("Introductory Econometrics") is a good starting point for anyone who is interested in causal modeling.

  • 1
    $\begingroup$ Thanks for the detailed answer.. I will check everyhting you mention above.. Regarding your first hint you are right.. I built Rnn regressor, Linear regressor and kNN regressor using all features at first, and then using only the top 3 features.. I turns out that the MSE is always smaller when I use all the features.. Also, in botth cases the linear regressor performs better than the other two models $\endgroup$
    – batman
    Commented Jan 4, 2020 at 13:02
  • $\begingroup$ Cool, check GAM, they are awesome $\endgroup$
    – Peter
    Commented Jan 4, 2020 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.