How to select variables based on the mean correlation in a correlation matrix?

Question

I have a set of independent variables and I am calculating the correlation matrix between them using the Pearson Correlation Coefficient in Python. A part of the matrix looks like this:

From this matrix, suppose I want to find out the strongly correlated components between the variable NoOfDoors and the rest(Symboling...Compression Ratio). The process I have adopted is that I have taken the mean of that column(which is calculated as 0.039604) and based on that, I have only considered those values greater than 0.039604.

Based on that, the following variables have been selected as strongly correlated:

(Make, Aspiration, Wheel Base, Length, Width, Height, Curb Weight, Engine Type, Bore, Compression Ratio)

I want to ask, is this selection correct? If yes then is there an efficient way to do this? And if no, what is the correct way? Since I am new to this field, a well explained article would be appreciated. Thanks!

Answer 1

There is no universal definition of "strongly correlated" and thus there is no "correct" answer to your question. One approach would be to perform a significance test to determine whether there is strong evidence that the correlations are nonzero. The mathematical details of the test are here, and the relevant Scipy API documentation is here.

However, "probably nonzero" may not be what you're looking for when you say "strongly correlated." If that's the case, you might want to consider why you're interested in finding the strongly correlated variables in the first place. Are you interested in identifying the strongly correlated features for a predictive problem? If so, you'll likely be better off skipping this exercise and starting to experiment directly with learning algorithms. Are you interested in determining what predictor variables cause changes in a certain response variable? In this case, you'll probably want to look into some kind of statistical regression model (probably starting with linear regression) and be very careful about all of the assumptions involved when drawing causal conclusions.

Answer 2

Great answer by @StubbornTurtle.

To give a few more details see this answer. This talks about suppressor, mediators, confounders, and moderators. Suppressors are weakly correlated with a predictor but the suppressor's inclusion in the model makes the model stronger since it helps predict the target where another predictor is weak or as I was taught explain the variance.

All data sets do not have suppressors. However, by only including "highly" correlated, you may lose these variables and make your model weaker.

Second, we often want some predictors that are uncorrelated with each other if the predictors make the model "better", where better is defined by the problem the model is trying to solve. Uncorrelated predictors may be latching onto different signals and noises in the model. If the predictors are all highly correlated, they may be all explaining the same signal and containing the same noise.