I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21).

Now visualising such large matrices becomes a very messy task and you end up hurting your eyes. So what I have done is set a threshold and to slice out those rows that have greater than this value (say 0.60). However, I'm getting a matrix that has now several NaNs. When I try to drop these null values, the matrix loses all data and what I'm left is a 0*0 matrix.

corr_matrix = data.corr()

threshold = 0.60

high_corr = corr_matrix.loc[corr_matrix >= 0.60]


Empty DataFrame
Columns = []

Visualising the matrix with nans is a good idea but it also results in empty squares. I'm looking for a way where only those rows that have values >= threshold are retained, with no nans. That would make a much smaller matrix which is much less messier when plotted in matplotlib. However I haven't been able to code it that way; can anyone suggest some strategies to deal with such large matrices?

  • 1
    $\begingroup$ Have you considered colorizing the matrix in the style of a heatmap, rather than poring over the numerical values? Your current approach won't help much if you find that there's high correlation among all features and can't drop any. See stackoverflow.com/questions/12286607/… $\endgroup$ – Nuclear Hoagie Sep 25 '20 at 18:04

Try this (note that I didn't add error checking, so it'll crash if your threshold removes all values). Also, I made it an absolute high pass rather than a normal high pass because I assume you'd be interested in strong negative correlation as well? If you're not, just remove the abs() in the filter function.

from numpy.random      import randn
from pandas            import DataFrame
from seaborn           import heatmap
from matplotlib.pyplot import show
from itertools         import combinations

def absHighPass(df, absThresh):
    passed = set()
    for (r,c) in combinations(df.columns, 2):
        if (abs(df.loc[r,c]) >= absThresh):
    passed = sorted(passed)
    return df.loc[passed,passed]

labels = [chr(x) for x in range(65,91)]
corrDf = DataFrame(randn(26,26), index=labels, columns=labels).corr()


This is the filtered heatmap:


And this is another run but with the unfiltered heatmap:


  • $\begingroup$ Impressive. I'll certainly try this out. $\endgroup$ – Shiv_90 Sep 26 '20 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.