I am at the feature selection phase of my project but I have my vectorised data. Is there a way to find highly correlated features and then remove them? After this I would then like to remove features that are not greatly important.

Im using tweets for my project and have done various pre-processing techniques. I lemmatised my data and then vectorised and store the vectorised data under 'X'. I now have to find the highly correlated features within X and then remove them.

  • 1
    $\begingroup$ Provide an example … $\endgroup$
    – n1tk
    Aug 12 '21 at 19:20
  • $\begingroup$ I think I literally just answered this question in a different topic datascience.stackexchange.com/a/100079/74387. Does it help? The point is that some features could be correlated with several others, and then pairwise correlation could be low despite all the features being strongly correlated. $\endgroup$
    – Cryo
    Aug 14 '21 at 23:03

Make a dataframe of your feature where each row is a sample and each columns is a variable. Then you can use the following line of code to plot the correlation heatmap. It is $2*2$ matrix where entry $(i,j)$ indicates the correlation value of $i$ and $j$ feature.

import seaborn as sns
import pandas as pd

corr = df.corr()  #df is your feature vector converted to dataframe

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.