Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am at the feature selection phase of my project but I have my vectorised data. Is there a way to find highly correlated features and then remove them? After this I would then like to remove features that are not greatly important.
Im using tweets for my project and have done various pre-processing techniques. I lemmatised my data and then vectorised and store the vectorised data under 'X'. I now have to find the highly correlated features within X and then remove them.
Make a dataframe of your feature where each row is a sample and each columns is a variable. Then you can use the following line of code to plot the correlation heatmap. It is $2*2$ matrix where entry $(i,j)$ indicates the correlation value of $i$ and $j$ feature.
import seaborn as sns
import pandas as pd
corr = df.corr() #df is your feature vector converted to dataframe
sns.heatmap(corr)
