I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why.
import pandas as pd data = pd.read_csv("data.csv") print(len(data.columns)) print(len(data.corr().columns))
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Pearson's correlation is the default correlation used with Pandas corr method.
Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It makes no sense to say if categorical_var1 is increased by one , categorical_var2 also increases by X ( X's value depends on the correlation between the 2 variables ).
That's why you only see numerical variables! There are other statistical tests you can apply to categorical variables to better understand them.
Note : some columns may appear as numerical at first glance, but a string may be there due to an input mistake, or simply when the formatting of the file was done, that column type was set to 'Object'. Make sure to test the values in your supposedly numerical columns and apply astype to set them back to int or float