1
$\begingroup$

Being a beginner in Data Analysis I ask you not to be much judgemental about this question.

I was tring to find if there is a standard way (function?) in Pandas module to identify redundancy of a column. For example, notably for categorical data types, if a column is a function of another it could be considered as redundant and thereof could be disregarded without any loss of prediction power. I would prefer not to implement any heavy machinery myself, but use common knowledge methods, therefore is my question.

I guess correlation is a good metric, but it has a different meaning and it is not a good fit for two categorical columns.

Any answers would be much appreciated, cheers.

$\endgroup$
2
$\begingroup$

Well, it always depends, for example, on what model you might be training (i.e. some are robust to multicollinearity). I am pretty sure you are aware, but to have it said as a rule of thumb it is always helpful if you know what you are looking for, rather than hoping naively one function or method would give all the answers.

Said that, there are good progress and in fact a powerful Python package pandas-profiling that can simply takes a pandas dataframe and returns lots of useful information in a blick of an eye (well if it is not super large dataset). And yes, for redundancy, correlation is a good start, and pandas-profiling highlights of highly correlated variables based on Spearman, Pearson and Kendall matrices. Take a look at this post for quick read about pandas-profiling.

For categorical columns, often things get trickier. In past, pandas-profiling did not offer anything. I just double checked, and interestingly they implemented Cramer’s V at this Closed Issue at their repo (see this nice post 'The Search for Categorical Correlation' if you want to learn about Cramer’s V4). I cannot confirm its functionality, you gotta test it on your own, but I believe it should be reliable as many contribute to this project. Unfortunately their documentations isn't thorough, but it seems it is there, see correlations.html page.

Update: Just figured that they have an advanced tutorial under Examples i.e. link to the Colab notebook Tutorial: report structure using Kaggle data (advanced), where the dataset is a combination of many data types including numerical and categorical, and they in fact show all the correlations elegantly, see a screenshot:

enter image description here

Good luck!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.