Pandas: To find duplicate columns

Question

I have a dataframe with different dtypes like int, float, object, datatime etc. I am performing data cleaning, to list or find duplicate column names in the dataframe. Duplicate criteria as below:

1. same column names or
2. columns having same data values

I tried using transpose approach df.T.duplicated() to list duplicate column names, But seems slow for big dataframe.

I come to know we can use pivot or pivot_table or corr to list duplicate column names.

Can someone explain how to use and interpret it Or Is there any other way to do it?

Answer 1

To list duplicate columns by name in a Pandas DataFrame, you can call the duplicated method on the .columns property:

df.columns.duplicated()

This should be rather fast, unless you have an enormous number of columns in your data.

Finding duplicate columns by values is a bit tricky, and definitely slower. I wouldn't use any of pivot, pivot_table or corr, as they are slow as well for large data sets. Of the three, corr would be the most straightforward to detect duplicate columns - two identical columns would have a correlation of one.

I've found this StackOverflow question to contain some ideas, and I think that this answer is what you are looking for.

Answer 2

If you can guess 2 columns are having same values, try:

df['column1'].equals(df['column2'])