0
$\begingroup$

I have a dataframe with different dtypes like int, float, object, datatime etc. I am performing data cleaning, to list or find duplicate column names in the dataframe. Duplicate criteria as below:

  1. same column names or
  2. columns having same data values

I tried using transpose approach df.T.duplicated() to list duplicate column names, But seems slow for big dataframe.

I come to know we can use pivot or pivot_table or corr to list duplicate column names.

Can someone explain how to use and interpret it Or Is there any other way to do it?

$\endgroup$

1 Answer 1

1
$\begingroup$

To list duplicate columns by name in a Pandas DataFrame, you can call the duplicated method on the .columns property:

df.columns.duplicated()

This should be rather fast, unless you have an enormous number of columns in your data.

Finding duplicate columns by values is a bit tricky, and definitely slower. I wouldn't use any of pivot, pivot_table or corr, as they are slow as well for large data sets. Of the three, corr would be the most straightforward to detect duplicate columns - two identical columns would have a correlation of one.

I've found this StackOverflow question to contain some ideas, and I think that this answer is what you are looking for.

$\endgroup$
1
  • $\begingroup$ Hi Stefan, Thanks for detail explanation. $\endgroup$
    – winter
    May 21, 2023 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.