17
$\begingroup$

I have two data frames df1 and df2. For my analysis, I need to remove rows from df1 that have identical column values (Email) in df2?

>>df1
   First  Last  Email
0 Adam   Smith  [email protected]
1 John   Brown  [email protected]
2 Joe    Max    [email protected]
3 Will   Bill   [email protected]


>>df2
  First  Last   Email
0 Adam   Smith  [email protected]
1 John   Brown  [email protected]
$\endgroup$
0

2 Answers 2

17
$\begingroup$

You can try this:

cond = df1['Email'].isin(df2['Email'])
df1.drop(df1[cond].index, inplace = True)

>>df1
    First   Last    Email
2   Joe     Max     [email protected]
3   Will    Bill    [email protected]
$\endgroup$
1
  • $\begingroup$ This method does not work if you don't have a unique column (email in this case). In this case you can join two or more columns into one with df1[['First', 'Last']].agg('-'.join, axis=1) if they are strings. $\endgroup$
    – alvitawa
    Apr 3, 2022 at 12:47
9
$\begingroup$

Simpler to use isin() with dropna()

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isin.html

df1[~df1.isin(df2)].dropna()
$\endgroup$
3
  • $\begingroup$ Quick question, what does the tilde stands for? $\endgroup$ Apr 23, 2021 at 18:23
  • $\begingroup$ It's a negation. Will turn all True to False and vice versa. $\endgroup$
    – Idodo
    Apr 26, 2021 at 8:50
  • 1
    $\begingroup$ This is wrong. It compares the values one at a time, a row can have mixed cases. Even when a row has all true, that doesn't mean that same row exists in the other dataframe, it means the values of this row exist in the columns of the other dataframe but in multiple rows. $\endgroup$
    – anishtain4
    May 13, 2022 at 17:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.