237 votes
Accepted

Difference between isna() and isnull() in pandas

Pandas isna() vs isnull(). I'm assuming you are referring to pandas.DataFrame.isna() vs <...
Djib2011's user avatar
  • 7,978
42 votes
Accepted

How do I compare columns in different data frames?

If you want to check equal values on a certain column, let's say Name, you can merge both DataFrames to a new one: ...
Tarek's user avatar
  • 724
29 votes

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

In case this can help anyone else. Here is a solution that is more computationally efficient. TL;DR version If each row already has a unique index, then do this: ...
Bon Ryu's user avatar
  • 391
28 votes

How do I compare columns in different data frames?

You can double check the exact number of common and different positions between two df by using isin and value_counts(). Like ...
QbS's user avatar
  • 381
22 votes

How do I compare columns in different data frames?

df1.where(df1.values==df2.values).notna() True entries show common elements. This also reveals the position of the common ...
pcko1's user avatar
  • 3,940
21 votes
Accepted

dataframe.columns.difference() use

The function dataframe.columns.difference() gives you complement of the values that you provide as argument. It can be used to create a new dataframe from an ...
bkshi's user avatar
  • 2,235
21 votes
Accepted

Delete/Drop only the rows which has all values as NaN in pandas

The complete command is this: df.dropna(axis = 0, how = 'all', inplace = True) you must add inplace = True argument, if you ...
Leevo's user avatar
  • 6,225
20 votes

How to plot two columns of single DataFrame on Y axis

Feeding your column names into the y values argument as a list works for me like so: ...
user2333453's user avatar
20 votes
Accepted

How to sum values grouped by two columns in pandas

pivot_table was made for this: df.pivot_table(index='Date',columns='Groups',aggfunc=sum) results in ...
Josh D.'s user avatar
  • 318
17 votes
Accepted

How to remove rows from a dataframe that are identical to another dataframe?

You can try this: ...
Mohit Motwani's user avatar
16 votes
Accepted

How to Write Multiple Data Frames in an Excel Sheet

an example to write in same sheet: ...
Frenchy's user avatar
  • 301
16 votes

Pandas merge column duplicate and sum value

You may use df2 = df.groupby(['address']).sum() or df2 = df.groupby(['address']).agg('sum') If there are columns other than <...
Esmailian's user avatar
  • 9,312
15 votes

How do I compare columns in different data frames?

Comparing values in two different columns Using set, get unique values in each column. The intersection of these two sets will provide the unique values in both the columns. Example: ...
aathiraks's user avatar
  • 704
14 votes
Accepted

One hot encoding alternatives for large categorical values

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power. I have also ...
tom's user avatar
  • 2,248
14 votes

shifting the last column in the dataframe to the first place

cols = list(df.columns) cols = [cols[-1]] + cols[:-1] df = df[cols]
Fred's user avatar
  • 403
12 votes
Accepted

How to find the count of consecutive same string values in a pandas dataframe?

Break col1 into sub-groups of consecutive strings. Extract first and last entry per sub-group. Something like this: ...
oW_'s user avatar
  • 6,347
11 votes
Accepted

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

You can do this. But I doubt the efficiency. >> import pandas as pd >> df = pd.DataFrame({'a':[1,1,3,3],'b':[4,5,6,3], 'c':[1,2,3,5]}) >> df a b c 0 1 4 1 1 1 5 2 2 3 6 3 3 3 3 5 >> ...
Kiritee Gak's user avatar
  • 1,799
11 votes

How duplicated items can be deleted from dataframe in pandas

The best way would be to use drop_duplicates(). If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked. ...
Russell's user avatar
  • 111
11 votes
Accepted

Find the consecutive zeros in a DataFrame and do a conditional replacement

Consider the following approach: ...
MaxU - stand with Ukraine's user avatar
11 votes
Accepted

How to rename columns that have the same name?

You can use this: df.columns = ['Goods_1', 'Durable goods','Services','Exports', 'Goods_2', 'Services', 'Imports', 'Goods_3', 'Services'] or if you have too many ...
Mohit Motwani's user avatar
9 votes

How to sum values grouped by two columns in pandas

Pandas black magic: ...
tuomastik's user avatar
  • 1,183
9 votes

How to remove rows from a dataframe that are identical to another dataframe?

Simpler to use isin() with dropna() https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isin.html <...
Idodo's user avatar
  • 191
8 votes

Using pandas, check a column for matching text and update new column if TRUE

You simply need to do: df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern) df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})
Suresh Kasipandy's user avatar
8 votes
Accepted

Pandas apply return: Must have equal len keys and value when setting with an iterable

I found the issue, I need to return a pd.Series() ...
ParisNakitaKejser's user avatar
8 votes
Accepted

Mapping column values of one DataFrame to another DataFrame using a key with different header names

You can convert df2 to a dictionary and use that to replace the values in df1 ...
Stephen Witkowski's user avatar
7 votes
Accepted

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

Based on the documentation 0.22 and 0.24.1, the flavor does not exist in the argument list of the to_sql method. You're probably running the ...
Green Falcon's user avatar
  • 14.1k
7 votes
Accepted

Pandas merge column duplicate and sum value

In another case when you have a dataset with several duplicated columns and you wouldn't want to select them separately use: ...
Denisa's user avatar
  • 86
6 votes
Accepted

Replacing column values in pandas

As Emre already mentioned, you may use the groupby function. After that, you should apply reset_index to move the MultiIndex to the columns: ...
desiato's user avatar
  • 176
6 votes

Mapping column values of one DataFrame to another DataFrame using a key with different header names

df3 = pd.merge(df1,df2,left_on=['cat'+str(i)], right_on = ['cat_codes'], how = 'left') I would iterate this for cat1,cat2 and cat3. This does not replace the ...
Danny's user avatar
  • 1,148
6 votes
Accepted

Export pandas dataframe to a nested dictionary from multiple columns

Using dict comprehension with nested groupby: d = {k: f.groupby('subgroup')['selectedCol'].apply(list).to_dict() for k, f in df.groupby('maingroup')} Output:...
acb's user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible