220 votes
Accepted

Difference between isna() and isnull() in pandas

Pandas isna() vs isnull(). I'm assuming you are referring to pandas.DataFrame.isna() vs <...
user avatar
  • 7,418
38 votes
Accepted

How do I compare columns in different data frames?

If you want to check equal values on a certain column, let's say Name, you can merge both DataFrames to a new one: ...
user avatar
  • 684
29 votes

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

In case this can help anyone else. Here is a solution that is more computationally efficient. TL;DR version If each row already has a unique index, then do this: ...
user avatar
  • 391
21 votes

How do I compare columns in different data frames?

df1.where(df1.values==df2.values).notna() True entries show common elements. This also reveals the position of the common ...
user avatar
  • 3,720
20 votes
Accepted

How to sum values grouped by two columns in pandas

pivot_table was made for this: df.pivot_table(index='Date',columns='Groups',aggfunc=sum) results in ...
user avatar
  • 318
20 votes

How do I compare columns in different data frames?

You can double check the exact number of common and different positions between two df by using isin and value_counts(). Like ...
user avatar
  • 301
18 votes

How to plot two columns of single DataFrame on Y axis

Feeding your column names into the y values argument as a list works for me like so: ...
user avatar
17 votes
Accepted

dataframe.columns.difference() use

The function dataframe.columns.difference() gives you complement of the values that you provide as argument. It can be used to create a new dataframe from an ...
user avatar
  • 2,135
16 votes

Pandas merge column duplicate and sum value

You may use df2 = df.groupby(['address']).sum() or df2 = df.groupby(['address']).agg('sum') If there are columns other than <...
user avatar
  • 8,459
15 votes

How do I compare columns in different data frames?

Comparing values in two different columns Using set, get unique values in each column. The intersection of these two sets will provide the unique values in both the columns. Example: ...
user avatar
  • 654
14 votes
Accepted

How to remove rows from a data frame that are identical to other df?

You can try this: ...
user avatar
14 votes
Accepted

How to Write Multiple Data Frames in an Excel Sheet

an example to write in same sheet: ...
user avatar
  • 281
13 votes
Accepted

One hot encoding alternatives for large categorical values

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power. I have also ...
user avatar
  • 2,108
13 votes

shifting the last column in the dataframe to the first place

cols = list(df.columns) cols = [cols[-1]] + cols[:-1] df = df[cols]
user avatar
  • 393
11 votes
Accepted

after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

You can do this. But I doubt the efficiency. >> import pandas as pd >> df = pd.DataFrame({'a':[1,1,3,3],'b':[4,5,6,3], 'c':[1,2,3,5]}) >> df a b c 0 1 4 1 1 1 5 2 2 3 6 3 3 3 3 5 >> ...
user avatar
  • 1,759
11 votes

How duplicated items can be deleted from dataframe in pandas

The best way would be to use drop_duplicates(). If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked. ...
user avatar
  • 111
11 votes
Accepted

How to rename columns that have the same name?

You can use this: df.columns = ['Goods_1', 'Durable goods','Services','Exports', 'Goods_2', 'Services', 'Imports', 'Goods_3', 'Services'] or if you have too many ...
user avatar
10 votes
Accepted

Find the consecutive zeros in a DataFrame and do a conditional replacement

Consider the following approach: ...
user avatar
10 votes
Accepted

Delete/Drop only the rows which has all values as NaN in pandas

The complete command is this: df.dropna(axis = 0, how = 'all', inplace = True) you must add inplace = True argument, if you ...
user avatar
  • 5,667
9 votes

How to sum values grouped by two columns in pandas

Pandas black magic: ...
user avatar
  • 1,134
9 votes
Accepted

How to find the count of consecutive same string values in a pandas dataframe?

Break col1 into sub-groups of consecutive strings. Extract first and last entry per sub-group. Something like this: ...
user avatar
  • 5,950
8 votes

Using pandas, check a column for matching text and update new column if TRUE

You simply need to do: df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern) df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})
user avatar
8 votes
Accepted

Pandas apply return: Must have equal len keys and value when setting with an iterable

I found the issue, I need to return a pd.Series() ...
user avatar
8 votes
Accepted

Mapping column values of one DataFrame to another DataFrame using a key with different header names

You can convert df2 to a dictionary and use that to replace the values in df1 ...
user avatar
7 votes
Accepted

Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

Based on the documentation 0.22 and 0.24.1, the flavor does not exist in the argument list of the to_sql method. You're probably running the ...
user avatar
  • 13.3k
7 votes
Accepted

Pandas merge column duplicate and sum value

In another case when you have a dataset with several duplicated columns and you wouldn't want to select them separately use: ...
user avatar
  • 86
6 votes
Accepted

Replacing column values in pandas

As Emre already mentioned, you may use the groupby function. After that, you should apply reset_index to move the MultiIndex to the columns: ...
user avatar
  • 176
6 votes

How to remove rows from a data frame that are identical to other df?

Simpler to use isin() with dropna() https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isin.html <...
user avatar
  • 161
6 votes

Mapping column values of one DataFrame to another DataFrame using a key with different header names

df3 = pd.merge(df1,df2,left_on=['cat'+str(i)], right_on = ['cat_codes'], how = 'left') I would iterate this for cat1,cat2 and cat3. This does not replace the ...
user avatar
  • 1,108
6 votes
Accepted

Export pandas dataframe to a nested dictionary from multiple columns

Using dict comprehension with nested groupby: d = {k: f.groupby('subgroup')['selectedCol'].apply(list).to_dict() for k, f in df.groupby('maingroup')} Output:...
user avatar
  • 101

Only top scored, non community-wiki answers of a minimum length are eligible