220
votes
Accepted
Difference between isna() and isnull() in pandas
Pandas isna() vs isnull().
I'm assuming you are referring to pandas.DataFrame.isna() vs <...
38
votes
Accepted
How do I compare columns in different data frames?
If you want to check equal values on a certain column, let's say Name, you can merge both DataFrames to a new one:
...
29
votes
after grouping to minimum value in pandas, how to display the matching row result entirely along min() value
In case this can help anyone else. Here is a solution that is more computationally efficient.
TL;DR version
If each row already has a unique index, then do this:
...
21
votes
How do I compare columns in different data frames?
df1.where(df1.values==df2.values).notna()
True entries show common elements. This also reveals the position of the common ...
20
votes
Accepted
How to sum values grouped by two columns in pandas
pivot_table was made for this:
df.pivot_table(index='Date',columns='Groups',aggfunc=sum)
results in
...
20
votes
How do I compare columns in different data frames?
You can double check the exact number of common and different positions between two df by using isin and value_counts().
Like ...
18
votes
How to plot two columns of single DataFrame on Y axis
Feeding your column names into the y values argument as a list works for me like so:
...
17
votes
Accepted
dataframe.columns.difference() use
The function dataframe.columns.difference() gives you complement of the values that you provide as argument. It can be used to create a new dataframe from an ...
16
votes
Pandas merge column duplicate and sum value
You may use
df2 = df.groupby(['address']).sum()
or
df2 = df.groupby(['address']).agg('sum')
If there are columns other than <...
15
votes
How do I compare columns in different data frames?
Comparing values in two different columns
Using set, get unique values in each column. The intersection of these two sets will provide the unique values in both the columns.
Example:
...
14
votes
Accepted
14
votes
Accepted
13
votes
Accepted
One hot encoding alternatives for large categorical values
One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power.
I have also ...
13
votes
shifting the last column in the dataframe to the first place
cols = list(df.columns)
cols = [cols[-1]] + cols[:-1]
df = df[cols]
11
votes
Accepted
after grouping to minimum value in pandas, how to display the matching row result entirely along min() value
You can do this. But I doubt the efficiency.
>> import pandas as pd
>> df = pd.DataFrame({'a':[1,1,3,3],'b':[4,5,6,3], 'c':[1,2,3,5]})
>> df
a b c
0 1 4 1
1 1 5 2
2 3 6 3
3 3 3 5
>> ...
11
votes
How duplicated items can be deleted from dataframe in pandas
The best way would be to use drop_duplicates(). If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked.
...
11
votes
Accepted
How to rename columns that have the same name?
You can use this:
df.columns = ['Goods_1', 'Durable goods','Services','Exports', 'Goods_2', 'Services', 'Imports', 'Goods_3', 'Services']
or if you have too many ...
10
votes
Accepted
Find the consecutive zeros in a DataFrame and do a conditional replacement
Consider the following approach:
...
10
votes
Accepted
Delete/Drop only the rows which has all values as NaN in pandas
The complete command is this:
df.dropna(axis = 0, how = 'all', inplace = True)
you must add inplace = True argument, if you ...
9
votes
9
votes
Accepted
How to find the count of consecutive same string values in a pandas dataframe?
Break col1 into sub-groups of consecutive strings. Extract first and last entry per sub-group.
Something like this:
...
oW_♦
- 5,950
8
votes
Using pandas, check a column for matching text and update new column if TRUE
You simply need to do:
df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern)
df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})
8
votes
Accepted
Pandas apply return: Must have equal len keys and value when setting with an iterable
I found the issue, I need to return a pd.Series()
...
8
votes
Accepted
Mapping column values of one DataFrame to another DataFrame using a key with different header names
You can convert df2 to a dictionary and use that to replace the values in df1
...
7
votes
Accepted
Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'
Based on the documentation 0.22 and 0.24.1, the flavor does not exist in the argument list of the to_sql method. You're probably running the ...
7
votes
Accepted
Pandas merge column duplicate and sum value
In another case when you have a dataset with several duplicated columns and you wouldn't want to select them separately use:
...
6
votes
Accepted
Replacing column values in pandas
As Emre already mentioned, you may use the groupby function. After that, you should apply reset_index to move the MultiIndex to the columns:
...
6
votes
How to remove rows from a data frame that are identical to other df?
Simpler to use isin() with dropna()
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isin.html
<...
6
votes
Mapping column values of one DataFrame to another DataFrame using a key with different header names
df3 = pd.merge(df1,df2,left_on=['cat'+str(i)], right_on = ['cat_codes'], how = 'left')
I would iterate this for cat1,cat2 and cat3. This does not replace the ...
6
votes
Accepted
Export pandas dataframe to a nested dictionary from multiple columns
Using dict comprehension with nested groupby:
d = {k: f.groupby('subgroup')['selectedCol'].apply(list).to_dict()
for k, f in df.groupby('maingroup')}
Output:...
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
dataframe × 318pandas × 216
python × 196
data-cleaning × 30
dataset × 21
r × 20
machine-learning × 18
python-3.x × 17
numpy × 14
matplotlib × 10
scikit-learn × 9
apache-spark × 9
pyspark × 9
csv × 9
time-series × 8
data × 7
groupby × 7
nlp × 6
visualization × 6
data-analysis × 6
bigdata × 5
data-science-model × 5
preprocessing × 5
jupyter × 5
excel × 5