How can I get the number of missing value in each row in Pandas dataframe. I would like to split dataframe to different dataframes which have same number of missing values in each row.
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
You can apply a count over the rows like this:
test_df.apply(lambda x: x.count(), axis=1)
A B C 0: 1 1 3 1: 2 nan nan 2: nan nan nan
0: 3 1: 1 2: 0
You can add the result as a column like this:
test_df['full_count'] = test_df.apply(lambda x: x.count(), axis=1)
A B C full_count 0: 1 1 3 3 1: 2 nan nan 1 2: nan nan nan 0
When using pandas, try to avoid performing operations in a loop, including
applymap etc. That's slow!
A DataFrame object has two axes: “axis 0” and “axis 1”. “axis 0” represents rows and “axis 1” represents columns.
If you want to count the missing values in each column, try:
df.isnull().sum() as default or
On the other hand, you can count in each row (which is your question) by:
It's roughly 10 times faster than Jan van der Vegt's solution(BTW he counts valid values, rather than missing values):
In : %timeit -n 1000 df.apply(lambda x: x.count(), axis=1) 1000 loops, best of 3: 3.31 ms per loop In : %timeit -n 1000 df.isnull().sum(axis=1) 1000 loops, best of 3: 329 µs per loop