1
$\begingroup$

I am quite new to Machine Learning/Data science so I started doing my first Kaggle competition problem from here: https://www.kaggle.com/c/house-prices-advanced-regression-techniques

I was cleaning the training dataset and filling some categorical features with mode values and after sometime I noticed something very strange. I filled the missing values of column 'MasVnrType' with mode and it turned out it filled all of them with None Values even though isna().sum() shows only 8 missing values. Is there some understanding that I'm lacking? Here's the problem

$\endgroup$
1
$\begingroup$

By default, value_counts has the parameter dropna=True that means that None values are not shown in the value count.

Try the code below. You should see the count of "real" None values. There should be a count of 8.

df['MasVnrType'].value_counts(dropna=False)

Why do you see None 864? That is probably because the the None values are actually "None". They are strings. Since they are strings they are not counted in isna(), which only counts actual None values.

How to fix it? When cleaning the data, don't add "None" as a string, but rather add None.

$\endgroup$
1
  • 1
    $\begingroup$ Spot on! Completely agree. $\endgroup$
    – fswings
    Aug 21 at 23:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.