0
$\begingroup$

I have a column named BsmntQual that gives a ranking on the height of the basement per each house. These are all of the unique values in this column:

print(train['BsmtQual'].unique().tolist())

>>> ['Gd', 'TA', 'Ex', nan, 'Fa']

This is the legend of this particular column:

BsmtQual: Evaluates the height of the basement

       Ex   Excellent (100+ inches) 
       Gd   Good (90-99 inches)
       TA   Typical (80-89 inches)
       Fa   Fair (70-79 inches)
       Po   Poor (<70 inches
       NA   No Basement < Not to be confused with the nan value above

This is what I did for my other ranked columns but this one did not have NaN values:

train['ExterQual'] = train['ExterQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [4, 3, 2, 1])  # Exterior Quality

For numerical values, a common way is to fill all NaN values with the mean of the column. But what is a good way of replacing the NaN values for columns such as these?

Here is the full dataset

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Your legend clearly states that missing values mean that there is no basement. You could fill the missing values with ’NoBase’ to make that point clearer (train[‘BsmtQual’].fillna(‘NoBase’, inplace=True)). When you rank them then, you just add another ranking for ‘NoBase’, maybe 0, based on your example rating that gives a higher value to more quality.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Read it again. The legend clearly states that NA = No Basement. If you look above that, I got a NaN value. They're not the same thing. $\endgroup$
    – Andros Adrianopolos
    Jun 30, 2019 at 10:22
  • $\begingroup$ @AndrosAdrianopolos If they’re not the same thing, then I’d suggest asking whoever gave you the dataset to make it clear what the missing values represent. $\endgroup$
    – m13op22
    Jul 1, 2019 at 3:10
  • $\begingroup$ Okay fine but it's pretty clear mate. NA & nan don't look that similar. Anyways. Do you have a solution to this problem? $\endgroup$
    – Andros Adrianopolos
    Jul 1, 2019 at 3:50
  • 1
    $\begingroup$ @AndrosAdrianopolos - when you use pd.read_csv the default parser automatically transforms the string 'NA' into nan values so this answer is correct. - there are no 'NA' values in your print.. $\endgroup$
    – yoav_aaa
    Jul 1, 2019 at 13:09
  • $\begingroup$ @yoav_aaa Good catch. Andros, if you want to ignore that behavior, you can set keep_default_na = False to keep pandas from reading NA as nan, and it will just leave those values as the string 'NA'. $\endgroup$
    – m13op22
    Jul 1, 2019 at 13:36
-1
$\begingroup$

method : {‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None Method to use for filling holes in reindexed Series pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use NEXT valid observation to fill gap

from pandas fillna, these methods can be used or fill in with the highest frequency category.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.