I have a column named BsmntQual that gives a ranking on the height of the basement per each house. These are all of the unique values in this column:


>>> ['Gd', 'TA', 'Ex', nan, 'Fa']

This is the legend of this particular column:

BsmtQual: Evaluates the height of the basement

       Ex   Excellent (100+ inches) 
       Gd   Good (90-99 inches)
       TA   Typical (80-89 inches)
       Fa   Fair (70-79 inches)
       Po   Poor (<70 inches
       NA   No Basement < Not to be confused with the nan value above

This is what I did for my other ranked columns but this one did not have NaN values:

train['ExterQual'] = train['ExterQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [4, 3, 2, 1])  # Exterior Quality

For numerical values, a common way is to fill all NaN values with the mean of the column. But what is a good way of replacing the NaN values for columns such as these?

Here is the full dataset


Your legend clearly states that missing values mean that there is no basement. You could fill the missing values with ’NoBase’ to make that point clearer (train[‘BsmtQual’].fillna(‘NoBase’, inplace=True)). When you rank them then, you just add another ranking for ‘NoBase’, maybe 0, based on your example rating that gives a higher value to more quality.

| improve this answer | |
  • $\begingroup$ Read it again. The legend clearly states that NA = No Basement. If you look above that, I got a NaN value. They're not the same thing. $\endgroup$ – Andros Adrianopolos Jun 30 '19 at 10:22
  • $\begingroup$ @AndrosAdrianopolos If they’re not the same thing, then I’d suggest asking whoever gave you the dataset to make it clear what the missing values represent. $\endgroup$ – HS-nebula Jul 1 '19 at 3:10
  • $\begingroup$ Okay fine but it's pretty clear mate. NA & nan don't look that similar. Anyways. Do you have a solution to this problem? $\endgroup$ – Andros Adrianopolos Jul 1 '19 at 3:50
  • 1
    $\begingroup$ @AndrosAdrianopolos - when you use pd.read_csv the default parser automatically transforms the string 'NA' into nan values so this answer is correct. - there are no 'NA' values in your print.. $\endgroup$ – yoav_aaa Jul 1 '19 at 13:09
  • $\begingroup$ @yoav_aaa Good catch. Andros, if you want to ignore that behavior, you can set keep_default_na = False to keep pandas from reading NA as nan, and it will just leave those values as the string 'NA'. $\endgroup$ – HS-nebula Jul 1 '19 at 13:36

method : {‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None Method to use for filling holes in reindexed Series pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use NEXT valid observation to fill gap

from pandas fillna, these methods can be used or fill in with the highest frequency category.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.