# Understanding missing values in dataset

I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) was sometimes null. Understanding what null means here could help me, I guess, decide what approach should I take to handle missing values for this column. Simple analyzis of the dataset has shown that it is always null, when GarageArea, GarageCars are 0, which probably means no garage exists at all for a given estate:

GarageYrBlt GarageArea  GarageCars

NaN          0           0
1973         45           2


I was wondering if there is a generic approach to understand missing values semantics in a given dataset.

Some approaches I could think of:

1. Go through feature and find those with related names. For instance, all having the word Garage. Then filter samples where those features have 0, null or some negative number that doesn't make sense in this particular context. This doesn't scale well for large number of features.

2. Employ Apriori algorithm for finding associative rules like the one I found above:

• {GarageArea ==0 AND GarageCars == 0} ==> {GarageYrBlt == null}
• or {GarageYrBlt == null} ==> {GarageArea ==0 AND GarageCars == 0}

(although I am not sure if Apriori supports finding the latter, which have multiple elements on the right side).

Anything else?

• What is your goal, specifically? What do you want to be able to say/know/do with this information? Presumably you are going to be removing observations which contain missing values, but the question seems like you are hoping to gain some insight from them. – Upper_Case May 24 at 18:10
• @Upper_Case My goal is to choose approach that improve final performance of model. ~23% of samples in the dataset I analyze have at least one feature missing, so removing them could affect overall perform (I need to check, though). Other choices are imputing mean or constant value, or dropping a feature for all samples. So how can I choose approach for handling those data? Either by trying each one and comparing performance or by understanding what does it mean for the value to be empty in a particular context. Or there's third way? – dzieciou May 28 at 9:09

No generic approach in data handling. Role of DS is understanding data. As you know sometimes NULL value means something, you have to check all aproaches to handling data and I find out which is most suitable to yours data. Usually I marked missing value and resampling datasets. So I getting one dataset without missing values and another one with marked missing value.

Best

• By "marking" you mean creating another boolean feature like GarageYrBlt_was_null for each sample? – dzieciou May 28 at 9:21
• For example, and split dataset to 2 parts with GarageYrBlt_was_null=0 and GarageYrBlt_was_null=1 – Paweł May 28 at 11:04

I would think that the only information that is attainable from null on YearGarageBuilt is what is the effect of having a garage or not on estate value. But it seems, you're asking for what information you should extract from it, but i have a feeling that what you're really looking for is a way to reformat your dataset ( feature engineer something more genuine and less likely to get 'nan' values ), i guess you could come up with a feature that could combine some of the features you presented if that's your real motive for this post.

Note : you could encode that feature in a way that nan would be 0, the oldest possible year would take 1, next 2 etc.. in this case 0 would mean no garage, 1 would mean very very old garage etc.. which should give you a positive slope in terms of YearGarageBuilt/Estate value (the newer the garage the higher the price)if my estate logic is correct.

• I understand having a garage at all, having a bigger garage and having a newer garage all may have an impact on the real estate value. It turned out that not all those features have equal impact. I learned that by trying different approaches (droping column, imputing mean value, etc.) and measuring performance of a model. I thought that understanding the missing values would be a better approach than just trying different approaches and comparing their performance metrics... – dzieciou May 28 at 9:31
• Imputing 0 for nan values makes sense to me, but mapping YearGarageBuilt to 1,2,3 values does not. Why would it change anything? Having information like 0,..., 1972, 2005,2016,2017,2018 should, theoretically, be sufficient for Random Forest to split real estates into classes of: estate with a very old garage (or no garage at all), old garage, new garage, etc. – dzieciou May 28 at 9:37
• resetting the first year available to 1 is meant to reduce the gap between your imputed 0 and the rest of the years, and since year is not a categorical variable but rather an interval, your algorithm would give more weight to bigger values ( it won't consider treating your garagetype as old/new etc.. ) , you should either make bins ( for a certain range of years => old, another range => not very old )etc..or keep it as an interval and rescale your variable to 0-1-2-3.. – Blenz May 28 at 12:03