I have a column named GarageYrBlt which just lists the year the garage of that house was built. I have one nan value for this column. Does it make sense to fill it with the columns median (This was a common approach for the missing age values in the Titanic competition)?
$\begingroup$
$\endgroup$
asked Jul 5, 2019 at 8:21
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
13
I suspect you're working with the Ames house price dataset - one of Kaggle's introductory competitions.
Replacing the missing values with the dataset mean / median is very general.
I believe you have a variable which describes which neighbourhood the house is in. A more focussed approach would be to replace that missing value in GarageYrBlt with the mean / median for that neighbourhood (subject to additional sensibility checks like the garage not being built before the house).
-
-
$\begingroup$ And yea I do have a column like that but can you break down your approach a little more? $\endgroup$ Commented Jul 5, 2019 at 8:41
-
$\begingroup$ It assumes that houses in the same neighbourhood have similar characteristics - e.g. built around the same time, of similar construction, similar sized plots of land, etc. So my suggestion would be that if your missing data point is from a house in neighbourhood A, replace the missing value with a statistic calculated using other houses in neighbourhood A. You don't have to use the neighbourhood - you could instead use year of construction, type of house, etc. $\endgroup$– bradSCommented Jul 5, 2019 at 12:17
-
$\begingroup$ The sense check was just to make sure that the statistic calculated using a broader dataset makes sense in a narrow application. E.g. I think garages are generally built at the same time (or after) a house is built, so it would make sense that the statistic you calculate to replace your missing value doesn't imply that the garage was built before the house. $\endgroup$– bradSCommented Jul 5, 2019 at 12:19
-
$\begingroup$ In the Titanic competition, I filled the age data with the mean of its column, relative to the sex so all female passengers with missing age got the mean age of all female passengers. Is this the idea you're referring to? $\endgroup$ Commented Jul 5, 2019 at 13:38