How to handle missing value if imputation doesnt make sense

Question

I have column/feature in my dataset showing years a person has been married "years_married". Since not every person is married there are NaN fields.

It does not make sense to fillna(0) "years_married" since 0 would mean the person just married.A mean imputation also doesnt make sence.

What is the best way to use this feature?

I thought about creating a second feature like "is_married" and using fillna(0) on the "years_married" and then hoping that the decision tree would understand the combination is_married=0 and years_since_married=0

Is solution ok, are there good solutions?

Answer 1

I think this is a good solution. You could also try to set a unique negative value for non-married people, especially if you are using a tree-based model.

Answer 2

You could consider setting years_married to -1, then it is different from columns for the ones that are just married and could thus be understood by a decision tree.

But remember that this might not be intuitive for users other than yourself and that some conditional statements etc. behave differently when turning NaNs into -1 values.

Answer 3

Your approach of a binary categorical feature, is_married definitely sounds good.

In some of my projects, I have checked for the percentage of missing values in a column. For instance, if a certain column has more than 40% of missing values then imputation is obviously out of the picture. It is either a replacement by -1 or dropping that column if it is not important.

Answer 4

1. If the variable is categorical and not ordered, it may make sense to create a new category 'not_married' to represent the missing values. This would allow you to keep the information about marital status and avoid imputing values that don't make sense (e.g., 0 for a newly married person).

2. If the variable is categorical and ordered (e.g., 0 = not married, 1 = newly married, 2 = married for 1 year, etc.), imputing a value of -1 for the missing values could make sense. This acknowledges that the missing values represent a previous state (not married) and allows you to preserve the ordering of the categories. However, this approach does sacrifice some granularity in the data.

3. If the variable is continuous, imputing a value of 0 may not be appropriate since it assumes that all missing values correspond to newly married individuals. One idea is to use a flag variable to indicate whether someone is married or not, and let the model learn the relationship between this variable and 'years_married'. Another option (that doesn´t reflect reality but may make this variable value less relevant to the model) is to impute values using a more complex approach, such as a regression model that predicts 'years_married' based on other variables in the dataset.

4. It's important to consider the context of the dataset and the modeling approach when deciding how to handle missing values. You could try different methods (e.g., imputing -1 vs. creating a 'not_married' category) and compare their performance in terms of how well they explain the variability in the outcome variable.

Answer 5

I'm not sure of the performance, but if it was me I would also explore categorical variables (One-hot encoding). You could make the features:

not_married.
newly_married (0-5 years)
a_while_ago_married (5-15 years)
long_term_married (15+ years)

dumb names, but you could try it out. Also with different bin and bin sizes.