Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I have column/feature in my dataset showing years a person has been married "years_married".
Since not every person is married there are NaN fields.
It does not make sense to fillna(0) "years_married" since 0 would mean the person just married.A mean imputation also doesnt make sence.
What is the best way to use this feature?
I thought about creating a second feature like "is_married" and using fillna(0) on the "years_married" and then hoping that the decision tree would understand the combination is_married=0 and years_since_married=0
Your approach of a binary categorical feature, is_married definitely sounds good.
In some of my projects, I have checked for the percentage of missing values in a column. For instance, if a certain column has more than 40% of missing values then imputation is obviously out of the picture. It is either a replacement by -1 or dropping that column if it is not important.
If the variable is categorical and not ordered, it may make sense to create a new category 'not_married' to represent the missing values. This would allow you to keep the information about marital status and avoid imputing values that don't make sense (e.g., 0 for a newly married person).
If the variable is categorical and ordered (e.g., 0 = not married, 1 = newly married, 2 = married for 1 year, etc.), imputing a value of -1 for the missing values could make sense. This acknowledges that the missing values represent a previous state (not married) and allows you to preserve the ordering of the categories. However, this approach does sacrifice some granularity in the data.
If the variable is continuous, imputing a value of 0 may not be appropriate since it assumes that all missing values correspond to newly married individuals. One idea is to use a flag variable to indicate whether someone is married or not, and let the model learn the relationship between this variable and 'years_married'. Another option (that doesn´t reflect reality but may make this variable value less relevant to the model) is to impute values using a more complex approach, such as a regression model that predicts 'years_married' based on other variables in the dataset.
It's important to consider the context of the dataset and the modeling approach when deciding how to handle missing values. You could try different methods (e.g., imputing -1 vs. creating a 'not_married' category) and compare their performance in terms of how well they explain the variability in the outcome variable.