4
$\begingroup$

My data looks like this:

enter image description here

birth_date has 634,990 missing values
gender has 328,849 missing values

Both of these are a substantial amounts since I have 900k entries, so I can't discard empty rows. For birth_date someone recommended using Multivariate imputation by Chained equation (MICE). I don't know what predictive model I should use for gender. Of the non-missing data, there are 5x more males than females.

Can someone tell me what would be best practice here? What would be the best way to fill in the missing values for gender ?

I'm using the data to predict bike-ride duration and final destination (I know they're shown on the table above)

$\endgroup$
  • $\begingroup$ Semi-Supervised Regression may be an option. I'm not a expert, though. $\endgroup$ – Rafael Muñoz-Mas May 14 at 7:58
5
$\begingroup$

I agree with Simon's advice. I find that the gains that you obtain from using any external method of imputation is often inferior to an internal method, and on top of this, exposes you to even more potential of severely screwing up with respect to data leakage.

That being said, besides using an algorithm that automatically handles missing data for you (which often are models based off trees/rules, though they do not all use the same method of imputation), there are external based methods that might be of interest. I find that as you get more "fancier" the results are not enough of an improvement compared to the computational pain it is to use them. Starting with the simplest;

1) Mode imputation; simply use the most common gender in your training data set. For your test dataset, use the most common gender that exists in your training data set. Since there are 5x more males than females, this would result in you almost certainly assigning male to all observations with missing gender. Obviously, this doesn't use a whole lot of information besides the observed frequency of the class, but this method is pretty common and often "good enough".

2) kNN imputation; take the k most closest neighbours (that do not have missing genders) to the observation that you wish to impute gender for. Then, simply treat each of these k neighbours as a committee of "voters" who use their own gender as their vote. Weight each vote by how close they are (based off other variables that aren't missing) to the observation with the missing gender value. Whichever gender wins in votes gives you the imputed gender.

This method to me, is a clear improvement over method 1) and is also quite fast. However, this will require you to center and scale your data (because we are using distances to define "closeness") and k is now a tuning parameter which further complicates matters.

3) Random Forest imputation; initially, use method 1) to temporarily fill in your missing genders (just mode impute). Then, run a random forest algorithm on the imputed dataset, generating N trees. Compute what is referred to as the "proximity matrix", where each $(i,j), i \ne j $ entry in this matrix (diagonal entries are all 0) is equal to the number of times observations $i$ and $j$ fall in the same terminal node through the entire forest divided by the number of trees in the forest. Using these proximities as weights, calculate a weighted vote of all the observations that do not have missing genders using their genders as their "vote". Change any prior "temporary" imputed genders from the initial mode imputation to what has been calculated by the random forest if they differ. Repeat (fit another random forest again), using the imputed genders from the previous random forest, until all observations converge to a single gender or until some stopping criteria.

This method is incredibly costly but is probably pretty accurate (I haven't used it much because it is slow). You will also have to deal with an additional tuning parameter; namely how many variables you wish to randomly select in each split.

4) MICE: I haven't really studied this method too closely, but you seem to have mentioned it. One thing I will say is that all of these methods can be used with any kinds of missing data; categorical (like gender) or continuous (like birth_date, though for method 1) you would probably use mean/median imputation instead for continuous variables, and for methods 2 and 3) you would no longer use a "vote" but a weighted average). Ultimately, MICE is just one of many methods of imputation that you can use which is why one needs to properly validate their modelling choices within cross validation if you choose to use an external method of imputation. If you have the time, try a bunch of methods and use the highest performing one. Otherwise, use a method that seems "reasonable enough" given time constraints.

$\endgroup$
5
$\begingroup$

You can use one of the popular gradient boosting tree implementations such LightGBM and XGBoost as your predictive model. They can handle missing values during training and the results are often better than any imputation done in preprocessing.

The way they achieve this is by using the splitting of the decision trees they are built on. The missing values are ignored when finding the best split, then once it is found they are allocated to whichever side will minimize the loss function most.

$\endgroup$
2
$\begingroup$

If you are planning to go with imputing values as a pre-processing step, I just want to add that it is better to impute missing values of multiple variables jointly because you will be able to better preserve relationships between those variables. The MICE algorithm should automatically take care of that for you.

Here is an example of how MICE works when there are multiple variables with missing values:

To make the chained equation approach more concrete, imagine a simple example where we have 3 variables in our dataset: age, income, and gender, and all 3 have at least some missing values. The MAR assumption would imply that the probability of a particular variable being missing depends only on the observed values, and that, for example, whether someone’s income is missing does not depend on their (unobserved) income. In step 1 of the MICE process, each variable would first be imputed using, e.g., mean imputation, temporarily setting any missing value equal to the mean observed value for that variable. Then in step 2 the imputed mean values of age would be set back to missing. In step 3, a linear regression of age predicted by income and gender would be run using all cases where age was observed. In step 4, predictions of the missing age values would be obtained from that regression equation and imputed. At this point, age does not have any missingness. Steps 2–4 would then be repeated for the income variable. The originally missing values of income would be set back to missing and a linear regression of income predicted by age and gender would be run using all cases with income observed; imputations (predictions) would be obtained from that regression equation for the missing income values. Then, steps 2–4 would again be repeated for the variable gender. The originally missing values of gender would be set back to missing and a logistic regression of gender on age and income would be run using all cases with gender observed; predictions from that logistic regression model would be used to impute the missing gender values. This entire process of iterating through the 3 variables would be repeated until convergence; the observed data and the final set of imputed values would then constitute one “complete” data set.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.