# How to deal with large proportion of missing values in categorical variable

I have a dataset of around 5,500 observations.

One of the variables is Gender for which at least 25% of the observations are missing.

Dropping the missing values seems a bit brute, however I have not found a good way of interpolating binary data.

Other variables of the data are Country, Date of birth, and Revenue. None of them with relevant correlation with Gender.

What is the best way to handle these NaNs?

I was thinking of using a logistic regression function with Gender as the target and the rest of the variables as the predictors, but I am not sure whether this is a good choice.

Thanks.

• If you have millions of samples, droping 25% may be OK if you still have enough data for your training and testing. This may be a problem to do this with 5500 observations. Is the gender important for your problem ? You could also drop gender feature if it is not relevant or may introduce an unwanted or unetical gender bias. You say Revenue is not correlated to gender, unfortunately it is not true in general and it seems to be often correlated Please give details about your problem/question you try to solve/answer? Have you some other features ? What is your target ?
– Malo
Commented Sep 11, 2021 at 9:41
• There is no way to impute the missing values as it seems. So no alternative to dropping this variable. You could use two models, one for data with known gender, one for data with missing gender Commented Sep 11, 2021 at 10:43
• Why not simply use the NA as one category ? Commented Jun 12, 2022 at 17:16

1.) Drop the Gender feature as you suggested. (only do this if you are sure that the gender feature is not related to your output feature)

2.) Impute the feature with the most frequent value (which is nothing but the mode or a constant using SimpleImputer.

3.) Try to predict the gender using a binary classification model as you suggested.

4.) Revenue can be correlated with gender. Use it to impute the values.

5.) Train a model with the imputed values of gender and another model with gender feature dropped. See which gives best results.

6.) Encode all the NaN's into one category (maybe unknown). Now you have 3 classes for the gender feature. Train the model using this feature.

Try all the methods and see which gives you the best results