0
$\begingroup$

I have a dataset of around 5,500 observations.

One of the variables is Gender for which at least 25% of the observations are missing.

Dropping the missing values seems a bit brute, however I have not found a good way of interpolating binary data.

Other variables of the data are Country, Date of birth, and Revenue. None of them with relevant correlation with Gender.

What is the best way to handle these NaNs?

I was thinking of using a logistic regression function with Gender as the target and the rest of the variables as the predictors, but I am not sure whether this is a good choice.

Thanks.

Improve this question
$\endgroup$
2
  • $\begingroup$ If you have millions of samples, droping 25% may be OK if you still have enough data for your training and testing. This may be a problem to do this with 5500 observations. Is the gender important for your problem ? You could also drop gender feature if it is not relevant or may introduce an unwanted or unetical gender bias. You say Revenue is not correlated to gender, unfortunately it is not true in general and it seems to be often correlated Please give details about your problem/question you try to solve/answer? Have you some other features ? What is your target ? $\endgroup$
    – Malo
    Sep 11 at 9:41
  • $\begingroup$ There is no way to impute the missing values as it seems. So no alternative to dropping this variable. You could use two models, one for data with known gender, one for data with missing gender $\endgroup$
    – Peter
    Sep 11 at 10:43
0
$\begingroup$

Your options:-

1.) Drop the Gender feature as you suggested. (only do this if you are sure that the gender feature is not related to your output feature)

2.) Impute the feature with the most frequent value (which is nothing but the mode or a constant using SimpleImputer.

3.) Try to predict the gender using a binary classification model as you suggested.

Try all the methods and see which gives you the best results

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.