I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever there is a 'U' in the gender column, there is an NA in two of the other columns namely Age and Tenure. This could basically be interpreted as a person who is not ready to disclose their Gender is not ready to disclose their age and tenure. How do I deal with such a situation? Apart from these three columns there are other 16 columns in the dataset that have got meaningful data in them. Would the normal imputation techniques out there like a KNN Imputation help me out in such a case?

Here is my reproducible example that I have tried my best with:

x<-data.frame(gender=c('M','M','F','F','U','F','M','U'),age=c(21,24,20,34,NA,40,56,NA),tenure=c(7,4,5,3,NA,2,4,NA),job=c('Doctor','IT','Banking','Truck Driver','Finance','Agriculture','Electrician','Teacher'),country=c('Australia','America','New Zealand','Sweden','England','France','Denmark','Norway'))

The Dataframe:

 gender age  tenure      job      country
1      M   21     7       Doctor    Australia
2      M   24     4        IT       America
3      F   20     5      Banking    New Zealand
4      F   34     3    Truck Driver Sweden
5      U   NA     NA      Finance    England
6      F   40      2    Agriculture  France
7      M   56      4    Electrician  Denmark
8      U   NA     NA      Teacher    Norway

As you can see from the example above whenever the gender is undefined, there are missing values in both age and tenure and this is the case overall in the entire dataset. What would be the best way to deal with such a situation? And this what is called a Missing at Random data, is that right? Any suggestions would be extremely helpful. Thanks a lot.

  • $\begingroup$ This is Missing Not At Random. $\endgroup$ – user2974951 Dec 11 '18 at 8:31

You kinda answered your question, if you pay close attention you already now the value the gender U has in your dataset. Why don't you train a different classifier (thus, building 2) for the gender U and then choose the classifier based if the user meets the condition of it's gender being "U" or not?.

Also, can you elaborate more on the goal of the model you're building? That might change the approach to the problem.

  • $\begingroup$ It is a dataset that contains details about a list of transactions and the associated user profile and I am trying to build a model to predict whether the transaction was successful or was it cancelled. Different classifier as in? Would you please be able to elobarate a bit on that please? I would like to explore your idea. Thanks. $\endgroup$ – The Asipiring one Sep 12 '18 at 4:21

If you plan to use LightGBM or XGBoost, the advise is "Do not do anything". These methods treat NA in a specific way, different in each decision tree and the results obtained are usually much better than using imputing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.