I have data set that has data for patients:

Arrival_Date : is when the patient has arrived

Seen_By_Nurse : is number of minutes patient take to be seen by nurse since arrival when value is NaN it means patient is still waiting and has not been seen by Nurse yet

Seen_By_Dr: is number of minutes patient take to be seen by dr since arrival when value is NaN it means patient is still waiting and has not been seen by Dr yet

Upset_Patient : Is this patient upset or not 1 or 0

The data look like this:

Arrival_Date   Seen_By_Nurse   Seen_By_Dr   ... many other fields ...   Upset_Patient
14/8/19 14:15  10              25                                       1
14/8/19 15:32  13              NaN                                      0
14/8/19 15:35  16              NaN                                      1
14/8/19 15:39  NaN             NaN                                      0
14/8/19 15:44  NaN             NaN                                      1

My question is: I know that working with NaN failing many predictive models. Therefore, I have to either fill it with other values, such as mean. Which is going to be wrong in my case. Or delete it which also going to be wrong in my case as the majority has not been seen by Dr or Nurse yet.

Should I fill it with big number such as 9999? Are there any other approaches?

What is the best way to deal with NaN in this case?

  • $\begingroup$ What i do is check whether sum of all Nans in a column is more than 70% .Also we can substitute it with 'other' in case of categorical column, then do one hot encoding and ultimately drop it. You can also substitute with mean /mode and plot it against Target variable to see whether it influences Target or not. $\endgroup$ – Shubh Sep 19 '19 at 6:15
  • $\begingroup$ @Shubh Cant replace it with other, it is numeric value $\endgroup$ – asmgx Sep 19 '19 at 6:36
  • $\begingroup$ with numeric you can only substitute for mean. $\endgroup$ – Shubh Sep 19 '19 at 6:47
  • $\begingroup$ @Shubh I have mentioned in my question that mean is not right thing to do in my case $\endgroup$ – asmgx Sep 19 '19 at 6:48
  • $\begingroup$ When the value is missing, can you generate the amount of time so far? (Is there a "current time" that the data is a snapshot of, or a variable "observation time" per patient?) $\endgroup$ – Ben Reiniger Sep 19 '19 at 18:32

Going according to your problem, you should set the NaN values to 0. It follows logically as the number of minutes spent with the nurse or doctor will be zero if the patient hasn't visited them yet. And yes as you said filling with mean is not correct as it would not represent the problem correctly.

Also, if you fill the NaN values with big number such as 9999 they might be treated as outliers which is not the case. Hence, purely thinking from a logical point of view you should replace them with 0. But I also suggest you to run trial experiments with some other values such as large values. Machine Learning is a lot about hyperparameter tuning and in this case how you replace your NaN values behaves like a hyperparameter!

  • $\begingroup$ I think OP is saying the variables are the amount of waiting time, not the duration of contact with the doctor/nurse. So filling with zero would be misleading, although it at least gets the "other"ness across. $\endgroup$ – Ben Reiniger Sep 19 '19 at 18:29

This is a feature engineering problem. That patients have not been visited yet is valuable informationen but also clearly not on the same scale as minutes until visit.

Create a new categorical variable 1=visited and 0=not visited.

Additionally instead of imputation try using a model that can cope with NA like randomforest or XGB.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.