Dealing with NaN for predictive models

Question

I have data set that has data for patients:

Arrival_Date : is when the patient has arrived

Seen_By_Nurse : is number of minutes patient take to be seen by nurse since arrival when value is NaN it means patient is still waiting and has not been seen by Nurse yet

Seen_By_Dr: is number of minutes patient take to be seen by dr since arrival when value is NaN it means patient is still waiting and has not been seen by Dr yet

Upset_Patient : Is this patient upset or not 1 or 0

The data look like this:

Arrival_Date   Seen_By_Nurse   Seen_By_Dr   ... many other fields ...   Upset_Patient
14/8/19 14:15  10              25                                       1
14/8/19 15:32  13              NaN                                      0
14/8/19 15:35  16              NaN                                      1
14/8/19 15:39  NaN             NaN                                      0
14/8/19 15:44  NaN             NaN                                      1

My question is: I know that working with NaN failing many predictive models. Therefore, I have to either fill it with other values, such as mean. Which is going to be wrong in my case. Or delete it which also going to be wrong in my case as the majority has not been seen by Dr or Nurse yet.

Should I fill it with big number such as 9999? Are there any other approaches?

What is the best way to deal with NaN in this case?

Answer 1

Going according to your problem, you should set the NaN values to 0. It follows logically as the number of minutes spent with the nurse or doctor will be zero if the patient hasn't visited them yet. And yes as you said filling with mean is not correct as it would not represent the problem correctly.

Also, if you fill the NaN values with big number such as 9999 they might be treated as outliers which is not the case. Hence, purely thinking from a logical point of view you should replace them with 0. But I also suggest you to run trial experiments with some other values such as large values. Machine Learning is a lot about hyperparameter tuning and in this case how you replace your NaN values behaves like a hyperparameter!

Answer 2

This is a feature engineering problem. That patients have not been visited yet is valuable informationen but also clearly not on the same scale as minutes until visit.

Create a new categorical variable 1=visited and 0=not visited.

Additionally instead of imputation try using a model that can cope with NA like randomforest or XGB.