# How do I deal with missing values in a data set?

I am trying to build a binary classification model which predicts whether a patient would me infected with a certain disease at the the end of his hospital stay or not. The features that I have are results of different standard medical tests. But the issue is almost all of these results have around 60% - 80% missing values as not all the tests are relevant for all the patients. So how do I deal with the missing values as dropping them is not an option here. Also since the medical test results lie on scale ranging from low to high, should i converted them to categorical variable with High, Low, Medium, Null (for missing data), based on the standard medical test ranges?

• Dan's answer is pretty much what you can do, I guess. Just one remark. You could "demean" continuous features ($x_i=x_i-x_{mean}$) and replace missings with 0. So all missings would neither be "positive" or "negative". In this sense you would have a continuous indication (instead of low/medium...), which means more information. But you really need to try if the strategy would work or not. – Peter Aug 6 '19 at 10:17
• Yes it makes sense. But I am dealing with medical lab test results, for e.g. Platelet count, as features here so putting them on a scale of High, Medium, Low makes more sense in my opinion. – Krantikari Aug 6 '19 at 11:20

If the features are categorical, just fill the NAs with "Missing" as a new category. If they're continuous there are a number of approaches you could try. As a starting point, you could simply fill them with the mean value (or median if you have outliers significantly skewing things) and add a new binary feature that flags the value as being missing in the original feature. So...

| Feature A |
| --------- |
|    14     |
|    nan    |
|    23     |


becomes

| Feature A | Feature A Missing |
| --------- | ----------------- |
|    14     |          0        |
|   18.5    |          1        |
|    23     |          0        |


I've never found a hard and fast rule for the "best" way of doing things; I suggest trying the above and cross-validating your model to make sure its performance is within acceptable limits.

Discretizing continuous variables into High/Medium/Low/Missing may also be helpful; again, try it and cross-validate to see if it improves performance. You should try it both with and without the original continuous variables too.

• Thanks Dan. I am not sure how helpful would be replacing the 70-80% missing values with central tendencies of 20-30% available data though. – Krantikari Aug 6 '19 at 11:17