I am trying to build a binary classification model which predicts whether a patient would me infected with a certain disease at the the end of his hospital stay or not. The features that I have are results of different standard medical tests. But the issue is almost all of these results have around 60% - 80% missing values as not all the tests are relevant for all the patients. So how do I deal with the missing values as dropping them is not an option here. Also since the medical test results lie on scale ranging from low to high, should i converted them to categorical variable with High, Low, Medium, Null (for missing data), based on the standard medical test ranges?

  • 1
    $\begingroup$ Dan's answer is pretty much what you can do, I guess. Just one remark. You could "demean" continuous features ($x_i=x_i-x_{mean}$) and replace missings with 0. So all missings would neither be "positive" or "negative". In this sense you would have a continuous indication (instead of low/medium...), which means more information. But you really need to try if the strategy would work or not. $\endgroup$ – Peter Aug 6 '19 at 10:17
  • $\begingroup$ Yes it makes sense. But I am dealing with medical lab test results, for e.g. Platelet count, as features here so putting them on a scale of High, Medium, Low makes more sense in my opinion. $\endgroup$ – Krantikari Aug 6 '19 at 11:20

If the features are categorical, just fill the NAs with "Missing" as a new category. If they're continuous there are a number of approaches you could try. As a starting point, you could simply fill them with the mean value (or median if you have outliers significantly skewing things) and add a new binary feature that flags the value as being missing in the original feature. So...

| Feature A |
| --------- |
|    14     |
|    nan    |
|    23     |


| Feature A | Feature A Missing |
| --------- | ----------------- |
|    14     |          0        |
|   18.5    |          1        |
|    23     |          0        | 

I've never found a hard and fast rule for the "best" way of doing things; I suggest trying the above and cross-validating your model to make sure its performance is within acceptable limits.

Discretizing continuous variables into High/Medium/Low/Missing may also be helpful; again, try it and cross-validate to see if it improves performance. You should try it both with and without the original continuous variables too.

| improve this answer | |
  • $\begingroup$ Thanks Dan. I am not sure how helpful would be replacing the 70-80% missing values with central tendencies of 20-30% available data though. $\endgroup$ – Krantikari Aug 6 '19 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.