I am trying to build a binary classification model which predicts whether a patient would me infected with a certain disease at the the end of his hospital stay or not. The features that I have are results of different standard medical tests. But the issue is almost all of these results have around 60% - 80% missing values as not all the tests are relevant for all the patients. So how do I deal with the missing values as dropping them is not an option here. Also since the medical test results lie on scale ranging from low to high, should i converted them to categorical variable with High, Low, Medium, Null (for missing data), based on the standard medical test ranges?
If the features are categorical, just fill the NAs with "Missing" as a new category. If they're continuous there are a number of approaches you could try. As a starting point, you could simply fill them with the mean value (or median if you have outliers significantly skewing things) and add a new binary feature that flags the value as being missing in the original feature. So...
| Feature A | | --------- | | 14 | | nan | | 23 |
| Feature A | Feature A Missing | | --------- | ----------------- | | 14 | 0 | | 18.5 | 1 | | 23 | 0 |
I've never found a hard and fast rule for the "best" way of doing things; I suggest trying the above and cross-validating your model to make sure its performance is within acceptable limits.
Discretizing continuous variables into High/Medium/Low/Missing may also be helpful; again, try it and cross-validate to see if it improves performance. You should try it both with and without the original continuous variables too.