Problem with important feature having a lot of missing value

Question

I am facing a dilemma with a project of mine. One of the variables (numerical) doesn't have enough data i,e almost 99% data are missing. However, upon talking to the domain experts, it appears that the particular variable is important to the problem we are trying to solve (model). Initially, I thought of converting it to a binary variable such that 1 will represent that the variable has a value at that position and 0 will represent the missing value. However, it seems that we are missing information by doing it.

Can anybody suggest any way go forward?

One thought came to me is to discretize the variables using quantiles, but then what to do with the missing values?

Another one is to include both the binary variable along with the original variable in the model with missing values replaced by some imputed values. But I cannot come to any logical reasoning as to why or why not this will work.

Any light on this matter would be greatly helpful (other than of course drop it altogether).

Thanks.

Answer 1

What is the best way to deal with this kind of missing value problem can only be answered empirically? It will vary depending on your dataset and algorithm of choice. But here is a few things you can try.

Impute the missing value

• Impute missing value of the mean
• Impute missing value with special values. For example, If the variable takes only positive values, then you can encode the missing value as 0.

Try to predict the missing value

• use the other variables to predict the missing value. (However, if you can actually make a good prediction of the missing value out of the rest, then this might suggest you can drop this variable altogether)

Another one is to include both the binary variable along with the original variable in the model with missing values replaced by some imputed values.

There is nothing inherently wrong with this method. You should certainly try. One possible downside of this method when you have 99% of value missing, this original variable is going to highly correlated with the derived is_missing variable. This can be problematic depends on the particular classification algorithm you are using. For example:

• It is widely known that multicollinearity is a huge problem for any variant of linear regression
• Support Vector Machine suffers a similar problem [Ref]
• Naive Bayes assumes independence among features. This is even stronger.
• The is_missing variable is a categorical variable, this makes it tricky to define a distance metrics in K-Nearest-Neighbour algorithm