1
$\begingroup$

I am facing a dilemma with a project of mine. One of the variables (numerical) doesn't have enough data i,e almost 99% data are missing. However, upon talking to the domain experts, it appears that the particular variable is important to the problem we are trying to solve (model). Initially, I thought of converting it to a binary variable such that 1 will represent that the variable has a value at that position and 0 will represent the missing value. However, it seems that we are missing information by doing it.

Can anybody suggest any way go forward?

One thought came to me is to discretize the variables using quantiles, but then what to do with the missing values?

Another one is to include both the binary variable along with the original variable in the model with missing values replaced by some imputed values. But I cannot come to any logical reasoning as to why or why not this will work.

Any light on this matter would be greatly helpful (other than of course drop it altogether).

Thanks.

$\endgroup$
1
$\begingroup$

What is the best way to deal with this kind of missing value problem can only be answered empirically? It will vary depending on your dataset and algorithm of choice. But here is a few things you can try.

Impute the missing value

  • Impute missing value of the mean
  • Impute missing value with special values. For example, If the variable takes only positive values, then you can encode the missing value as 0.

Try to predict the missing value

  • use the other variables to predict the missing value. (However, if you can actually make a good prediction of the missing value out of the rest, then this might suggest you can drop this variable altogether)

Another one is to include both the binary variable along with the original variable in the model with missing values replaced by some imputed values.

There is nothing inherently wrong with this method. You should certainly try. One possible downside of this method when you have 99% of value missing, this original variable is going to highly correlated with the derived is_missing variable. This can be problematic depends on the particular classification algorithm you are using. For example:

  • It is widely known that multicollinearity is a huge problem for any variant of linear regression
  • Support Vector Machine suffers a similar problem [Ref]
  • Naive Bayes assumes independence among features. This is even stronger.
  • The is_missing variable is a categorical variable, this makes it tricky to define a distance metrics in K-Nearest-Neighbour algorithm
| improve this answer | |
$\endgroup$
  • $\begingroup$ Why not introduce an is_missing feature, and then impute them mean? The is_missing feature makes the imputation irrelevent for predictive purposes. $\endgroup$ – Matthew Drury Feb 9 '19 at 6:43
  • $\begingroup$ there is nothing wrong inherently wrong with introducing an is_missing feature. One can certainly try. The OP has mentioned that method in the question. Again, what's the best way can only be determined empirically. I don't understand what do you mean by "makes the imputation irrelevant". Can you elaborate a bit more? $\endgroup$ – Louis T Feb 9 '19 at 8:23
  • $\begingroup$ I added a paragraph for this. $\endgroup$ – Louis T Feb 9 '19 at 8:31
  • $\begingroup$ Nice, thanks for adding. What classification algorithms suffer from feature correlation? I was under the impression that this can be problematic for statistical inference, but not for predictive purposes (though some gradient based algorithms can suffer from long valleys in the loss function I suppose). $\endgroup$ – Matthew Drury Feb 9 '19 at 20:11
  • $\begingroup$ What I mean by "makes the imputation irrelevent" is that, in the presence of an is_missing feature, the predicted values from most models are relatively insensitive to what value you choose to fill the missings in the original feature. $\endgroup$ – Matthew Drury Feb 9 '19 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.