Alternatives to imputation of missing values?

Question

So I'm quickly learning that dealing with missing values for feature(s) in some of your observations is a part of every day life in data. I get the gist of imputation, when/how it's appropriate and when it's not, and I'll read up on it in the near future. But, what about this:

Suppose you have predictors $X_1, \dots, X_p$, and you want to model, say, a binary response $Y$ via, say, logistic regression. Instead of just imputing values to missing predictor values, couldn't you have a separate model built for every possible subset of the predictors, and apply that model for prediction when those predictors happen to be present? Each of those models would naturally be trained on just the data for which those predictors (and also others) are present. This seems to me a more reasonable approach than just making up values, but I have no theoretical justification for this.

I do realize that this involves building $2^p$ different models, with $2^p$ different model matrices, etc., but for a moderate $p$ and $n$ it could be feasible, and especially if only a few of your features tend to be missing more often than others.

Is this ever done? And if so, is there a standard way to implement this in R? In the case of logistic regression, you can specify to R's glm function how you want it to handle NA values, but your only options seem to be either tossing out observations altogether, or some kind of imputation scheme.

Thoughts?

Answer 1

These are my thoughts:

This seems to me a more reasonable approach than just making up values, but I have no theoretical justification for this.

Actually, there is not much theoretical justification for classical imputation. I think your method makes sense when just very few of the predictors have missing values. For instance, if there is just one predictor that has missing values, you can build a model to imput those missing values. However, as you said, things scale very badly. Moreover, apart from computational issues, if there are lots of missing values in every predictor your data is not very good, and therefore any predictive model won't be very good anyway.

This is done sometimes, but I don't think there's any library in R that implements it, you'll have to code it yourself (I don't think it is difficult). If you do it and your final model is a logistic regression, I don't recommend to fit the missing values with another linear model, as you will suffer from a collinearity problem.

One other thing that is typically done is that, for every predictor with missing values, create a binary predictor that is $0$ if the other predictor's value is missing and $1$ otherwise.

Answer 2

Supposedly you had these models, divided into subspaces, then what if in your test dataset a point is close proximity to where the missing point is in your training data. What would be the best solution? Have the average output of the two closest models? Is the output binary? And then would you have to weight each model, based on that proximity? Then I am thinking that many submodels could overfit on the subset of the data. Just some thoughts, which may be challenging to overcome.

Moreover, you could use other types of Classifiers which deal with missing data, like some decision trees implementations.

Finally, data imputation is a delicate art, and doing it right will leverage your model. For instance, you wouldn't want to impute all the missing data with the average as then you may change your data distribution. Knn techniques or ensemble regressions would be preferred.