Handling many missing values

Question

I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of experience I am confused on how to clean them.

First of all, 6 out of 11 columns have more than 277000 NA values. In that case I know that I cannot remove them because they are a big portion of the dataset so I have to impute them. Usually I would do mean substitution but I've read that this approach might create bias in the data and I wouldn't like to have that. I have tried the packages Amelia and mice in R, mice couldn't run properly and after a while it was giving me an error, Amelia was very fast and completed 5 imputations but it introduced many negative values in the dataset.

Anyone with similar experience on that?

Answer 1

There are a number of different ways to impute data when you have missing values. Far to many people just delete those records, which often contain useful training data. Further, some people advocate for simple methods like using the mean of the feature when it is missing. I don't like this method as it seems too simplistic.

Here is a good presentation detailing a couple of methods and particularly drawing attention to imputation through maximum likelihood.

However, my favorite method is collaborative filtering, which is detailed in this paper. The nice thing about collaborative filtering is that it is easy to write from scratch and is also very popular, so is usually contained in most machine learning libraries.

Also think about taking a look at the imputation techniques that might be available in the specific machine learning library that you may be using.

Finally, a word of caution... imputing data adds some linear dependence, so you should expect that you might have to increase your regularization slightly in order to avoid high variance scenarios. I would experiment with (cross validate) removing features, removing records, then imputing data using collaborative filtering and see what produces the best results.

Hope this helps!

Answer 2

This is not a technical question. You should decide which one is more important for you: the columns or the rows that contain these 277000 NA values. Probably the columns are more important.

I recommend deleting the rows at first, building a model (starting from one variable and adding more variables gradually), calculating the model measures (such as adjusted R-Squared) . At this point you have some baseline.

Then restore these deleted rows and see how your model measures change. Maybe these 6 columns are not predictors of your response variable and you are worried for nothing. There is no magic to say which method works for your data. Give it a try. This is exactly what is done in statistics and machine learning.

After understanding your data better, you can decide to fill your missing values or not. Keep in mind that data analysis is an iterative process.