Filling missing data with other than mean values

Question

What are all the options available for filling in missing data?

One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy.

So how do we deal with missing values if they are are lot of them?

Answer 1

There are of course other choices to fill in for missing data. The median was already mentioned, and it may work better in certain cases.

There may even be much better alternatives, which may be very specific to your problem. To find out whether this is the case, you must find out more about the nature of your missing data. When you understand in detail why data is missing, the probability of coming up with a good solution will be much higher.

You might want to start your investigation of missing data by finding out whether you have informative or non-informative missings. The first category is produced by random data loss; in this case, the observations with missing values are no different from the ones with complete data. As for informative missing data, this one tells you something about your observation. A simple example is a customer record with a missing contract cancellation date meaning that this customer's contract has not been cancelled so far. You usually don't want to fill in informative missings with a mean or a median, but you may want to generate a separate feature from them.

You may also find out that there are several kinds of missing data, being produced by different mechanisms. In this case, you might want to produce default values in different ways.

Answer 2

When it comes to missing data, there are many different methods of filling these values. However, the imputation method you choose, depends largely on the amount of missing data and the type of variable. For example, you won't impute the mean value for missing categorical data, you would choose the mode instead. No matter which method you choose, there will be some bias associated with it. One method which does a good job at reducing the bias associated with imputing missing values, is multiple imputation. It can be quite a long-winded approach but it is the most sound approach I've seen so far to imputing large amounts of missing values. I believe there may be an R library for multiple imputation.

Of course, another alternative may be that if variable x has 50% missing data for example, there may be a good explanation as to why this is. Rather than trying to impute it or lose the information associated with the variable, it can sometimes be useful to create a new variable, called variable_x_flag_missing. This would be a binary indicator variable where an observation is coded as 1 if variable x contains a missing value and coded as a 0 if it does not.

Answer 3

There is a difference between data with missing values and sparse data. Missing values are generally there because of invalid input, loss or error during data collection or are created when cleaning or processing data.

If these values are very less in number, the corresponding instances can be ignored or if are around 5-10% of the data, can be filled using various methods (carry forward last observation, fill with mean/median, interpolate the data etc). If you are working in Python, go through Pandas documentation for Working with Missing Values, to learn in detail about these options (even if you aren't working in Python, this is a good read).

But if your data set has a large number of missing values (say ~ >30% ), then the data is sparse. Such data sets create various bias in your modelling, and there are special ways to deal with them, though I don't about them much.

Answer 4

If the values are missing at random and you are sure that your data matrix is of low rank, you can use nuclear norm basis pursuit method (also known as matrix completion). The method (among others) is implemented in TFOCS.

In many real-world applications, the data matrix has rarely full rank, so the assumption of low-rank matrix can be acceptable. On the other hand, the values might not be missing truly at random.

Another approach would be to use Singular Spectrum Analysis (SSA), also known as the Caterpillar algorithm. It can be used for time-series data with missing values. This algorithm is not very well-known but in literature it is sometimes called "PCA for time-series data".

Answer 5

If missingness process could be assumed as MAR (Missing at Random) I strongly suggest multiple imputation.

The idea of multiple imputation for missing data was proposed by Rubin in 1977.

The idea is attractive because enable to separate the imputation and the analysis steps.

• The first step of multiple imputation for missing data is to multiply impute the missing values by using an appropriate model, the imputation model which mostly depends on the type of variable at hands.
• The second step is to separately analyze the imputed datasets.
• The third step is to combine the estimates of the analysis model in each imputed dataset.

This allows to have robust estimates.

I am able only to perform it on R with the mice package.