6
$\begingroup$

What are all the options available for filling in missing data?

One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy.

So how do we deal with missing values if they are are lot of them?

$\endgroup$
  • $\begingroup$ In addition to the answers below, the correct way of imputation depends on how you intend to use it. Is it a target variable, an explanatory variable, or a classification variable? $\endgroup$ – Paul Oct 6 '15 at 16:57
  • $\begingroup$ This is a statistics question - you'll get better response on the statistics stackexchange site: stats.stackexchange.com/search?q=missing+values $\endgroup$ – Spacedman Oct 7 '15 at 6:55
  • $\begingroup$ Along with many other answers on the topic on Cross Validated, you might find my related answer and links within useful. $\endgroup$ – Aleksandr Blekh Oct 10 '15 at 9:43
4
$\begingroup$

There are of course other choices to fill in for missing data. The median was already mentioned, and it may work better in certain cases.

There may even be much better alternatives, which may be very specific to your problem. To find out whether this is the case, you must find out more about the nature of your missing data. When you understand in detail why data is missing, the probability of coming up with a good solution will be much higher.

You might want to start your investigation of missing data by finding out whether you have informative or non-informative missings. The first category is produced by random data loss; in this case, the observations with missing values are no different from the ones with complete data. As for informative missing data, this one tells you something about your observation. A simple example is a customer record with a missing contract cancellation date meaning that this customer's contract has not been cancelled so far. You usually don't want to fill in informative missings with a mean or a median, but you may want to generate a separate feature from them.

You may also find out that there are several kinds of missing data, being produced by different mechanisms. In this case, you might want to produce default values in different ways.

$\endgroup$
  • $\begingroup$ Do you know any good resource on this topic ? Also how do we know that we have to use mean or median ? $\endgroup$ – mach Oct 6 '15 at 13:57
  • $\begingroup$ Yould read "Missing Values Analysis and Data Imputation" by David Garson for a basic introduction. Beyond the basics, everything gets very model-specific. Maybe you could add some details about what you want to do with the data. Also don't get too caught up with mean vs. median -- these are just two very simple examples of a broad range of possibilities, including those that use data from other features as well, not just the one with the missing data. Also the decision of mean vs. median will depend very much on your specific problem. $\endgroup$ – MightyCurious Oct 6 '15 at 14:18
2
$\begingroup$

When it comes to missing data, there are many different methods of filling these values. However, the imputation method you choose, depends largely on the amount of missing data and the type of variable. For example, you won't impute the mean value for missing categorical data, you would choose the mode instead. No matter which method you choose, there will be some bias associated with it. One method which does a good job at reducing the bias associated with imputing missing values, is multiple imputation. It can be quite a long-winded approach but it is the most sound approach I've seen so far to imputing large amounts of missing values. I believe there may be an R library for multiple imputation.

Of course, another alternative may be that if variable x has 50% missing data for example, there may be a good explanation as to why this is. Rather than trying to impute it or lose the information associated with the variable, it can sometimes be useful to create a new variable, called variable_x_flag_missing. This would be a binary indicator variable where an observation is coded as 1 if variable x contains a missing value and coded as a 0 if it does not.

$\endgroup$
1
$\begingroup$

If the values are missing at random and you are sure that your data matrix is of low rank, you can use nuclear norm basis pursuit method (also known as matrix completion). The method (among others) is implemented in TFOCS.

In many real-world applications, the data matrix has rarely full rank, so the assumption of low-rank matrix can be acceptable. On the other hand, the values might not be missing truly at random.

Another approach would be to use Singular Spectrum Analysis (SSA), also known as the Caterpillar algorithm. It can be used for time-series data with missing values. This algorithm is not very well-known but in literature it is sometimes called "PCA for time-series data".

$\endgroup$
1
$\begingroup$

There is a difference between data with missing values and sparse data. Missing values are generally there because of invalid input, loss or error during data collection or are created when cleaning or processing data.

If these values are very less in number, the corresponding instances can be ignored or if are around 5-10% of the data, can be filled using various methods (carry forward last observation, fill with mean/median, interpolate the data etc). If you are working in Python, go through Pandas documentation for Working with Missing Values, to learn in detail about these options (even if you aren't working in Python, this is a good read).

But if your data set has a large number of missing values (say ~ >30% ), then the data is sparse. Such data sets create various bias in your modelling, and there are special ways to deal with them, though I don't about them much.

$\endgroup$
0
$\begingroup$

If missingness process could be assumed as MAR (Missing at Random) I strongly suggest multiple imputation.

The idea of multiple imputation for missing data was proposed by Rubin in 1977.

The idea is attractive because enable to separate the imputation and the analysis steps.

  • The first step of multiple imputation for missing data is to multiply impute the missing values by using an appropriate model, the imputation model which mostly depends on the type of variable at hands.
  • The second step is to separately analyze the imputed datasets.
  • The third step is to combine the estimates of the analysis model in each imputed dataset.

This allows to have robust estimates.

I am able only to perform it on R with the mice package.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.