Missing Values in Data [duplicate]

Question

I have experienced that most of the datasets contain missing values, which make our task bit challenging.

Please let me know how to fill up those missing values in an efficient way? and is there any specific techniques to handle missing values?

Answer 1

Various methods are available for fill missing values in data.

1. Ignore the tuple is the simplest and not effective method.

2. Fill the missing value manually.

3. Use a global constant to fill the missing value.

4. Use attribute mean value to fill missing value.

5. Use attribute mean for all samples belonging to the same class as the given tuple.

6. Use most probable value to fill in the missing value (this may be determined with regression, inference tool, or decision tree induction)

Reference:

Data Mining – Concepts and Techniques - JIAWEI HAN & MICHELINE KAMBER, ELSEVIER, 2nd Edition.

Answer 2

First of all, if most of your data is missing, you are in trouble anyway. You need to ask why is most of the data missing, and also, why are the data you observed not missing. Being missing is very likely telling you something in your data.

All methods of correcting missing data, including the naive interpolation, mean replacement, and median replacement methods, assume that you can largely ignore the reason for the data being missing - These are the Missing at Random [MAR], and the (much stronger) Missing Completely at Random [MCAR], assumptions. If one, or both, of these are not true, which is very likely if most of your data are missing, then no asymptotically reliable method of imputation is known to exist. This doesn't mean you can do nothing - see here for some suggestions.

In most circumstances, with MAR data, people use model based methods. Essentially these impute a dataset many times, filling in the missing values using plausible models of the missing data, and then run analyses on the ensemble of imputed datasets.

I've usually used the mice package in r. A paper is here. A useful webpage is at web.maths.unsw.edu.au/~dwarton/missingDataLab.html.

If the process which leads to your missing data is not ignorable, in other words, if being missing tells you something, none of this will work. It will produce nice looking numbers, but no-one, you included, will have any idea what they mean.

Answer 3

Yes there are so many approaches to handle missing data or missing values depending on the task and the property of the data itself. For example in time series you can think about forward filling or even backward filling, max, mean or median over a time lapse. There are also 'smarter' ways like training a model over the available data and try to predict the missing ones. The latest requires much data so can give efficient results. You can refer to these two links for more information.

1. How to Handle Missing Data with Python
2. Machine learning algorithms to handle missing data

Answer 4

One of the most common way to fill up the missing values is using linear interpolation : The previous valid values before the missing value and the following valid values after the missing value are used for the interpolation.

For example, in Python-Numpy package, you can use numpy.interpfor filling in missing values using linear interpolation. Here is an example for a simple array:

>> import numpy as np
>> data=np.array([1,2,np.nan,6,9,np.nan,15])
>> print(data)
[  1.   2.  nan   6.   9.  nan  15.]
>> nans, xf= np.isnan(data), lambda z: z.nonzero()[0]
>> data[nans]= np.interp(xf(nans), xf(~nans), data[~nans])
>> print(data)
[  1.   2.   4.   6.   9.  12.  15.]