1
$\begingroup$

I am new to ML. I found that one of the preprocessing steps is to handle missing data.

My query is

  • Is there a way to understand nature of missing data
  • I can see that the mostly missing data is dropped or replaced with mean/mode is that the right way to go about it
$\endgroup$
1
  • 1
    $\begingroup$ For your second question: yes, some people do this but it is essentially wrong except for very specific cases addressed in @Archana David's answer. This is because complete case analysis often causes bias and reduces power, while mean/median imputation can cause bias due to underestimating the variance. Please do not do this. $\endgroup$ – Aolon Jul 22 at 5:49
1
$\begingroup$

Looks like your query is a 3 part question

  • How missing data occur: Missing data can occur when no data value is stored for a variable, or because of nonresponse, or the information is just not available

  • Understanding missingness: forms of missingness take different types

    1. MCAR(missing completely at random): Missing data values do not relate to any other data in the dataset and there is no pattern to the actual values of the missing data themselves.
    2. MAR(missing at random): Missing data do have a relationship with other variables in the dataset. However, the actual values that are missing are random.
    3. MNAR(missing not at random): The pattern of missingness is related to other variables in the dataset, but in addition, the values of the missing data are not random.
  • Handling missing data: Once you have understood what missingness is all about. Next step is to handle them, dropping missing data or replacing with mean/mode is few of the techniques.

    1. Deletion methods
      • Listwise deletion: ideal for MCAR
      • Pairwise deletion: ideal for MAR or MCAR
    2. Single imputation:
      • Mean/Median/Mode substitution
      • Regression imputation
      • LOCF(Last observation carried forward)
    3. Model-Based methods:
      • Maximum likelihood: best maximum likelihood technique is EM (Expectation-Maximization)
      • Multiple Imputation: MICE algorithm, Amelia(ideal for time series) are few packages that handle multiple imputation.

Look into these for better understanding of your missingness

$\endgroup$
1
$\begingroup$

Pick up a copy of the Bad Data Handbook (Q. Ethan McCallum, 2012).

It's got 18 case studies by different authors on their experience with dealing with pre-processing in the wild. Missing data is not your worst case scenario. Bad data can lead you down the wrong path if you don't sanity-check your results early in the process. It's a very enlightening read on how things can go wrong and the authors' approaches to wrangling data into something usable and useful.

... wonders if all data is bad data to start with ...

$\endgroup$
0
$\begingroup$

I can see that the mostly missing data is dropped or replaced with mean/mode is that the right way to go about it

It's mostly because you might be referring to a tutorial/blog which is not focused on that. Or, data volume is quite high as compared to missing rows. So, might have been ignored.

Is there a way to understand the nature of missing data

Yes, few good reads -

$\endgroup$
1
  • 1
    $\begingroup$ For understanding missing data, I would recommend Stef van Buuren's book: Flexible imputation of missing data. You can read it online on the author's website in full. $\endgroup$ – Aolon Jul 22 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy