x1 x2 x3 x4 - - - - - - x10 .... x21 x22 x23
 1                                           |
 .         Complete                          |       
 .           data                            |
 88                                          |
 89                                          |
 90 ------------------------------------------
 .          |              |
 .complete  |   missing    |  complete
 .          |              |
 .          |              |
 . -------------------------------------------
 101           Complete data

The dataset looks like this. 10 % of the data is missing. It doesn't appear missing at random.

From row 90 to 99. Variables x4 to x10 are missing. All other rows do not have any missing values. It is not missing at random. Is there any statistical way to investigate why they are missing.

My initial plan is to create a new column, 0 is not missing, 1 is missing. Run a logistic regression on non-missing columns. Is this the correct way to do it or not?

 My questions:
     How should I investigate why data is missing by just playing around with the data set?

     If I found out the reasons or data is not missing not random, What should I do next?

1 Answer 1


Add the binary column but start with summary statistics by group (missing/not missing) for all of the x’s. That may reveal an implied ‘why’, eg the values are missing whenever some other x is always above/below some value.

  • $\begingroup$ Once I created a new column, I should group the data by missing and not missing. Then, What kinds of information should I look for? Could you be more specific? $\endgroup$
    – Mr. Z
    Commented Sep 30, 2018 at 16:08
  • $\begingroup$ Summary statistics: mean, max, min, variance. Also potentially the most frequently occurring values. $\endgroup$ Commented Sep 30, 2018 at 19:30
  • $\begingroup$ Just a follow up, if mean, variance, max, min are approximate the same. Then it means that missing values are not related to other complete columns. If mean, variance, max, min are different. Then we missing values are kinda influential. $\endgroup$
    – Mr. Z
    Commented Sep 30, 2018 at 20:04
  • $\begingroup$ Perhaps. More simply I’m trying to assess whether or not the omission of the missing values from any future analysis would bias the result. Let’s say you had data on students, and one of your most important x’s was missing for 90% of females. You would not want to extrapolate the results on males + 10% of females across the general population. The missing values could be influential or not, I’m interested in how similar they are to the rest of your data. $\endgroup$ Commented Sep 30, 2018 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.