0
$\begingroup$ 
  x1 x2 x3 x4 - - - - - - x10 .... x21 x22 x23
 1                                           |
 .         Complete                          |       
 .           data                            |
 88                                          |
 89                                          |
 90 ------------------------------------------
 .          |              |
 .complete  |   missing    |  complete
 .          |              |
 .          |              |
 . -------------------------------------------
 100
 101           Complete data
 102------------------------------------------

The dataset looks like this. 10 % of the data is missing. It doesn't appear missing at random.

From row 90 to 99. Variables x4 to x10 are missing. All other rows do not have any missing values. It is not missing at random. Is there any statistical way to investigate why they are missing.

My initial plan is to create a new column, 0 is not missing, 1 is missing. Run a logistic regression on non-missing columns. Is this the correct way to do it or not? 

 My questions:
     How should I investigate why data is missing by just playing around with the data set?

     If I found out the reasons or data is not missing not random, What should I do next?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Add the binary column but start with summary statistics by group (missing/not missing) for all of the x’s. That may reveal an implied ‘why’, eg the values are missing whenever some other x is always above/below some value.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Once I created a new column, I should group the data by missing and not missing. Then, What kinds of information should I look for? Could you be more specific? $\endgroup$
    – Mr. Z
    Sep 30, 2018 at 16:08
  • $\begingroup$ Summary statistics: mean, max, min, variance. Also potentially the most frequently occurring values. $\endgroup$
    – Chris Umphlett
    Sep 30, 2018 at 19:30
  • $\begingroup$ Just a follow up, if mean, variance, max, min are approximate the same. Then it means that missing values are not related to other complete columns. If mean, variance, max, min are different. Then we missing values are kinda influential. $\endgroup$
    – Mr. Z
    Sep 30, 2018 at 20:04
  • $\begingroup$ Perhaps. More simply I’m trying to assess whether or not the omission of the missing values from any future analysis would bias the result. Let’s say you had data on students, and one of your most important x’s was missing for 90% of females. You would not want to extrapolate the results on males + 10% of females across the general population. The missing values could be influential or not, I’m interested in how similar they are to the rest of your data. $\endgroup$
    – Chris Umphlett
    Sep 30, 2018 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.