I read up some books on missing values. They have mentioned that listwise deletion is the least preferred method even though the sample size maybe be large (Newman, D. A. 2014. Missing Data: Five Practical Guide-lines.Organizational Research Methods 17: 372–411)

enter image description here

Next, I went to Kaggle to checkout how the missing value is handled for this data set https://www.kaggle.com/c/house-prices-advanced-regression-techniques. However the top rated code handle it by totally removing it. Some other top code use mean, mode etc. but so far I have not found any use Multiple Imputation, Maximum Likelihood etc. Moreover, the missingness (MCAR / MAR) was not determined.

Hence, I am confused in which situation do we need to go into details to handle missing value. What is the industry standard approach?


The best technique to handle missing data comes from understanding your data better and differs from case to case

Step 1: Do a exploratory data analysis along with your missing data. Python package missingno helps to Visualize data with all missing values. Here's the python package link

Click here for missingno youtube demo

Step 2: Do analyze the nature of missingness to help you better understand how to handle missing data. Based on the reasons of missing data, there are three types of missing data

  • MCAR(missing completely at random): Missing data values do not relate to any other data in the dataset and there is no pattern to the actual values of the missing data themselves.
  • MAR(missing at random): Missing data do have a relationship with other variables in the dataset. However, the actual values that are missing are random.
  • MNAR(missing not at random): The pattern of missingness is related to other variables in the dataset, but in addition, the values of the missing data are not random.

Step 3: Utilize right techniques to handle missing data

  1. Deletion methods
    • Listwise deletion: ideal for MCAR
    • Pairwise deletion: ideal for MAR or MCAR
  2. Single imputation:
    • Mean/Median/Mode substitution
    • Regression imputation
    • LOCF(Last observation carried forward)
  3. Model-Based methods:
    • Maximum likelihood: best maximum likelihood technique is EM (Expectation-Maximization)
    • Multiple Imputation: MICE algorithm, Amelia(ideal for time series) are few packages that handle multiple imputation.

Look into these for better understanding of your missingness


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.