I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing values with some estimate like the mean or median? What is the impact of these approaches on data analysis? Is there any other technique I can use to deal with missing data?

Mainly they are shipping data for chemical products, I'll give you three:





1 Answer 1


From personal experience, I would say a good first step would be to try to understand why there's data missing in the first place.

There are scenarios in which understanding this could help you figure out an approach to deal with it. For instance, I've dealt with datasets in the past where missing values simply represented 0, which allowed me to fill these fields with accurate values. It might also help you answer questions like: "will removing these values introduce selection bias?".

After that there a many techniques you could consider. Like you mentioned, you could

  • remove the data (if the data remaining would still be a good representation of the population, and you are left with enough rows to analyze)
  • fill the fields through a simple metric like the mean of median, though the quality of your dataset will suffer (especially if you're talking about a significant amount of rows)

There are also more sophisticated techniques like:

  • developing a separate model to predict the missing values
  • many more

In the end, what you choose as method depends heavily on your situation and dataset, but some good investigation into your data (if the scope of your project allows) to help you understand why these values are missing would be a good first step (based on my personal experience)

  • $\begingroup$ In fact, each data (line of different data) has a maximum of 10 attributes (or columns), adding up to a total of about 4 million different data. It's a lot of data numerically speaking, but I've never done that, so I thought it was strange that 4 million data is only 4 GB, you know? $\endgroup$
    – Horbeam
    May 1, 2023 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.