# Visualize/analyze data before or after imputing missing values?

My understanding is that we impute missing values in order to preserve those training examples so our Machine Learning algorithms have as many training examples as possible.

To me, it would make intuitive sense to visualize/analyze the data before imputing the missing values, as imputation will skew distributions and may lead to false assumptions about the real data before imputation. I can see a case for doing both before and after, but that adds time to the feature analysis. On the other hand, I can see a case for not imputing values if the % of missing values is high enough to affect the data's distribution.

In Python I imagine using something along the lines of pd.Series.dropna() to isolate the existing values.

TL;DR: Should one impute missing values before or after visualizing the data and pulling insights from it?

Why not do both? Like you mention, it might be worth first computing the percentage of all values that are values. Generally you might also have a percentage in mind that is acceptable, like up to 10% missing values, if they are scattered at random throughout your dataset.

There are libraries built specifically for visualising missing data, such as missingno, which offers quite a few ideas. Here is an example heatmap of missing variables across features:

"missing" normally implies you have a sequential dataset, for example time-series data. If you had discrete observations e.g. of people's height versus shoe size, there is no sequential causality (autocorrelation: dependency on previous values). In this case, imputing makes little sense.

So assuming you do have sequential data, whether or not to impute or drop time steps with missing values will really depend on your use case. Also perhaps the frequency of the data. If all the missing vaiues appear in one chunk at either end of the time series, it is cokmmon to simply leave out that chunk.

For example, if you have minute frequency data and you wish to predict a value once per day, then missing a few minutes here and there might be tolerable, and imputation of some kind (e.g. fill-forwad) wouldn't have a huge impact overall, but could help the model optimisation work more effectively. Some models cannot handle missing values, so imputation is necessary.

In any case, it would always visualise the data before and after imputation. You can usually run the same visualisation anyway. Sure it costs a few extra minutes, but you might catch important issues. This can save a lot of time compared to only finding the issues later on while debugging your trained model.

• +1 for mentioning missingno. Didn't know it, and seems valuable. – 89f3a1c Aug 12 '19 at 15:11

I think you've done a good job identifying the trade-offs in this situation. If you impute missing values before visualization, then you won't be visualizing the "true" data. But sometimes a lot of data is missing, and if you drop all examples with missing attributes then you're unlikely to be visualizing a representative sample of the data you might use to train a model, or, worse, you might miss out on some important insights because so much data is missing.

Maybe there's a way you can get the best of both worlds? I would recommend imputing missing values before visualization, but marking them visually. For example, you might generate a plot where examples with no missing data are colored green, examples with one missing field are yellow, and examples with 2+ missing fields are red.

To me, it would make intuitive sense to visualize/analyze the data before imputing the missing values, as imputation will skew distributions and may lead to false assumptions about the real data before imputation.

I think there's a crucial detail to answer here which might deepen your analysis a bit. How are you planning to imput the missing values? That might well depend on two things:

1. the actual distribution of the values. Depending on the distribution, different techniques may be appropriate for each case: most common value, mean, some machine learning algorithm to predict the missing values based on other data... But to decide which method to use, you have to first understand the raw data.
2. how many values are missing, which you covered later.

On the other hand, I can see a case for not imputing values if the % of missing values is high enough to affect the data's distribution.

That's correct. Think of it this way: if the percentage of missing values is too high, then you have no basis on which to accurately fill the missing values. You might just be inventing too much data. How will that play later? Are you adding something valuable, or are you putting in too much invention on your own just to save that one attribute?

Hope this helps!