Correlation Analysis dealing with missing value, when the missing values are actually expected

Question

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here are my attributes:

Deal: Binary, Indicates if there was a successful deal or not.

Deal_amount: Float, The amount of the deal, only exists if there is a deal otherwise missing.

So, whenever there is no deal, the Deal_amount values are missing, as a novice, the first thing that came to my mind was to just replace the missing values with zero but I am afraid that might skew my data because whenever there is a deal the amount is generally very large (in Millions).

What would be the best way to handle such a case? I actually found a Similar Question in this platform,

Answer 1

The answer to your question will heavily rely on the kind of analysis you are going to run. Let's consider couple of examples.

1. You want to know the "correlation" between the Deal_amount and the mean size of counteragents companies. In this case you can simply drop observations, for which the transaction has not occur.
2. You want to know if the mean size of the counteragent companies affects the fact that the transaction will be succesfull (correlate with Deal). In this case, you do not have any missing values.
3. You want to know the correlation between Deal_amount and the season of the year. Again, you do not need to include missing values into the analysis, since the transaction has not occur.

In these scenarios you do not need to perform any imputation of Deal_amount. Now, there are possible scenarios were you do like to impute, e.g. a deal was almost done, but dropped at the last minute, and your goal is to estimate the market size. But again, the imputation strategy will heavily rely on a specific question you want to answer.

Answer 2

If you don't want to delete observations with missing cells, and you cannot ignore them, you may be able to model them statistically. You assume/estimate the probability distribution for the data, e.g. Deal is a Bernoulli variable with some probability and Deal_value is Gamma-distributed variable. You then fill the missing values by sampling from these probability distributions, and estimate the quantities you care about. Ideally you do this many times so that you get a distribution of your estimates. The statistical models for your data may also include correlation with other non-missing data-fields, if you want.

Another approach is to have a statistical model for the missingess, i.e. something that predicts whether the data will be missing for this observation. In which case you don't need to impute at all since missing values will become equivalent to normal observations.