# What should you do with NaN Values?

I have a dataset with a number of NaN values in it. I believe ~13,000 rows are affected out of ~500,000, so about 2.6% of the dataset.

I know that I can remove these rows or impute values for them. In general, when is one method preferable to the other, and which is best in my specific scenario?

Before you do anything, it's good to assess which of the following three categories your NaN rows fall into:

1. Missing Completely at Random (MCAR),
2. Missing at Random (MAR),
3. Missing Not at Random (MNAR).

## MCAR

If they're MCAR that means the "missing data" (i.e., the data that would have been there if it had been entered correctly or received in the first place) has the same distribution of values as the non-missing data. For example, if there's a categorical column with four categories A, B, C, D, which have a combined ratio a:b:c:d in the non-missing data, then any missing data will have the same ratio a:b:c:d. In this case it's possible to impute the missing values using the distribution of the non-missing values.

## MAR

If the missing data is MAR then there is a possibility that the distribution of the missing data and the non-missing data is different. This can occur, for example, in medical tests, where a test that is often performed on older people, who get worse test results, is less likely to be performed on younger people, who get better test results. In this case, it's not possible to impute the missing values with the distribution of the non-missing values. However, if a variable in your model is largely responsible for the change in distribution, in this case age, you can stratify your data by that variable, so that within your chosen strata the distribution for the missing and non-missing data are basically the same. That means that within your strata you can still use imputation to predict the missing values.

## MNAR

Finally, if your data are MNAR then there is a possibility that the distribution of the missing data and the non-missing data is different, but your model doesn't contain another variable that can explain that difference and stratify your data. This situation can occur when the chance of recording a variables value is dependent on the value of that variable and that variable alone. For example, alcoholics may decline to answer a question on alcohol consumption out of embarrassment, whereas T-Totallers would have no issue answering. In this case, you have to come up with some sort of model that explains why the data might be missing and use that to impute values for the missing data.