Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a dataset with a number of NaN values in it. I believe ~13,000 rows are affected out of ~500,000, so about 2.6% of the dataset.
I know that I can remove these rows or impute values for them. In general, when is one method preferable to the other, and which is best in my specific scenario?
Before you do anything, it's good to assess which of the following three categories your NaN rows fall into:
If they're MCAR that means the "missing data" (i.e., the data that would have been there if it had been entered correctly or received in the first place) has the same distribution of values as the non-missing data. For example, if there's a categorical column with four categories A, B, C, D, which have a combined ratio a:b:c:d in the non-missing data, then any missing data will have the same ratio a:b:c:d. In this case it's possible to impute the missing values using the distribution of the non-missing values.
If the missing data is MAR then there is a possibility that the distribution of the missing data and the non-missing data is different. This can occur, for example, in medical tests, where a test that is often performed on older people, who get worse test results, is less likely to be performed on younger people, who get better test results. In this case, it's not possible to impute the missing values with the distribution of the non-missing values. However, if a variable in your model is largely responsible for the change in distribution, in this case age, you can stratify your data by that variable, so that within your chosen strata the distribution for the missing and non-missing data are basically the same. That means that within your strata you can still use imputation to predict the missing values.
Finally, if your data are MNAR then there is a possibility that the distribution of the missing data and the non-missing data is different, but your model doesn't contain another variable that can explain that difference and stratify your data. This situation can occur when the chance of recording a variables value is dependent on the value of that variable and that variable alone. For example, alcoholics may decline to answer a question on alcohol consumption out of embarrassment, whereas T-Totallers would have no issue answering. In this case, you have to come up with some sort of model that explains why the data might be missing and use that to impute values for the missing data.
If you'd like to know more about the above three definitions read this article, and this web page: MNAR,
Once you know what category your missing data falls into, you can make a decision about whether or not to delete the rows or impute the data. In your specific case, I would find out if your data is either MCAR or MAR and, if it is, delete it. This data is known as "ignorable data" and, as it's only 2.6% of all data, its exclusion shouldn't make an appreciable difference to the effectiveness of your model.
If it's MNAR then you need to figure out what the pattern behind its exclusion is and use that to impute the data. For more information on the various methods of imputing data look at the following article from Towards Data Science.
