Drop or impute the missing values?

Question

I am working with a dataset having 45k rows and I was a bit confused on whether or not to drop the missing values OR impute the missing values.

Column wise missing value distribution :

As per this answer: https://stackoverflow.com/a/28199556/12298398), I calculated the number of rows containing missing values

>>> np.count_nonzero(df.isnull().values.ravel())
2057

But now I am a bit confused on whether or not I should drop these rows containing missing values since dropping them will cost a loss of data or I should impute those columns which have missing values greater than 500.

Let me know your thoughts on the same, Thank you

Answer 1

In most cases, dropping data only makes sense when you have a large number of nan values. For example of you have a feature with 98% nan values, it is not going to be of much use to any algorithm. Also imputing that feature is not going to work as you don't have much data to go on with.

But if there are reasonable number of nan values, then the best option is to try to impute them. There are 2 ways you can impute nan values:-

1. Univariate Imputation: You use the feature itself that has nan values to impute the nan values. Techniques include mean/median/mode imputation, although it is advised not to use these techniques as they distort the distribution of the feature. Other techniques might include creating a new feature to capture the missingness of that feature. You should Google this topic as there are literally hundreds of articles and blogs.

2. Multivariate Imputation: As the name suggests, you use multiple columns to impute nan values in a specific feature/column. This method is the most preferred as it results in better imputation results than Univariate Imputation. Some of the most used techniques are KNNImputer and IterativeImputer. Again Google is your best friend!

Bottom line being, only drop nan values when your feature has a majority if it's values as nan. If not, it's usually better to impute.

Cheers!