1
$\begingroup$

I am working with a dataset having 45k rows and I was a bit confused on whether or not to drop the missing values OR impute the missing values.

Column wise missing value distribution :

missing val distribution

As per this answer: https://stackoverflow.com/a/28199556/12298398), I calculated the number of rows containing missing values

>>> np.count_nonzero(df.isnull().values.ravel())
2057

But now I am a bit confused on whether or not I should drop these rows containing missing values since dropping them will cost a loss of data or I should impute those columns which have missing values greater than 500.

Let me know your thoughts on the same, Thank you

$\endgroup$
2
  • $\begingroup$ Hi, This would depend on the task you are trying to solve with this dataset, could provide us with a bit more context, so we can give you more adapted answers ? $\endgroup$ Nov 26, 2021 at 9:23
  • $\begingroup$ It is a part of the ongoing competition so I can't tell much about the problem. So that's why I asked in general what should be the approach but I get that it depends upon the problem so I'll try experimenting with it for this problem, Thank you $\endgroup$ Nov 27, 2021 at 8:48

1 Answer 1

2
$\begingroup$

In most cases, dropping data only makes sense when you have a large number of nan values. For example of you have a feature with 98% nan values, it is not going to be of much use to any algorithm. Also imputing that feature is not going to work as you don't have much data to go on with.

But if there are reasonable number of nan values, then the best option is to try to impute them. There are 2 ways you can impute nan values:-

1. Univariate Imputation: You use the feature itself that has nan values to impute the nan values. Techniques include mean/median/mode imputation, although it is advised not to use these techniques as they distort the distribution of the feature. Other techniques might include creating a new feature to capture the missingness of that feature. You should Google this topic as there are literally hundreds of articles and blogs.

2. Multivariate Imputation: As the name suggests, you use multiple columns to impute nan values in a specific feature/column. This method is the most preferred as it results in better imputation results than Univariate Imputation. Some of the most used techniques are KNNImputer and IterativeImputer. Again Google is your best friend!

Bottom line being, only drop nan values when your feature has a majority if it's values as nan. If not, it's usually better to impute.

Cheers!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.