I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that specific column ? Replacing by zeros or the mode doesn't really make sense sometimes... When is it relevant to do so ?
2
-
$\begingroup$ Are the NaN random or is there some censoring in the data, so that NaN follow a certain pattern? $\endgroup$– PeterNov 29, 2021 at 16:11
-
$\begingroup$ What is your goal after that ? $\endgroup$– Lucas MorinApr 21 at 20:48
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You are right in saying that replacing with a simple mean, mode... is a common but unreliable imputation strategy in many cases. You have in scikit learn some utilities for imputation of missing values (have a look at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.impute) using for instance the knn imputer as an additional strategy.
Take into account you cannot assume your feature of interest follows a normal distribution, so instead you can actually apply a kernel density estimator to model such distribution, see here: http://scikit-learn.org/stable/modules/density.html
$\begingroup$
$\endgroup$
There is no one size fits all. So you cannot assume that one technique will work the best for all the datasets.
That being said the goal of imputing missing values is to ensure that after imputation, the distribution of the column does not change. So if you have a feature that follows a left skewed distribution, then after imputation the distribution should not change much.
Following this logic use multiple imputation techniques to see which one retains the original distribution of the feature you are imputing the values for.
