0
$\begingroup$

I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that specific column ? Replacing by zeros or the mode doesn't really make sense sometimes... When is it relevant to do so ?

$\endgroup$
2
  • $\begingroup$ Are the NaN random or is there some censoring in the data, so that NaN follow a certain pattern? $\endgroup$
    – Peter
    Commented Nov 29, 2021 at 16:11
  • $\begingroup$ What is your goal after that ? $\endgroup$ Commented Apr 21, 2023 at 20:48

3 Answers 3

1
$\begingroup$

There is no one size fits all. So you cannot assume that one technique will work the best for all the datasets.

That being said the goal of imputing missing values is to ensure that after imputation, the distribution of the column does not change. So if you have a feature that follows a left skewed distribution, then after imputation the distribution should not change much.

Following this logic use multiple imputation techniques to see which one retains the original distribution of the feature you are imputing the values for.

$\endgroup$
1
$\begingroup$

Mean is suitable when you have a Gaussian distribution of continuous data. Mode is suitable when your column has categorical data and one category is clearly more like to occur than others. Median is better when your data has outliers which can skew the mean. You can opt to remove rows with missing values if the numbers of rows is very small compared to the total number of rows. There are other techniques which can be useful depending on the situation like training a model to fill missing values, MICE (for missing at Random type data), KNNImputer and LOCF.

Alternatively, if you have a significant number of missing values, you can see how the results are different when you impute missing values and when you ignore rows with missing values.

$\endgroup$
0
$\begingroup$

You are right in saying that replacing with a simple mean, mode... is a common but unreliable imputation strategy in many cases. You have in scikit learn some utilities for imputation of missing values (have a look at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.impute) using for instance the knn imputer as an additional strategy.

Take into account you cannot assume your feature of interest follows a normal distribution, so instead you can actually apply a kernel density estimator to model such distribution, see here: http://scikit-learn.org/stable/modules/density.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.