4
$\begingroup$

I have a dataset of 70 columns that have missing values. Each column has a few columns (3-5) that it is significantly more correlated than the others but each column's correlated columns are very different from other columns. I would like to perform a tweaked version of KNN imputation and before I start writing this from scratch, I'd like to know if there's something similar out there so I don't go about reinventing the wheel.

On initialisation, a correlation matrix is obtained between all the variables. For every missing cell that needs to be imputed, the algorithm retrieves from the correlation matrix the top n highest correlated columns and uses only those n columns to select the k nearest neighbours and use the mean value of those neighbours.

Thanks and apologies if I am not very clear with my question. I'll try to clarify if you have any doubts.

$\endgroup$
  • $\begingroup$ Unfortunately I am not aware of such a package already existing but I think it is an interesting idea. If computationally it is not to heavy maybe you can also test to calculate the distance by weighting them with their correlation. $\endgroup$ – Viktor Jun 21 '19 at 17:24
  • $\begingroup$ Use a for-loop on each column - 1. to select your N Columns and 2. use sklearn.impute.KNNImputer as suggested by @Brian to predict and fill the missing values $\endgroup$ – 10xAI May 22 at 11:27
2
$\begingroup$

You can extend scikit-learn's sklearn.impute.KNNImputer. The metric parameter can accept the custom function you described.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.