I have a dataset of 70 columns that have missing values. Each column has a few columns (3-5) that it is significantly more correlated than the others but each column's correlated columns are very different from other columns. I would like to perform a tweaked version of KNN imputation and before I start writing this from scratch, I'd like to know if there's something similar out there so I don't go about reinventing the wheel.

On initialisation, a correlation matrix is obtained between all the variables. For every missing cell that needs to be imputed, the algorithm retrieves from the correlation matrix the top n highest correlated columns and uses only those n columns to select the k nearest neighbours and use the mean value of those neighbours.

Thanks and apologies if I am not very clear with my question. I'll try to clarify if you have any doubts.

  • $\begingroup$ Unfortunately I am not aware of such a package already existing but I think it is an interesting idea. If computationally it is not to heavy maybe you can also test to calculate the distance by weighting them with their correlation. $\endgroup$ – Viktor Jun 21 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.