0
$\begingroup$

I'm currently trying to use fancyimpute KNN to impute some data separate train/test sets. However, I'm running into the problem of not being able to do fit_transform on train data then using only transform on the test set because it appears that:

transform not implemented! This imputation algorithm likely doesn't support inductive mode. Only fit_transform is supported at this time.

Is it alright to concat the two sets for imputing before splitting them after? Or is my approach even correct by calling only fit on the train set before applying that to impute the test set? Should I be calling separate fit_transform on both imputation?

Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ Hey Joshua, Welcome to our community. I am unable to understand your problem, could you try putting your code to illustrate what you are trying to do? $\endgroup$
    – Pedro Henrique Monforte
    Apr 11, 2019 at 20:03
  • $\begingroup$ fancyimpute.KNN is an imputer. I'm trying to call imputer.fit_transform on test set and imputer.transform on the train set but there's no transform method built in yet. I'm looking for a workaround by combining test and train set so I don't lose the fit. $\endgroup$
    – Joshua
    Apr 11, 2019 at 22:28

1 Answer 1

Reset to default
2
$\begingroup$

They way you tried to do it is correct and no, you should not concatenate your train and test set to get around this.

Unfortunately it seems that the transformer you are trying to use just does not implement the functionality you need. From the docs:

There are some imputation algorithms that are inductive, meaning they can be applied to new data after a call to solver.fit or solver.fit_transform. Currently only IterativeImputer supports the full scikit-learn API: fit, fit_transform, and transform, but we are actively looking for contributions that extend other imputers to support induction. At least the KNN and SimpleFill imputers can be extended in a straightforward manner.

Getting around this by concatenating your train and test set together will cause a data leak. This happens when your model gets information from the test set during training time, which invalidates the test evaluation.

And data leaks aside, what imputer will you use when you want to make predictions on new data?

As I see it your options now are:

  • Find another KNN imputation implementation that supports transform on new data
  • Use another inductive imputer
  • Implement KNN imputation yourself
Improve this answer
$\endgroup$
7
  • $\begingroup$ Do you by any chance know any alternatives or workarounds for what I'm trying to do? $\endgroup$
    – Joshua
    Apr 11, 2019 at 22:26
  • $\begingroup$ Updated my answer. I did a quick look around and there seem to be some implementations floating around but nothing really user friendly unfortunately. :/ $\endgroup$
    – Simon Larsson
    Apr 11, 2019 at 22:42
  • 1
    $\begingroup$ I think the easiest would be to just to use another imputer and do what you tried to do initially. $\endgroup$
    – Simon Larsson
    Apr 11, 2019 at 22:43
  • $\begingroup$ Would it be advisable to separately call fit_transform on the train and test sets since fit should not really matter for imputation? $\endgroup$
    – Joshua
    Apr 13, 2019 at 0:32
  • $\begingroup$ No that is not a good idea. Fit does matter for imputation. This will make the model bad or useless on new data. If it gets a single sample with a missing value fit_transform with fail since it will not have any neighbors to impute with. $\endgroup$
    – Simon Larsson
    Apr 13, 2019 at 5:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.