1
$\begingroup$

When I use a linear or random forest regression to train from data, I can make predictions with test data. I have a problem when I introduce the "impute" data widget in my pipeline to fill in the missing values from the training data. My predictions then are all the same for all rows, even though the data table is correct. I tried to swap the impute widget for the pre-processor and I have the same behaviour. Below is my graph:

This works correctly:

Adding an impute widget and all my predictions become the same values:

$\endgroup$
  • $\begingroup$ Does connecting Test Data directly into Predictions widget work perhaps? $\endgroup$ – K3---rnc Nov 21 '16 at 18:57
  • $\begingroup$ Wow, this worked! Any reason why it shouldn't work with the impute widget in the test data? $\endgroup$ – alew3 Nov 21 '16 at 22:16
1
$\begingroup$

When imputing, certain characteristics of the data have to be recorded. E.g. when you impute missing data by mean value parameter $\mu$, you have to record that $\mu$ from training data in order to be able to reapply the exact same transformation on the test data.

Thus you only need to connect your training data to the Impute widget, while the testing data will be imputed with the same parameters when the two domains are matched (in Predictions).

If you connect testing data to its own Impute widget, it gets imputed with its own, testing-data-specific parameters, which is, as you have discovered, incorrect.

See also: Difference between fit and fit_transform in scikit_learn models?

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.