I have a question on using sklearn pipelines to predict/classify data. I understand how to build a pipeline, train it with training data, test it with test data; but after that is where I get lost. How do I use the pipeline to predict values on new/unseen data?

I built a pipeline which transforms categorical data with a OneHotEncoder and another that transforms numerical data by scaling the columns with a StandardScaler. I then used FeatureUnion to combine the two pipelines and join a DecisionTreeClassifier() at the end, to classify the data. I fit the training data sucessfully with pipeline.fit(X_train,y_train) and thereafter predict with pipeline.predict(X_test). This all works fine.

Next I want to use the pipeline to predict classification of new/unseen data; but when I call pipeline.predict(X_unseen) I get a ValueError. The error relates to one of the categorical features of the data, which is the names of a cities. The pipeline does not seem to transform the unseen data.

Reading through the documentation on pipelines and several examples I understand that when .fit() and .predict is called the data is passed through the entire pipeline. If my understanding is correct then the pipeline.predict(X_unseen) should pass the new data through the pipeline, transforming it and then classifying it. However this does not seem to be the case. Can anyone tell me what I'm missing or misunderstanding? How do I use the pipeline I built to predict on new data?

  • $\begingroup$ Did you solve this? I am trying to do the same. I don't get an error when I run pipeline.predict(X_unseen) but my predictions are bloated and don't line up with X_test at all. $\endgroup$ – Neels May 28 '20 at 4:39

Could you share the code? It is kind of hard to debug with out the code

When you do pipeline.predict it uses the transform method for all the steps.

Might be that in the One Hot Encoding you have not specified how to handle the unknown.

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.