1
$\begingroup$

I think the question is self-explanatory. But let's say you have a data with a few features with categorical data, and when building a model for example XGBoost you one-hot encode categorical features. Now you want to do prediction based on test data using the saved model. Obviously the test data needs to be one-hot encoded and need have similar features as training set. The question is whether it is possible to find a way not one-hot encode the test data and directly use it for prediction? Would this be somehow possible?

To me it appears that whatever comes in to my saved model need to be as it was used during training i.e. one-hot encoded features! But this is not neat, especially when building widgets and dashboards!

Any comments/hints are appreciated.

$\endgroup$
1
$\begingroup$

Since its pretty old post, possibly this response is helpful for others.

Its true that some of the algo's accept data in Categorical format and internally converts into OneHotEncoding. In such cases, model accept the data in raw format and doesnt require any explicit conversion handling.

In case if it's not supported, we have to save both the models, i.e.

  1. Model used for Encoding the data
  2. Model used for predicting the data

In a simpler way we can save related models in a single file as well. Refer code snippet below:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import OneHotEncoder
import pickle as pk
import pandas as pd
import numpy as np

df = pd.read_csv(<"Some_file.csv">) #replace with actual csv file
X = df['Features']
y = df['Labels']
file = open("models.pkl", "wb")  

encoder = OneHotEncoder(sparse=False)
oneHotEncodedFeature = encoder.fit_transform(X[<'Categorical_feature'>].values.reshape(-1,1))
pk.dump(encoder, file) #dumping Encoder model

# Some processing for concatenating oneHotEncodedFeature with other features and assume it its X again.
linReg = LinearRegression()
linReg.fit(X,y)
pk.dump(linReg, file) #dumping linear Reg. model
file.close() #Create single pickle file, which has both the trained model.

#For prediction

file = open("models.pkl", "rb")
trained_encoder = pk.load(file)  #Pickle file first load the OneHotEncoder 
trained_model_for_prediction = pk.load(file) #Reading same pickle again will load the trained Linear Reg Model.
$\endgroup$
2
  • $\begingroup$ Thanks for the answer. I know it has been long to get back to this question, but wondering if you can shed more light what happens here. You first create a "models.pkl" that you can write into . Then dump the encoder into it first, then here pk.dump(encoder,linReg) dumps the fitted model again into a same file (in a serial fashion, how? This part is a bit puzzling for me. At the here open("model.pkl", "rb") you open the file i.e. model (name doesn't match though "model.pkl", it is a typo right?), and once loaded you call for prediction have two models called on sequentially and encoding too? $\endgroup$ Apr 4 '20 at 8:31
  • $\begingroup$ My bad. Fixed some typo's . But yes single pickle file can hold data of multiple model in serialized manner. As Im doing here, first saving OneHotEncoder and then LinearReg Model. Similarly, when we call load twice, it will first load OneHotEncoder and then LinearReg. Like normal File reading operation. Hoping able to answer your question. $\endgroup$ Apr 9 '20 at 9:38
3
$\begingroup$

A model is built on a specific set of features, which may include categorical features encoded using one-hot encoding. If you have new data with additional categories, your model has no idea how to interpret the significance of those categories. You should either map the new value to none of the 1-hot values identified in training, or to an 'other' value.

For example, say you trained on data that had color=[blue,green]. Your one-hot fields would have color_blue and color_green. You could also have a field called color=other, that you might use to encode very infrequent values. That's a data preparation choice. So for 'red', you could encode that as either:

  • color_green = 0
  • color_blue = 0

or

  • color_green = 0
  • color_blue = 0
  • color_other = 1

Using either of these techniques will work with xgboost, but as xgboost only accepts numeric inputs, you will have to choose one of these methods as a data pre-processing step.

$\endgroup$
3
  • $\begingroup$ Thanks Tom for your time and explanation. This part I am fully aware of and I am already using "concatenating train and test for one-hot encoding" in my model. That was not my question though. Perhaps I did not ask properly. The question was about the next step. After training, when model is deployed/saved (pickle, flask etc.) for future use, is it possible somehow to use original test features for predictions? Meaning that somehow have that the one-hot encoding step built-in in the saved model! Or the saved model ONLY understands/sees the one-hot encoded features? $\endgroup$ Jan 19 '18 at 7:00
  • $\begingroup$ I think for xgboost specifically, the saved model only handles 1-hot encoded features, so you have to do those transformations manually first. Some models (or rather, particular implementations of some models) handle categorical variables without needing explicit pre-processing. Many of the models built into Weka (en.wikipedia.org/wiki/Weka_(machine_learning)) accept numeric and categorical features without preprocessing, for example. $\endgroup$
    – tom
    Jan 19 '18 at 14:59
  • $\begingroup$ Once again thanks. You are right. Since you mentioned Weka, I would like to mention CatBoost by Yandex (github.com/catboost/catboost), while it is quite fresh and community is small but it works like a charm compared to XGBoost especially when dealing with Categorical Features. It automatically account for them, and default parameters are already better than XGBoost. I am still experimenting but looks promising I'd say. $\endgroup$ Jan 20 '18 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.