I have a regression model that I want to make prediction based on values that I will get from an end user.
In my dataset, I have one categorical variable region which I one-hot encoded, which generated 53 new columns (54 regions).
Now my data has the shape 1000x72. I then split into training and testing sets and my model is working fine.
But I'm confused about how my model would predict new values. Since I will only be getting one value for region from the end user, my model will one-hot encode a single value, and it will no longer fit the shape it has been trained on, as it will have the shape 1x18. I'm really confused as in how would I fit it into the model this way... Do I just make 53 other columns and put a dummy 0 in each one??
Sorry if this is a trivial question, I'm very beginner to this and any help would be greatly appreciated!!

region_ohe = OneHotEncoder(categories = "auto", handle_unknown = "ignore")
X_encoded = region_ohe.fit_transform(df['region'].values.reshape(-1,1)).toarray()
  • $\begingroup$ You have to re-train your network on the data using one-hot encoding where appropriate. When one-hot encoding, you have to view each new column you have created as a legitimate feature that is separate from all other one-hot encoded features. If you do not teach your regression model how to apply the proper coefficients to these new columns, how can you expect it to use this data in the future? $\endgroup$ – stefanLopez Jul 24 '19 at 20:06

With sklearn's OneHotEncoder, the categories are baked in after fitting. You can apply the encoding to new data with region_ohe.transform(x_new). (And, as you might guess, fit_transform just calls fit then transform.)


Say you have a column with numerical regions:


One hot encoding (aka „dummys“ or indicators) gives:

r1 r2 r3
1  0  0
0  1  0
0  0  1

Read the docs for Pandas: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html

Needless to say that your trained model need to see the same data structure (viz. variables or features) as the data you want to predict.

If trained on one hot, you just need to set all other region values to zero to make predictions based on user supplied input for a region.

  • 1
    $\begingroup$ Thanks @Peter! Ya I wasn't sure and thanks for confirming that I would indeed have to make 53 columns and set them to 0 every time I want to predict a new value. $\endgroup$ – IngridX Jul 24 '19 at 20:33
  • $\begingroup$ Is there an easy way to map user's input to the right column? Say I have columns region_0 to region_54 (I'd much rather have the actual name of the region as column names but onehotencoder automatically name the columns as numbers), and the user inputs region_12 for the region field. I would have to create 11 columns of 0, concat the region_12 column of value 1, and concat another 42 columns of 0. This is of course the dumbest way of doing it. Do you know any better way? $\endgroup$ – IngridX Jul 24 '19 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.