How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how to keep track of the new one hot encoded data with the original dataset?


I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes.

import numpy as np
import pandas as pd     
from category_encoders.one_hot import OneHotEncoder

pd.options.display.float_format = '{:.2f}'.format # to make legible

# make some data
df = pd.DataFrame({'a': ['aa','bb','cc']*2,
                   'b': [True, False] * 3,
                   'c': [1.0, 2.0] * 3})

cols_encoding = df.select_dtypes(include='object').columns
ohe = OneHotEncoder(cols=cols_encoding)
encoded = ohe.fit_transform(df) 

Note that you can change the way you handle unseen data with

handle_unknown: str

options are ‘error’, ‘return_nan’, ‘value’, and ‘indicator’. The default is ‘value’. Warning: if indicator is used, an extra column will be added in if the transform matrix has unknown categories. This can cause unexpected changes in dimension in some cases.

  • 1
    $\begingroup$ I would add here that is critical to avoid overfitting to perform this step (fitting the encoder) only on the training data. This will also add a nice test whether the encoding really works with unseen categories as there may be some categories present only in the test data. $\endgroup$
    – Fnguyen
    Jul 13 '20 at 8:28
  • 1
    $\begingroup$ Even though I agree, using it as a transformer in a pipeline would be the right way to evaluate this, the OP asked about how to encode data not how to evaluate the encoding. I am not sure if the OP is interested in validation. It's a great suggestion thought! $\endgroup$ Jul 13 '20 at 12:23
  • $\begingroup$ I need to like to ask more thing though, if I have a dataset having 2 columns of categorical features ('M','F') and ("Week days"). I can convert the whole dataset using the same code. I want to pass a list through a function, which will return me the encoded list. How can we approach to that? I know here dictionary play a crucial role, keeping track of the values that what changes to what.. $\endgroup$ Jul 14 '20 at 9:42
  • $\begingroup$ I am not sure I understand your question right, feel free to make a separate question, and tag me. By the looks of it, it seems like a question more of stack overflow $\endgroup$ Jul 14 '20 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.