How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how to keep track of the new one hot encoded data with the original dataset?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes.
import numpy as np
import pandas as pd
from category_encoders.one_hot import OneHotEncoder
pd.options.display.float_format = '{:.2f}'.format # to make legible
# make some data
df = pd.DataFrame({'a': ['aa','bb','cc']*2,
'b': [True, False] * 3,
'c': [1.0, 2.0] * 3})
cols_encoding = df.select_dtypes(include='object').columns
ohe = OneHotEncoder(cols=cols_encoding)
encoded = ohe.fit_transform(df)
Note that you can change the way you handle unseen data with
handle_unknown: str
options are ‘error’, ‘return_nan’, ‘value’, and ‘indicator’. The default is ‘value’. Warning: if indicator is used, an extra column will be added in if the transform matrix has unknown categories. This can cause unexpected changes in dimension in some cases.
answered Jul 10, 2020 at 4:20
-
1$\begingroup$ I would add here that is critical to avoid overfitting to perform this step (fitting the encoder) only on the training data. This will also add a nice test whether the encoding really works with unseen categories as there may be some categories present only in the test data. $\endgroup$– FnguyenCommented Jul 13, 2020 at 8:28
-
1$\begingroup$ Even though I agree, using it as a transformer in a pipeline would be the right way to evaluate this, the OP asked about how to encode data not how to evaluate the encoding. I am not sure if the OP is interested in validation. It's a great suggestion thought! $\endgroup$ Commented Jul 13, 2020 at 12:23
-
$\begingroup$ I need to like to ask more thing though, if I have a dataset having 2 columns of categorical features ('M','F') and ("Week days"). I can convert the whole dataset using the same code. I want to pass a list through a function, which will return me the encoded list. How can we approach to that? I know here dictionary play a crucial role, keeping track of the values that what changes to what.. $\endgroup$ Commented Jul 14, 2020 at 9:42
-
$\begingroup$ I am not sure I understand your question right, feel free to make a separate question, and tag me. By the looks of it, it seems like a question more of stack overflow $\endgroup$ Commented Jul 14, 2020 at 9:53