How to perform one hot encoding on multiple categorical columns

Question

I am trying to perform one-hot encoding on some categorical columns. From the tutorial I am following, I am supposed to do LabelEncoding before One hot encoding. I have successfully performed the labelencoding as shown below

#categorical data
categorical_cols = ['a', 'b', 'c', 'd'] 
from sklearn.preprocessing import LabelEncoder
# instantiate labelencoder object
le = LabelEncoder()
# apply le on categorical feature columns
data[categorical_cols] = data[categorical_cols].apply(lambda col: le.fit_transform(col))

Now I am stuck with how to perform one hot encoding and then join the encoded columns to the dataframe (data).

Please how do I do this?

Answer 1

LabelEncoder is not made to transform the data but the target (also known as labels) as explained here. If you want to encode the data you should use OrdinalEncoder.

If you really need to do it this way:

categorical_cols = ['a', 'b', 'c', 'd'] 

from sklearn.preprocessing import LabelEncoder
# instantiate labelencoder object
le = LabelEncoder()

# apply le on categorical feature columns
data[categorical_cols] = data[categorical_cols].apply(lambda col: le.fit_transform(col))    
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()

#One-hot-encode the categorical columns.
#Unfortunately outputs an array instead of dataframe.
array_hot_encoded = ohe.fit_transform(data[categorical_cols])

#Convert it to df
data_hot_encoded = pd.DataFrame(array_hot_encoded, index=data.index)

#Extract only the columns that didnt need to be encoded
data_other_cols = data.drop(columns=categorical_cols)

#Concatenate the two dataframes : 
data_out = pd.concat([data_hot_encoded, data_other_cols], axis=1)

Otherwise:

I suggest you to use pandas.get_dummies if you want to achieve one-hot-encoding from raw data (without having to use OrdinalEncoder before) :

#categorical data
categorical_cols = ['a', 'b', 'c', 'd'] 

#import pandas as pd
df = pd.get_dummies(data, columns = categorical_cols)

You can also use drop_first argument to remove one of the one-hot-encoded columns, as some models require.

Answer 2

You can do dummy encoding using Pandas in order to get one-hot encoding as shown below:

import pandas as pd

# Multiple categorical columns
categorical_cols = ['a', 'b', 'c', 'd']

pd.get_dummies(data, columns=categorical_cols)

If you want to do one-hot encoding using sklearn library, you can get it done as shown below:

from sklearn.preprocessing import OneHotEncoder
onehotencoder = OneHotEncoder()

transformed_data = onehotencoder.fit_transform(data[categorical_cols])

# the above transformed_data is an array so convert it to dataframe
encoded_data = pd.DataFrame(transformed_data, index=data.index)

# now concatenate the original data and the encoded data using pandas
concatenated_data = pd.concat([data, encoded_data], axis=1)

If a single column has more than 500 categories, the aforementioned way of one-hot encoding is not a good approach. In this case, we can do one-hot encoding for the top 10 or 20 categories that are occurring most for a particular column. A sample code is shown below:

categorical_cols = ['a', 'b', 'c', 'd']

# Let's say we have a column 'b' which has more than 500 categories.
# Find the top 10 most frequent categories for column 'b'
data.b.value_counts().sort_values(ascending = False).head(20)

# make a list of the most frequent categories of the column
top_10_occurring_cat = [cat for cat in data.b.value_counts().sort_values(ascending = False).head(10).index]

# now make the 10 binary variables
for cat in top_10_occurring_cat:
    data[cat] = np.where(data['b'] == cat, 1, 0) # whenever data['b'] == cat replace it with 1 else 0

# This is done for one categorical column, similarly you can repeat for all categorical columns

Answer 3

Create a Pandas DataFrame with multiple one-hot-encoded columns

Let's say you have a Pandas dataframe flags with many columns you want to one-hot-encode.

You want a Pandas dataframe flags_ohe, which has the same columns as flags, but columns 'Mainhue', 'Landmass','Zone','Language','Religion', 'Topleft', 'Botright' are replaced with one-hot-encoded versions with clear column names such as Mainhue_red and Mainhue_blue.

flags_ohe = flags
categorical_columns = ['Landmass','Zone','Language','Religion', 
                       'Mainhue', 'Topleft', 'Botright']
for col in categorical_columns:
    col_ohe = pd.get_dummies(flags[col], prefix=col)
    flags_ohe = pd.concat((flags_ohe, col_ohe), axis=1).drop(col, axis=1)

Here's before.

print(flags.columns)

# Output:
# Index(['Name', 'Landmass', 'Zone', 'Area', 'Population', 'Language',
#  'Religion', 'Bars', 'Stripes', 'Colors', 'Red', 'Green', 'Blue', 'Gold',
#  'White', 'Black', 'Orange', 'Mainhue', 'Circles', 'Crosses', 'Saltires',
#  'Quarters', 'Sunstars', 'Crescent', 'Triangle', 'Icon', 'Animate',
#  'Text', 'Topleft', 'Botright'],
# dtype='object')
# dtype='object')

Here's after.

print(flags_ohe.columns)

# Output:
# Index(['Name', 'Area', 'Population', 'Bars', 'Stripes', 'Colors', 'Red',
#  'Green', 'Blue', 'Gold', 'White', 'Black', 'Orange', 'Circles',
#  'Crosses', 'Saltires', 'Quarters', 'Sunstars', 'Crescent', 'Triangle',
#  'Icon', 'Animate', 'Text', 'Landmass_1', 'Landmass_2', 'Landmass_3',
#  'Landmass_4', 'Landmass_5', 'Landmass_6', 'Zone_1', 'Zone_2', 'Zone_3',
#  'Zone_4', 'Language_1', 'Language_2', 'Language_3', 'Language_4',
#  'Language_5', 'Language_6', 'Language_7', 'Language_8', 'Language_9',
#  'Language_10', 'Religion_0', 'Religion_1', 'Religion_2', 'Religion_3',
#  'Religion_4', 'Religion_5', 'Religion_6', 'Religion_7', 'Mainhue_black',
#  'Mainhue_blue', 'Mainhue_brown', 'Mainhue_gold', 'Mainhue_green',
#  'Mainhue_orange', 'Mainhue_red', 'Mainhue_white', 'Topleft_black',
#  'Topleft_blue', 'Topleft_gold', 'Topleft_green', 'Topleft_orange',
#  'Topleft_red', 'Topleft_white', 'Botright_black', 'Botright_blue',
#  'Botright_brown', 'Botright_gold', 'Botright_green', 'Botright_orange',
#  'Botright_red', 'Botright_white'],
# dtype='object')

Answer 4

Use make_column_transformer and fit_transform for this:

from sklearn.preprocessing import oneHotEncoder
from sklearn.compose import make_column_transformer

transformer=make_column_transformer(oneHotEncoder(),categorical_column,remainder="passthrough")
transformed=transformer.fit_transform(data)
transformed_df=pd.dataframe(transforme,columns=transformer.get_feature_names())

You will get the data frame which has both encoded categorical data and other numerical columns

Answer 5

This is NOT my solution but when I tried the accepted solution I got "unusable" data..to my understanding OneHotEncode produces binary equivalent "values" for each categorical (text) value So this minor adjustment is needed

ohe = OneHotEncoder(sparse=False, categories='auto') 

and you go from this kind of encoding : [36 (0, 5)\t1.0\n (0, 46)\t1.0] to this : [36 0.0 0.0 0.0 0.0 0.0 1.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 1.0 0.0] The code is borrowed from Jason Brownlee and his article : https://machinelearningmastery.com/data-preparation-gradient-boosting-xgboost-python/