You can do dummy encoding using Pandas in order to get one-hot encoding as shown below:
import pandas as pd
# Multiple categorical columns
categorical_cols = ['a', 'b', 'c', 'd']
pd.get_dummies(data, columns=categorical_cols)
If you want to do one-hot encoding using sklearn library, you can get it done as shown below:
from sklearn.preprocessing import OneHotEncoder
onehotencoder = OneHotEncoder()
transformed_data = onehotencoder.fit_transform(data[categorical_cols])
# the above transformed_data is an array so convert it to dataframe
encoded_data = pd.DataFrame(transformed_data, index=data.index)
# now concatenate the original data and the encoded data using pandas
concatenated_data = pd.concat([data, encoded_data], axis=1)
If a single column has more than 500 categories, the aforementioned way of one-hot encoding is not a good approach. In this case, we can do one-hot encoding for the top 10 or 20 categories that are occurring most for a particular column. A sample code is shown below:
categorical_cols = ['a', 'b', 'c', 'd']
# Let's say we have a column 'b' which has more than 500 categories.
# Find the top 10 most frequent categories for column 'b'
data.b.value_counts().sort_values(ascending = False).head(20)
# make a list of the most frequent categories of the column
top_10_occurring_cat = [cat for cat in data.b.value_counts().sort_values(ascending = False).head(10).index]
# now make the 10 binary variables
for cat in top_10_occurring_cat:
data[cat] = np.where(data['b'] == cat, 1, 0) # whenever data['b'] == cat replace it with 1 else 0
# This is done for one categorical column, similarly you can repeat for all categorical columns