Cleaning input data with pd.get_dummies()

Question

What is the advantage of converting a series like

>>> df
   Color
0    Red
1   Blue
2  Green
3    Red

To a multiple series like the below?

>>> pd.get_dummies(df)
   Color_Blue  Color_Green  Color_Red
0           0            0          1
1           1            0          0
2           0            1          0
3           0            0          1

One could as well have a hot encoded values for the Color Column as below?

>>> labels=list(set(df.Color))
>>> pd.DataFrame(df.Color.map({x:labels.index(x) for x in labels}).rename('Color_Code'))
   Color_Code
0           1
1           2
2           0
3           1

I know syntactically pd.get_dummies looks much simpler, but somehow I want to lean towards a lesser number of features than more number of features...

Answer 1

Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this blog and this one to learn what are the advantages and disadvantages of choosing each. The first method you showed is called one-hot encoding that you can get easily by pd.get_dummies as you mentioned, and frequently used by people in algorithms like XGBoost. One of the biggest downfall of the first method OHE, is increasing often unnecessarily the dimensionality of your feature space. Other cons could be not being a good representative of feature space as well as problem of missing features in test data. The second method is LabelEncoding, usually used in target variables or when you have very few categorical features.

The problem surfaces when you start having a large number of categorical features (>50 % of feature space) with each feature having many sub levels. Here you need more clever way of feature encoding like Target-based mean encoding (smoothing, you can find some kernels in Kaggle like this). If you are Python user, I have found a fairly new package under scikit-learn-contrib offering a wide range of Categorical Encoding Methods. This Kdnuggets post also compares some of the aspect of using different methods.