4
$\begingroup$

What is the advantage of converting a series like

>>> df
   Color
0    Red
1   Blue
2  Green
3    Red

To a multiple series like the below?

>>> pd.get_dummies(df)
   Color_Blue  Color_Green  Color_Red
0           0            0          1
1           1            0          0
2           0            1          0
3           0            0          1

One could as well have a hot encoded values for the Color Column as below?

>>> labels=list(set(df.Color))
>>> pd.DataFrame(df.Color.map({x:labels.index(x) for x in labels}).rename('Color_Code'))
   Color_Code
0           1
1           2
2           0
3           1

I know syntactically pd.get_dummies looks much simpler, but somehow I want to lean towards a lesser number of features than more number of features...

$\endgroup$
4
$\begingroup$

Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this blog and this one to learn what are the advantages and disadvantages of choosing each. The first method you showed is called one-hot encoding that you can get easily by pd.get_dummies as you mentioned, and frequently used by people in algorithms like XGBoost. One of the biggest downfall of the first method OHE, is increasing often unnecessarily the dimensionality of your feature space. Other cons could be not being a good representative of feature space as well as problem of missing features in test data. The second method is LabelEncoding, usually used in target variables or when you have very few categorical features.

The problem surfaces when you start having a large number of categorical features (>50 % of feature space) with each feature having many sub levels. Here you need more clever way of feature encoding like Target-based mean encoding (smoothing, you can find some kernels in Kaggle like this). If you are Python user, I have found a fairly new package under scikit-learn-contrib offering a wide range of Categorical Encoding Methods. This Kdnuggets post also compares some of the aspect of using different methods.

| improve this answer | |
$\endgroup$
  • $\begingroup$ hey @majid thanks for the quick response will go through the posts you mentioned. Can you elaborate on Target-based mean encoding ? $\endgroup$ – kaza Jan 24 '18 at 22:22
  • $\begingroup$ You are welcome. It is simple. I should have mentioned this nice and clear blog (saedsayad.com/encoding.htm) as well. Please read the second section to learn about Target-based Encoding (in essence, the mean of target for each categorical feature is used to convert that feature into numerical value in regression case). $\endgroup$ – TwinPenguins Jan 24 '18 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.