2
$\begingroup$

I am testing the Play Golf data set using Decision tree classifier:

enter image description here

I am splitting the data into Outlook, Temp, Humidity and windy as features, and Play Golf as target feature.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_excel('data.xlsx')
X = dataset.iloc[:, 0:4]
y = dataset.iloc[:, 4]

To test the DecisionTreeClassifier I need to transform categorical data into numeric:

from sklearn.preprocessing import LabelEncoder
lb = LabelEncoder()

X['Outlook'] = lb.fit_transform(X['Outlook'])
X['Temp'] = lb.fit_transform(X['Temp'])
X['Humidity'] = lb.fit_transform(X['Humidity'])
X['Windy'] = lb.fit_transform(X['Windy'])
y = lb.fit_transform(y)

The result is:

ix  Otk T   H   W
0   1   1   0   0
1   1   1   0   1
2   0   1   0   0
3   2   2   0   0
4   2   0   1   0
5   2   0   1   1
6   0   0   1   1
7   1   2   0   0
8   1   0   1   0
9   2   2   1   0
10  1   2   1   1
11  0   2   0   1
12  0   1   1   0
13  2   2   0   1

Should I use OneHotEncoder() class afterwards? Or 2 isn't that big to do so?

$\endgroup$
3
$\begingroup$

The purpose of one-hot encoding is to binarize categorical labels so that your model doesn't learn a spurious ordinal relationship.

For example if you had categories Red, Yellow, Blue, it would be bad to encode Red as 0, Yellow as 1, and Blue as 2 because your model might accidentally "learn" that Red < Yellow < Blue. In this case you should use one-hot encoding.

But for some categorical features, an ordinal encoding makes sense. For example, your dataset has the feature "Temperature", which can be Cool, Mild, or Hot. In this case, the relationship Cool < Mild < Hot is semantically meaningful, so you can probably get away with ordinal encoding if you want.

However, you still have to ensure that the ordinal relationship is preserved in your encoding, and scikit-learn's LabelEncoder does not take care of this for you. To illustrate, look at how "Temperature" is being encoded:

  • Hot => 1
  • Mild => 2
  • Cool => 0

That's no good! You want Hot to have the highest value. Instead you should use scikit's OrdinalEncoder and explicitly tell it how to order the categories:

from sklearn.preprocessing import LabelEncoder, OrdinalEncoder

outlook_encoder = OrdinalEncoder(categories=['Rainy', 'Overcast', 'Sunny'])
X['Outlook'] = outlook_encoder.fit_transform(X['Outlook'])

temp_encoder = OrdinalEncoder(categories=['Cool', 'Mild', 'Hot'])
X['Temp'] = temp_encoder.fit_transform(X['Temp'])

# humidity and wind are binary features, so we don't need the ordinal encoder

humidity_encoder = LabelEncoder()
X['Humidity'] = humidity_encoder.fit_transform(X['Humidity'])

windy_encoder = LabelEncoder()
X['Windy'] = windy_encoder.fit_transform(X['Windy'])


Although there do exist ordinal relationships in most of your categorical features, you should still feel free to use one-hot encoding. It's always a safe bet when you're dealing with categorical features.

Since your dataset is small, you should experiment with both and see which gives the best results!

$\endgroup$
4
  • $\begingroup$ It is really confusing. How could I use one hot encoder with setting label encoder first ? I read in all docs that LabelEncoder should be used first then one hot encoder is called.0 $\endgroup$
    – alim1990
    Sep 30 '19 at 18:34
  • 2
    $\begingroup$ You don't need to apply the label encoder first. With recent changes in the library, you can use OneHotEncoder directly on your string data, and it should work fine. $\endgroup$
    – zachdj
    Sep 30 '19 at 21:36
  • $\begingroup$ some tutorials online explaining entropy and decision tree, didn't split categorical into numeric data before they start the calculations. Isn't for simplicity they skip it or the internet is now full of newbies who wants to share what they learned online? like this medium.com/coinmonks/… or ccg.doc.gold.ac.uk/ccg_old/teaching/artificial_intelligence/… $\endgroup$
    – alim1990
    Oct 1 '19 at 5:14
  • 1
    $\begingroup$ When writing a tutorial to explain entropy or to explain how decision trees work, I think it makes sense to use human-friendly labels. The goal in this context is to impart understanding. Using one-hot encoded vectors would just muddy the water and present a barrier to comprehension. $\endgroup$
    – zachdj
    Oct 1 '19 at 13:11
2
$\begingroup$

To answer the question straightaway:

Should I use OneHotEncoder() class afterwards?

No. After LabelEncoder you do not need to do OneHotEncoder again. I do not know what you meant by:

Or 2 isn't that big to do so?

HOWEVER I suggest you do NOT do LabelEncoder. Think for a sec what you encoded. Basically you assigned numerical values to categories in a particular order. For example for the Outlook variable you do Rainy: 1, Overcast:0 and Sunny:2. In this way you imply to the model to learn that:

2 (Sunny) is greater than 1 (Rainy)

which is not the case, and obviously is wrong!

Here actually OneHotEncoder would be a better choice. My educated guess is that you should be able to see that a trained model would perform better based on OneHotEncoder (maybe not! :D). If Order matters or have a meaning, like in the case of Humadity variable, OrdinalEncoder can also be used.

I further suggest reading more about categorical encoding. There are many articles, blog posts, like this one, or this one, or this benchmark, and so many others. This is just the beginning. ;-)

$\endgroup$
4
  • $\begingroup$ But to build a decision tree shouldn't we encode the categorical features ? $\endgroup$
    – alim1990
    Sep 30 '19 at 18:23
  • $\begingroup$ And if I should only use OneHotEncoder, LabelEncoder should be used first right ? $\endgroup$
    – alim1990
    Sep 30 '19 at 18:25
  • 2
    $\begingroup$ Yes you have to encode categorical features. But to use OneHotEncoder you do not need to do LabelEncoder first! Check this post out, they have used the same dataset and used One Hot Encoding using Pandas: medium.com/@randerson112358/… $\endgroup$ Sep 30 '19 at 19:27
  • $\begingroup$ some tutorials online explaining entropy and decision tree, didn't split categorical into numeric data before they start the calculations. Isn't for simplicity they skip it or the internet is now full of newbies who wants to share what they learned online? like this medium.com/coinmonks/… or ccg.doc.gold.ac.uk/ccg_old/teaching/artificial_intelligence/… $\endgroup$
    – alim1990
    Oct 1 '19 at 5:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.