I am trying to understand
- The reason behind encoding (one-hot encoding and dummy encoding)
- How one-hot and dummy are different from each other
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.
Now lets address your second query lets look into what is
one-hot encoding and
dummy encoding and then see the difference
One hot Encoding: Take the example of column name Fruit which can have different types of fruits like Blackberry, Grape, Orange. Here each category is mapped to binary variable containing either 0 or 1. Widely utilized when features are nominal.
|Fruit||Price (dollars per pound)|
Post one hot encoding the table now looks as shown below
|Blackberry||Grape||Orange||Price (dollars per pound)|
Dummy Encoding: similar to one hot encoding. While one hot encoding utilises
N binary variables for
N categories in a variable. Dummy encoding uses
N-1 features to represent
|Column||One Hot Code||Dummy Code|
The purpose of one-hot encoding is to assign numbers to categorical variables which does not create a false, meaningless numerical pattern.
If you have categorical variables "Apple", "Orange", "Cherry", "Tomato" and you assign them numerical values 0, 1, 2, 3, then these numerical values have interpretations like "Cherry is between Tomato and Apple, but closer to Tomato" because 2 is between 0 and 3, but closer to 3. This is nonsense. It's bad nonsense, because algorithms to analyze this data (like regressions, or whatever) can pick up on it and read too much into it.
If you instead represent "Apple", "Orange", "Cherry", and "Tomato" as the 4-tuples (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1), then you don't have this problem. Each coordinate is either 0 or 1, and measures the "Appleness" or the "Orangeness" or the "Cherriness" or the "Tomatoness" of your fruit. That's one-hot encoding.
As an example of this, suppose that the average apple weighs 200 grams, the average orange weighs 150 grams, the average cherry 30 grams, and the average tomato 100 grams. With one-hot encoding $(x_1, x_2, x_3, x_4)$, this average weight is a linear function of the encoding: $200x_1 + 150x_2 + 30x_3 + 100x_4$. This is something a regression can figure out from data. With the 0, 1, 2, 3 encoding, there's no nice function that will give you the average weight of a fruit given its number.
Now, as for dummy encoding: one-hot encoding still has a problem, which is that the linear function is not unique. The function $100 + 100x_1 + 50x_2 - 70x_3$ gives the same values as the previous function at the points (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1). That's because each valid point $(x_1, x_2, x_3, x_4)$ satisfies $x_1 + x_2 + x_3 + x_4 = 1$.
(Again, this is not just a curiosity; this affects the way we analyze the data. For example, linear regressions behave badly when an $n$-dimensional input doesn't actually range freely across all $n$ dimensions.)
Dummy encoding drops one of the coordinates, since it can be inferred from the other three, to avoid this issue. The four fruits might be encoded as (1,0,0), (0,1,0), (0,0,1), and (0,0,0).
To complete Archana David's answer:
From what I encountered, the big advantage of
sklearn.preprocessing.OneHotEncoder is that you can save it as an scikit-learn encoder, so you can train it on a train set, and apply it on your test based on what you train (you'll recreate the same columns). On the other hand,
pandas.get_dummies only applies directly on your dataframe, so you won't be able to train it on a set and then apply it to another based on what you trained first. This then causes issues with machine learning problems. Basically always use one-hot.
Dog Cat Rabbit
Train your one-hot encoder on that and apply it:
Dog Cat Rabbit 1 0 0 0 1 0 0 0 1
Dog Horse Cat
The one-hot encoder trained from your training set applied to your test:
Dog Cat Rabbit 1 0 0 0 0 0 0 1 0
If you used Pandas
pandas.get_dummies, you could only apply it directly on your test:
Dog Horse Cat 1 0 0 0 1 0 0 0 1
And you'll have a column mismatch between your train and test:
['Dog', 'Cat', 'Rabbit'] differs from
['Dog', 'Horse', 'Cat']