I am trying to understand
- The reason behind encoding (one-hot encoding and dummy encoding)
- How one-hot and dummy are different from each other
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.
Now lets address your second query lets look into what is
one-hot encoding and
dummy encoding and then see the difference
Nbinary variables for
Ncategories in a variable. Dummy encoding uses
N-1features to represent
To complete the accepted answer :
From what I encountered, the big advantage of
sklearn.preprocessing.OneHotEncoder is that you can save it as an sklearn encoder, so you can train it on a train set, and apply it on your test based on what you train (you'll re-create the same columns). In the other side,
pandas.get_dummies only applies directly on your dataframe, so you won't be able to train it on a set and then apply it to another based on what you trained first. This then causes issues on Machine Learning problems. Basically always use OneHot.
Example : Training set :
Dog Cat Rabbit
Train your One Hot Encoder on that and applies it :
Dog Cat Rabbit 1 0 0 0 1 0 0 0 1
Test Set :
Dog Horse Cat
One Hot Encoder trained from your Training set applied to your Test :
Dog Cat Rabbit 1 0 0 0 0 0 0 1 0
If you used pandas
pandas.get_dummies, you could only apply it directly on your test :
Dog Horse Cat 1 0 0 0 1 0 0 0 1
And you'll have a column mismatch between your train and test :
['Dog', 'Cat', 'Rabbit'] differs from
['Dog', 'Horse', 'Cat']
The purpose of one-hot encoding is to assign numbers to categorical variables which does not create a false, meaningless numerical pattern.
If you have categorical variables "Apple", "Orange", "Cherry", "Tomato" and you assign them numerical values 0, 1, 2, 3, then these numerical values have interpretations like "Cherry is between Tomato and Apple, but closer to Tomato" because 2 is between 0 and 3, but closer to 3. This is nonsense. It's bad nonsense, because algorithms to analyze this data (like regressions, or whatever) can pick up on it and read too much into it.
If you instead represent "Apple", "Orange", "Cherry", and "Tomato" as the 4-tuples (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1), then you don't have this problem. Each coordinate is either 0 or 1, and measures the "Appleness" or the "Orangeness" or the "Cherriness" or the "Tomatoness" of your fruit. That's one-hot encoding.
As an example of this, suppose that the average apple weighs 200 grams, the average orange weighs 150 grams, the average cherry 30 grams, and the average tomato 100 grams. With one-hot encoding $(x_1, x_2, x_3, x_4)$, this average weight is a linear function of the encoding: $200x_1 + 150x_2 + 30x_3 + 100x_4$. This is something a regression can figure out from data. With the 0, 1, 2, 3 encoding, there's no nice function that will give you the average weight of a fruit given its number.
Now, as for dummy encoding: one-hot encoding still has a problem, which is that the linear function is not unique. The function $100 + 100x_1 + 50x_2 - 70x_3$ gives the same values as the previous function at the points (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1). That's because each valid point $(x_1, x_2, x_3, x_4)$ satisfies $x_1 + x_2 + x_3 + x_4 = 1$.
(Again, this is not just a curiosity; this affects the way we analyze the data. For example, linear regressions behave badly when an $n$-dimensional input doesn't actually range freely across all $n$ dimensions.)
Dummy encoding drops one of the coordinates, since it can be inferred from the other three, to avoid this issue. The four fruits might be encoded as (1,0,0), (0,1,0), (0,0,1), and (0,0,0).