What is the difference between one-hot and dummy encoding?

Question

I am trying to understand

• The reason behind encoding (one-hot encoding and dummy encoding)
• How one-hot and dummy are different from each other

Answer 1

Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.

Now lets address your second query lets look into what is one-hot encoding and dummy encoding and then see the difference

• One hot Encoding: Take the example of column name Fruit which can have different types of fruits like Blackberry, Grape, Orange. Here each category is mapped to binary variable containing either 0 or 1. Widely utilized when features are nominal.

  Fruit	Price (dollars per pound)
  Blackberry	3.82
  Grape	1.2
  Orange	.64

  Post one hot encoding the table now looks as shown below

  One Hot Encoded table

  Blackberry	Grape	Orange	Price (dollars per pound)
  1	0	0	3.82
  0	1	0	1.2
  0	0	1	.64

• Dummy Encoding: similar to one hot encoding. While one hot encoding utilises N binary variables for N categories in a variable. Dummy encoding uses N-1 features to represent N labels/categories

  One Hot Coding Vs Dummy Coding

  Column	One Hot Code	Dummy Code
  Blackberry	100	10
  Grape	010	01
  Orange	001	00

Answer 2

The purpose of one-hot encoding is to assign numbers to categorical variables which does not create a false, meaningless numerical pattern.

If you have categorical variables "Apple", "Orange", "Cherry", "Tomato" and you assign them numerical values 0, 1, 2, 3, then these numerical values have interpretations like "Cherry is between Tomato and Apple, but closer to Tomato" because 2 is between 0 and 3, but closer to 3. This is nonsense. It's bad nonsense, because algorithms to analyze this data (like regressions, or whatever) can pick up on it and read too much into it.

If you instead represent "Apple", "Orange", "Cherry", and "Tomato" as the 4-tuples (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1), then you don't have this problem. Each coordinate is either 0 or 1, and measures the "Appleness" or the "Orangeness" or the "Cherriness" or the "Tomatoness" of your fruit. That's one-hot encoding.

As an example of this, suppose that the average apple weighs 200 grams, the average orange weighs 150 grams, the average cherry 30 grams, and the average tomato 100 grams. With one-hot encoding $(x_1, x_2, x_3, x_4)$, this average weight is a linear function of the encoding: $200x_1 + 150x_2 + 30x_3 + 100x_4$. This is something a regression can figure out from data. With the 0, 1, 2, 3 encoding, there's no nice function that will give you the average weight of a fruit given its number.

Now, as for dummy encoding: one-hot encoding still has a problem, which is that the linear function is not unique. The function $100 + 100x_1 + 50x_2 - 70x_3$ gives the same values as the previous function at the points (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1). That's because each valid point $(x_1, x_2, x_3, x_4)$ satisfies $x_1 + x_2 + x_3 + x_4 = 1$.

(Again, this is not just a curiosity; this affects the way we analyze the data. For example, linear regressions behave badly when an $n$-dimensional input doesn't actually range freely across all $n$ dimensions.)

Dummy encoding drops one of the coordinates, since it can be inferred from the other three, to avoid this issue. The four fruits might be encoded as (1,0,0), (0,1,0), (0,0,1), and (0,0,0).

Answer 3

To complete Archana David's answer:

From what I encountered, the big advantage of sklearn.preprocessing.OneHotEncoder is that you can save it as an scikit-learn encoder, so you can train it on a train set, and apply it on your test based on what you train (you'll recreate the same columns). On the other hand, pandas.get_dummies only applies directly on your dataframe, so you won't be able to train it on a set and then apply it to another based on what you trained first. This then causes issues with machine learning problems. Basically always use one-hot.

Example:

Training set:

Dog
Cat
Rabbit

Train your one-hot encoder on that and apply it:

Dog  Cat  Rabbit
1    0    0
0    1    0
0    0    1

Test Set:

Dog
Horse
Cat

The one-hot encoder trained from your training set applied to your test:

Dog  Cat  Rabbit
1    0    0
0    0    0
0    1    0

If you used Pandas pandas.get_dummies, you could only apply it directly on your test:

Dog  Horse Cat
1    0     0
0    1     0
0    0     1

And you'll have a column mismatch between your train and test: ['Dog', 'Cat', 'Rabbit'] differs from ['Dog', 'Horse', 'Cat']