5
$\begingroup$

I am trying to understand

  • The reason behind encoding (one-hot encoding and dummy encoding)
  • How one-hot and dummy are different from each other
$\endgroup$
2
  • $\begingroup$ See this for an example and my answer, so you best understand what OneHot does and how it works : datascience.stackexchange.com/q/80809/101580 $\endgroup$ – BeamsAdept Jul 22 at 7:10
  • $\begingroup$ One-hot encoding and dummy encoding historically mean the exact same thing. The former term originated from machine learning, the latter from statistics. However, it does seem that over the years the two have separated to represent whether to drop one level as in Archana's answer. Just be aware that not everyone will take that view (e.g. the first comment on this question). $\endgroup$ – Ben Reiniger yesterday
4
$\begingroup$

Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.

Now lets address your second query lets look into what is one-hot encoding and dummy encoding and then see the difference

  • One hot Encoding: Take the example of column FoodName which can have different types of food like apple, chicken, brocolli. Here each category is mapped to binary variable containing either 0 or 1. Widely utilized when features are nominal. enter image description here
  • Dummy Encoding: similar to one hot encoding. While one hot encoding utilises N binary variables for N categories in a variable. Dummy encoding uses N-1 features to represent N labels/categories enter image description here
$\endgroup$
2
  • $\begingroup$ When would you use one and when would you use the other? $\endgroup$ – NotThatGuy Jul 22 at 17:31
  • 2
    $\begingroup$ Dummy encoding is needed in models where multicollinearity is a problem. In, say, a logistic regression, if you use N dummy variables (as in one-hot encoding), then the intercept term will be redundant with the dummy variables, which will cause the model to behave badly. This is less of a problem in nonlinear ML models. $\endgroup$ – Nobody Jul 23 at 14:30
3
$\begingroup$

To complete the accepted answer : From what I encountered, the big advantage of sklearn.preprocessing.OneHotEncoder is that you can save it as an sklearn encoder, so you can train it on a train set, and apply it on your test based on what you train (you'll re-create the same columns). In the other side, pandas.get_dummies only applies directly on your dataframe, so you won't be able to train it on a set and then apply it to another based on what you trained first. This then causes issues on Machine Learning problems. Basically always use OneHot.

Example : Training set :

Dog
Cat
Rabbit

Train your One Hot Encoder on that and applies it :

Dog  Cat  Rabbit
1    0    0
0    1    0
0    0    1

Test Set :

Dog
Horse
Cat

One Hot Encoder trained from your Training set applied to your Test :

Dog  Cat  Rabbit
1    0    0
0    0    0
0    1    0

If you used pandas pandas.get_dummies, you could only apply it directly on your test :

Dog  Horse Cat
1    0     0
0    1     0
0    0     1

And you'll have a column mismatch between your train and test : ['Dog', 'Cat', 'Rabbit'] differs from ['Dog', 'Horse', 'Cat']

$\endgroup$
1
  • $\begingroup$ Note that sklearn's OneHotEncoder has a parameter drop to allow for dropping reference levels. $\endgroup$ – Ben Reiniger yesterday
1
$\begingroup$

The purpose of one-hot encoding is to assign numbers to categorical variables which does not create a false, meaningless numerical pattern.

If you have categorical variables "Apple", "Orange", "Cherry", "Tomato" and you assign them numerical values 0, 1, 2, 3, then these numerical values have interpretations like "Cherry is between Tomato and Apple, but closer to Tomato" because 2 is between 0 and 3, but closer to 3. This is nonsense. It's bad nonsense, because algorithms to analyze this data (like regressions, or whatever) can pick up on it and read too much into it.

If you instead represent "Apple", "Orange", "Cherry", and "Tomato" as the 4-tuples (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1), then you don't have this problem. Each coordinate is either 0 or 1, and measures the "Appleness" or the "Orangeness" or the "Cherriness" or the "Tomatoness" of your fruit. That's one-hot encoding.

As an example of this, suppose that the average apple weighs 200 grams, the average orange weighs 150 grams, the average cherry 30 grams, and the average tomato 100 grams. With one-hot encoding $(x_1, x_2, x_3, x_4)$, this average weight is a linear function of the encoding: $200x_1 + 150x_2 + 30x_3 + 100x_4$. This is something a regression can figure out from data. With the 0, 1, 2, 3 encoding, there's no nice function that will give you the average weight of a fruit given its number.

Now, as for dummy encoding: one-hot encoding still has a problem, which is that the linear function is not unique. The function $100 + 100x_1 + 50x_2 - 70x_3$ gives the same values as the previous function at the points (1,0,0,0), (0,1,0,0), (0,0,1,0), and (0,0,0,1). That's because each valid point $(x_1, x_2, x_3, x_4)$ satisfies $x_1 + x_2 + x_3 + x_4 = 1$.

(Again, this is not just a curiosity; this affects the way we analyze the data. For example, linear regressions behave badly when an $n$-dimensional input doesn't actually range freely across all $n$ dimensions.)

Dummy encoding drops one of the coordinates, since it can be inferred from the other three, to avoid this issue. The four fruits might be encoded as (1,0,0), (0,1,0), (0,0,1), and (0,0,0).

$\endgroup$
3
  • 1
    $\begingroup$ Your last example doesn't seem to hold? For (1,0,0,0) it's 200, for (0,1,0,0) it's 150, etc. $\endgroup$ – justhalf Jul 23 at 8:57
  • $\begingroup$ Aren't those the values it's supposed to have? e.g., (1,0,0,0) is an apple, which weighs 200 grams, and both functions give 200 and (1,0,0,0). $\endgroup$ – Misha Lavrov Jul 23 at 13:24
  • $\begingroup$ Ah, ok. I misunderstood. So you meant it gives the same answer as the previous linear function. I thought it gives the same answer for all four points. $\endgroup$ – justhalf Jul 23 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy