26
$\begingroup$

I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable like location: 

Location 
----------
Californian
NY
Florida

We have to convert them like:

1  0  0
0  1  0
0  0  1

However, it was suggested that we have to discard one dummy variable, no matter how many dummy variables are there.

Why do we need to discard one dummy variable?

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ Because the third dummy can be explained as the linear combination of the first two: FL = 1 - (CA + NY). $\endgroup$
    – chainD
    Feb 18, 2018 at 18:15
  • $\begingroup$ @chainD but what is the explanation for more than three dummy variable ? $\endgroup$
    – Mithun Sarker Shuvro
    Feb 18, 2018 at 18:20
  • 2
    $\begingroup$ Whatever the total, it will just be 1 less than the total number of categories that you have. Extending your example, say all 50 states were represented in the dataset. For a given individual, say you look at the first 49 dummies which happen to be all zeros, then you know that the last dummy is a 1 even without looking (assuming everyone in the dataset is from one of the 50 states). In other words, the information of the last dummy is already contained in the result of the first 49, so to speak. $\endgroup$
    – chainD
    Feb 18, 2018 at 18:29
  • 1
    $\begingroup$ if it is not spring, not summer and not autumn then it is winter! $\endgroup$
    – Stev
    Feb 19, 2018 at 17:08

2 Answers 2

Reset to default
16
$\begingroup$

Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical variable of K categories, or levels, usually enters a regression as a sequence of K-1 dummy variables. This amounts to a linear hypothesis on the level means."

This is already discussed at this very nice stats.stackexchange answer.

I was told there is an advanced course by Yandex in Coursera that covers this subject in more details if you still have doubts, see here. Note you can always audit the course content for free. ;-)

Another nice post if you want a thorough explanation with lots of examples with statistical perspective and not being limited to only dummy coding, see this from UCLA (in R)

Note that if you using pandas.get_dummies, there is a parameter i.e. drop_first so that whether to get k-1 dummies out of k categorical levels by removing the first level. Please note default = False, meaning that the reference is not dropped and k dummies created out of k categorical levels!

Improve this answer
$\endgroup$
2
  • 7
    $\begingroup$ Notice that this is only true if your model has an intercept (i.e., a constant term). Otherwise, by using one-hot-encoding and not discarding one dummy variable, you are implicitly adding an intercept. $\endgroup$
    – Elias Strehle
    Feb 19, 2018 at 17:11
  • $\begingroup$ See this discussion for a follow up question: datascience.stackexchange.com/a/84061/71442 $\endgroup$
    – Peter
    Oct 15, 2020 at 22:08
1
$\begingroup$

You don't need to drop a level, depending on your use case.

See
In which cases shouldn't we drop the first level of categorical variables?
and the much more general question
In supervised learning, why is it bad to have correlated features?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.