I am working on a binary classification problem where I have a mix of continuous and categorical variables.

Categorical variables were created by me using get_dummies function in pandas.

Now my questions are,

1) I see that there is a parameter called drop_first which usually is given the value True. Why do we have to do this? Let's say for the purpose of example, we have 2 values in gender column namely Male,Female. If I use drop_first=True, it returns only one column. like gender_male with binary 1 and 0 as values For example, If my feature importance returns gender_male as an important feature, Am I right to infer that it is only Male gender that influences the outcome (because male is denoted as 1 and female is 0) and female (0's) don't impact the model outcome? or 0's in general doesn't play any role in ML model predictions?

2) Let's say my gender has 3 values for example Male,Female,Transgender. In this case if I use drop_first=True, it would only returns two columns

gender_male with 1 and 0 - Here 0 represents Transgender right?

gender_female with 1 and 0 - Here 0 represents Transgender right?

3) What's the disadvantage of not using drop_first=True? Is it only about the increase in number of columns

Can you help me with the above queries?


1) Using drop_first=True is more common in statistics and often referred to as "dummy encoding" while using drop_first=False gives you "one hot-encoding" which is more common in ML. For algorithmic approaches like Random Forests it does not make a difference. Also see "Introduction to Machine Learning with Python"; Mueller, Guido; 2016:

The one-hot encoding we use is quite similar, but not identical, to the dummy encoding used in statistics. For simplicity, we encode each category with a different binary feature. In statistics, it is common to encode a categorical feature with k different possible values into k–1 features (the last one is represented as all zeros). This is done to simplify the analysis (more technically, this will avoid making the data matrix rank-deficient).

However, using dummy encoding on a binary variable does not mean that a 0 has no relevance. If gender_male has high importance that does not generally say anything about the importance of gender_male==0 vs gender_male==1. It is variable importance and accordingly calculated per variable. If you, for example, use impurity based estimates in Trees it only gives you the average reduction in impurity achieved by splitting on this very variable.

Moreover, if your gender variable is binary, gender_male==1 is equivalent to gender_female==0. Therefore from a high variable importance of gender_male you cannot infer that being female (or not) is not relevant.

2) In this case gender_male==0 AND gender_female==0 means Transgender is true.

3) see 1). For algorithmic approaches in ML there is no statistical disadvantage using one-hot-encoding. (as pointed out in the comments it might even be advantageous since tree-based models can directly split on all features when none is being dropped)

  • $\begingroup$ Hi, thanks for the response. Upvoted. Sorry I didn't get your point regarding gender_male . Yes, my gender is a binary variable and I see this as a important variable. Now if I have to make an interpretation of my model, how do I do it? Because it has 1 which indicates male and 0 indicates female. So can you please help me understand this much better in ordinary layman terms for interpretation part alone? I understood the first half of point 1. $\endgroup$ – The Great Dec 29 '19 at 13:32
  • $\begingroup$ should I understand that gender being male can influence outcome more than gender being female? At the end of the day, there has to be some interpretation or usefulness of gender_male and am trying to know what it is $\endgroup$ – The Great Dec 29 '19 at 13:33
  • $\begingroup$ What I meant is that if gender is a binary variable you cannot make any inference whether being male (or not) is more important than being female (or not) since these two are equivalent. If you use dummy encoding (drop_first=True) and your resulting independent variable gender_male turns out to be of high importance, then you can only infer that gender is important for your model. $\endgroup$ – Sammy Dec 29 '19 at 13:58
  • $\begingroup$ fantastic got it. $\endgroup$ – The Great Dec 29 '19 at 13:59
  • 1
    $\begingroup$ I disagree that drop_first doesn't make a difference in random forest, when the original variable has more than two levels. In order to select for the missing category, a tree has to make several subsequent selections of not each (other) dummy variable. $\endgroup$ – Ben Reiniger Dec 30 '19 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.