In the linear regression, when we have a categorical explanatory variable with $n$ levels, we usually remove one level and call it a baseline level and fit the model on the remaining levels. And the final intercept is the intercept plus the coefficient of baseline level. Now my questions are:

  1. Does it matter which level I choose to remove? I am working on a dataset to predict house price. When I use linear model with intercept on the testset for prediction, I get negative values for some houses which is not correct, as house price cannot be negative. But when I fit a regression without an intercept all the prediction values are positive as expected. I guess it has to do something with the baseline level I chose.
  2. Do I have to remove one level from all my categorical variables in the dataset before fit a random forest, KNN, Ridge, Lasso,... too?!
  1. For linear regression, we have to do one hot encoding and it creates one less number of variables then levels of the categorical variable. In latest tools you don't have to do it manually it automatically does I have tried in R. One hot encoding has no impact on which level you choose, it is basically a binary way of representing the level True or False. This will create whole new variables which with just 1 and 0.

  2. For tree-based algorithms, you basically don't need to apply the one hot encoding. These algorithms are capable enough to handle the categorical variables.

Reference: https://gerardnico.com/wiki/data_mining/dummy

  • $\begingroup$ Thank you. Could you please add some references to what you said. The reference for regression is Applied Linear Statistical Models by John netter.chapter 7 or 8 talks about one hot encoding and why we are doing it. $\endgroup$
    – Mary
    May 31 '17 at 11:45
  • $\begingroup$ @ Mary I have updated the post to give you one reference. Hope this helps. $\endgroup$ Jun 1 '17 at 5:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.