1
$\begingroup$

I ran an OLS model on a dataset with 2 categorical variables. One of them was gender. The other one had 3 different categories. I used one-hot encoding for it during pre-processing before running my model.

Variables in the image: Embarked_C and Embarked_Q. The results showed a p-value for Embarked_Q as 0.785. In this case, should I remove both Embarked_Q and C or just Q?

Regression Results

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

You should keep all of levels as they collectively describe the feature. Removing the insignificant ones will bias your coefficients and distort your interpretation (e.i. change the reference level).

Here are some stats.exchange references:

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you. I removed the entire category and realized that it gives me the best adjusted R-squared value. I might have been lucky, but this wouldn't always be the case for every dataset. $\endgroup$
    – Trina Ghosh
    Commented Apr 22, 2020 at 16:51
0
$\begingroup$

By all means keep Embarked_C. Consider the following example: You're predicting whether or not someone's favorite color is blue. You know the color of their favorite shirt- it's either blue, yellow, or red. Color_Blue is going to be significant, the other two one hot encoded variables would not be. You'd still want to keep Color_Blue as a feature.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer, Brandon. I did remove all the other non-significant variables and finally, Embarked_Q. But my adjusted R-squared value decreased by 0.01. Turns out, if I remove the entire category (both C and Q), it gives me the best adjusted R-Squared value. $\endgroup$
    – Trina Ghosh
    Commented Apr 22, 2020 at 16:48
  • $\begingroup$ I'd expect the R-squared value to drop a little bit when removing Q. The trade-off is that you're reducing the propensity to overfit when you remove less significant features. $\endgroup$
    – Brandon Schabell
    Commented Apr 22, 2020 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.