2
$\begingroup$

I was wondering if there is any value to creating combined features out of multiple categorical variables when the individual categorical variables are already one-hot encoded?

Simple example: there is a variable P with categories {X, Y} and a variable Q with categories {Z, W}. After one-hot, we would have 4 variables: P.X, P.Y, Q.Z, and Q.W.

In this scenario, I'm wondering if the algorithm (Xgboost or a deep neural network) would sufficiently learn interaction effects between these or is there further value to creating variables: X.Z, X.W, Y.Z, Y.W which would be the unique combinations of P and Q.

The reason I am asking is to try to assess whether to embark on creating these interaction variables in my real-world scenario, where I have 7 such categorical features and 6-15 categories each which would mean thousands of new variables to account for all possible levels of permutations.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ At least with tree models (boosting etc), interactions will likely have a limited impact since each interaction will be considered as a separate variable which may add to a tree but not in all of the models. Interactions in tree models work in a different way than in linear models. However, interactions can be extremely useful since tree models usually don‘t „find“ interactions without explicitly showing them. When you use regulation, you may benefit quite a lot from including interactions. Can be x*z or x/z, dependent on the task. $\endgroup$
    – Peter
    Commented Aug 30, 2020 at 22:00
  • $\begingroup$ Thank you for your response, Peter. Would this still apply to Binary variables resulting from combinations of one-hot encoded categorical variables? In this case x*z will equal x/z. $\endgroup$
    – Artur Motruk
    Commented Aug 30, 2020 at 23:42
  • $\begingroup$ Yes, I think so. However, it is ex ante unclear how much explanatory power these interactions have, but it would be worth a try I guess. $\endgroup$
    – Peter
    Commented Aug 31, 2020 at 7:16

1 Answer 1

Reset to default
0
$\begingroup$

Tree models will be able to figure out the Interaction whether Label Encoded or OHE.
If you separately create all the combinations of Category values as new Features, then there is nothing left for interaction (At least 1st level).

e.g. let's take the example Features
P - {X,Y}
Q - {Z,W}

Let's assume, P and Q have interaction on X and W.
If we Label encode - We will observe multiple consecutive splits on P==X and Q==W.
If you OHE - you will observe multiple consecutive splits on Is_P_X and Is_Q_W (since both are separate feature now).
In these cases, you will have to observe the splits to see the interaction or can use a partial dependence plot or similar approaches.

If you Create features on all combinations -
In this case, you will observe very high Feature importance for feature X_W that is actually because of interaction but you will not observe it as consecutive splits on two feature but a split on single feature i.e X_W
In these cases, you can observe the interaction with Feature importance.

On a dummy data where I added the interaction effect on output for Sunday and Breezy weather.

Image 01 - Showing Interaction in the splits

enter image description here

Image 02 - Simply splitting on Sun_Breezy when every combination is a separate feature

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.