I am working with interaction features in my machine learning model, where I create new features by multiplying a numeric variable with an encoded categorical feature. My question is:
Should normalization be applied to these interaction terms? If yes, so should it be applied before interaction features are created or after the interaction features are created ?
In my case, I am using Neural Networks and GBM (Gradient Boosting Machines), and I’m wondering whether normalization of interaction terms will affect model performance. By normalization, I mean scaling to a 0-1 range.
Challenges when creating interaction terms:
- Numerical Features Normalized Before Interaction:
While this prevents the dominance of large numerical scales, it can distort the relative magnitude of the interaction term, especially when the categorical encoding carries meaningful differences (e.g., weighted target encoding). This approach may also introduce encoded categorical bias, where the interaction term does not accurately reflect the true relationship between the categorical and numerical features.
- Interaction Term Normalized After Creation:
This approach equalizes the scale of the interaction terms but can introduce a loss of interpretability for the interaction term and cause an uneven scale impact. In this case, the encoded categorical feature’s signal might get diluted, which could lead to potential bias in the model’s understanding of the relationships between the features.
How do you handle these challenges in your workflow?
- Do you always normalize numerical features before creating interactions?
- How do you preserve interpretability when normalizing interaction terms?
- Any tips or best practices to balance scale, bias, and interpretability in such cases?
For example, if I multiply a normalized numeric feature with a one-hot encoded categorical variable, does that change the original relationship between the numeric feature and the category, or does it still capture the intended interaction?
What did I try:
I have experimented with normalizing the numeric feature before creating the interaction term. Specifically, I normalized the numeric variable and then multiplied it with an encoded categorical feature. I also tried creating interaction features without normalizing the numeric variable first to compare the two approaches.
What was I expecting:
I was hoping to understand if normalizing the numeric feature before creating the interaction term would affect the model's ability to capture the intended relationships between the numeric and categorical features. I was also curious whether the meaning of the interaction term would be preserved or distorted due to the normalization of the numeric feature. I expected to learn whether normalization would cause the interaction term to lose its original scale and significance or if it would be beneficial for model convergence and performance.
