How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

Question

I am using XGBoost for a machine learning model that learns from tabular data.

XGBoost uses boosting method on decision trees. When I look at the decision-making logic of decision trees, I notice the logic is based on 1 feature at one time. In real life, certain multiple features are related to each other.

Currently, when I feed data to the model, I simply feed all the features to it without telling the model how certain features are related to each other.

Let me describe a hypothetical example to be clearer. Suppose I have 2 features - gender and length of hair. In this hypothetical problem, I know from my domain knowledge that if gender is female, length of hair matters in determining the outcome. If gender is male, length of hair is irrelevant. How do I tell the machine learning model this valuable piece of information so that the model can learn better?

I am using XGBoost on python 3.7

Answer 1

I am going to talk about some ways you could do it later but first I want to talk about whether you should!

If the relation that you describe exists XGB will be able to learn and detect it! There is no real benefit in "hard-coding" a rule into the algorithm, it won't speed up the training, it won't benefit accuracy, etc. Simply put the benefit of ML algorithms is that they are able to detect exactly these relationships and model them in the best possible way.

Now if you still insist that this is something that must be done, you can. The easiest way to achieve this would be feature engineering:

1. Introduce NAs -simply leave out hair length of male respondents and fill with NA

2. Create interaction factors - instead of having hair length and gender as a simple variables you could also code it in a way that represents the known interaction like this:

gender_hair = [male, female_short,female_medium,female_long] # example factor levels

But again if you compare models with those engineered features to a simpler model you will see no benefit I'd wager.