How to handle associated features in machine learning

Question

I am working on a classification project in which some features are linked and I'm not sure how to handle them.

I will simplify my project like that :

• There are different jobs, and multiple persons working on those jobs.
• Those persons are fit for their job.
• Replacing someone can only have negative effect or no effect.

I have a person1 that can be replaced by a person2 for a job, and the goal is to predict if it has a negative impact or not.

Each person has is own properties like : weight, age, height, IQ,..
A job also has properties like : manualJob, localization, temperature,..

When I list my features I have something like that :
(Person1 = P1, Person2 = P2, and the data has been normalized)

     P1_weight   P2_weight   P1_IQ    P2_IQ    manualJob   tempJob   neg_impact 
0       0.25       0.50      0.25     0.25        1        0.25       1 
1       0.75       0.25      0.50     0.25        0        0.50       0  
2       0.50       0.75      0.75     0.50        1        0.25       1  
... 

There should be a high interaction between P1_weight && P2 weight features, (as well between IQs features) that we want to capture in order to predict the neg_impact feature.

1. Now the change between P1_weight and P2_weight is important, but does a classic model as RandomForest can capture the link between those 2 features ? Same for other subject properties (P1_IQ && P2_IQ, P1_height && P2_height,..)

2. I'm afraid that if I reduce the difference between P1_weight and P2_weight into a single feature as the diff (P1_weight - P2_weight), I will lose some information. For example the P1_weight would probably be correlated to the 'manualJob' feature, and if I delete P1_weight this information would be lost?

3. I was thinking that maybe I could preprocess those linked features with out-of-fold prediction, and use those predictions as input with the rest of the features original features (manualJob, tempJob, ..). Is it a good idea ? Which kind of model would be better for the preprocessing to capture the correlation of those linked features ?

NB : my set only contains ~1000 elements

Answer 1

1. The RandomForest algorithm may capture such interaction by splitting an upper node by some condition on p1 and a lower node by another condition on p2, the link that you are expecting is captured if this happens under the same branch (p2 is on the path of p1). However, the ability to capture such information depends on the structure of the whole model (depth and number of trees) and the number of labels that you use.
2. It is a common practice to extract a third feature from coupled features to explicitly express the linkage between them, for example, the ratio of change or the product. For not losing important information as you suggested (since a diff of 0.1 may be derived by subtracting 0.9-0.8 and 0.2-0.1) you may add to the model the third feature in addition to one of the original features.
3. I'm not sure that I understand the meaning of out-of-fold, but anyway note that using your labels twice (once for features preprocessing and once for training) can lead you to overfit the model since the labels leaked to the features in the preprocessing step. It is ok to explore your features in advance and in separate to the labels for understanding the links, subcategories, missing values, measuring the correlations and so.

2 last note:

• there might be a confusion here between the meaning of correlation and interaction.
• you can consider using linear statistical modeling for your needs and number of data points. these models enabling to capture interactions and with their output you can learn about the significance of the correlation and to interpret them with their features importance. -- If you do so, you could also consider trying GLMM to see if there are job effects, i.e there is a difference in the interaction between different jobs.