3
$\begingroup$

During feature engineering, we can create new features out of existing ones by using arithmetic operations albeit linear or not.

Let's say we have two features x and z. We can then create (engineer) a new feature f by summing x & z, assuming this makes sense in the context of the use case, to therefore become, f = x + z.

Or if a non linear feature is to be created then something like the following can be implemented, f = x*x + z.

My question is, given that we have x & z and knowing the strength of their correlation to the target variable, what is the point of creating a third feature which is just a combination of the original two?

What can the third feature point to that the first two can't?

Maybe if the combination is non-linear, I can understand, but what if the combination is linear? Why should it help?

P.S I have stumbled on a post on Cross-Validated addressing this issue, but given the nature of the answer, it still left me wondering and unsatisfied.

$\endgroup$
4
$\begingroup$

Feature engineering has at least two important benefits:

  1. You can simplify the task for your model by including interactions between features which the model otherwise would have to learn
  2. Beyond simplification, you can inject prior knowledge (e.g. expert knowledge) into the data and eventually the model

Here are two examples:

The non-linear case - Suppose you're developing a model to predict the risk of severe Covid19 (e.g. defined as patients being hospitalized or dying). If your datasets includes features, such as weight and height, your model could learn the association between these features and the outcome. However, given that empirical evidence demonstrated an increased risk of severe Covid19 for obese people you could feature engineer an independent variable BMI or obesity. If you do so your model does not need to learn the, most likely, non-linear relationship between height and weight with regards to the target variable.

The linear case - Suppose your developing a model to predict whether a household will purchase a luxury car. If your dataset includes income from salary, income from investments and other income. Then feature engineering a linear combination of these 3 total income can make it easier for your model to learn the relationship between total income and the target variable. To make it more concrete: Take, for example, a decision tree. Without total income it might need several split points including the different income variables to derive a prediction. In contrast, splitting on total income might result in a prediction with much fewer nodes required.

The non-linear case is usually more relevant though.

$\endgroup$
3
  • $\begingroup$ So how would you make the distinction between features that have a high multi-collinearity aspect to them and their usefulness to the model given that we want to simplify the task or as you mention, inject prior knowledge? Or is this just a given side effect? $\endgroup$ Aug 17 at 12:09
  • 1
    $\begingroup$ @KamalRaydan Regarding your previous (now deleted) comment: It is not introducing "new data" but certainly it can be "new ideas". To provide a less simple example: In this answer I briefly describe the application of PageRank to football match prediction. You could argue that all this information is already included in historical match data. And in a narrow sense that is true. However, I would certainly think of a PageRank feature as "a new idea" compared to the feature baseline of only having historical results. $\endgroup$
    – Sammy
    Aug 17 at 12:11
  • 1
    $\begingroup$ @KamalRaydan "how would you make the distinction between features that have a high multi-collinearity aspect to them and their usefulness to the model given that we want to simplify the task or as you mention, inject prior knowledge?" - If you're applying a model which is not vulnerable to redundant features, such as Random Forest, it is not a concern. In other cases, I'd apply feature selection after feature engineering. $\endgroup$
    – Sammy
    Aug 17 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.