What is the point of generating new features (linear or non linear) out of existing features in a dataset?

Question

During feature engineering, we can create new features out of existing ones by using arithmetic operations albeit linear or not.

Let's say we have two features x and z. We can then create (engineer) a new feature f by summing x & z, assuming this makes sense in the context of the use case, to therefore become, f = x + z.

Or if a non linear feature is to be created then something like the following can be implemented, f = x*x + z.

My question is, given that we have x & z and knowing the strength of their correlation to the target variable, what is the point of creating a third feature which is just a combination of the original two?

What can the third feature point to that the first two can't?

Maybe if the combination is non-linear, I can understand, but what if the combination is linear? Why should it help?

P.S I have stumbled on a post on Cross-Validated addressing this issue, but given the nature of the answer, it still left me wondering and unsatisfied.

Answer 1

Feature engineering has at least two important benefits:

1. You can simplify the task for your model by including interactions between features which the model otherwise would have to learn
2. Beyond simplification, you can inject prior knowledge (e.g. expert knowledge) into the data and eventually the model

Here are two examples:

The non-linear case - Suppose you're developing a model to predict the risk of severe Covid19 (e.g. defined as patients being hospitalized or dying). If your datasets includes features, such as weight and height, your model could learn the association between these features and the outcome. However, given that empirical evidence demonstrated an increased risk of severe Covid19 for obese people you could feature engineer an independent variable BMI or obesity. If you do so your model does not need to learn the, most likely, non-linear relationship between height and weight with regards to the target variable.

The linear case - Suppose your developing a model to predict whether a household will purchase a luxury car. If your dataset includes income from salary, income from investments and other income. Then feature engineering a linear combination of these 3 total income can make it easier for your model to learn the relationship between total income and the target variable. To make it more concrete: Take, for example, a decision tree. Without total income it might need several split points including the different income variables to derive a prediction. In contrast, splitting on total income might result in a prediction with much fewer nodes required.

The non-linear case is usually more relevant though.