Say I've got two (not necessarily independent) features A and B for my dataset. Should I create metafeatures from them? say for example the ratio: $$\frac{A}{B}$$



You should think about the physical meaning of each proposed metafeature, and whether it's relevant to the problem at hand. For example, suppose you're interested in patient temperature. You might add in $height^2$ and $height^3$, as they are roughly proportional to surface area (through which patient loses heat to a cool environment) and proportional to volume (tissue respirating to generate heat). Of course, you might choose to discard that last one as silly if patient $weight$ is part of your dataset. If your subjects are mammals ranging from mouse up to elephant, that cubic entry might still hold some predictive power.

Multiplying a pair of prices to obtain a column with units of dollars squared would not be a good idea.

In the familiar iris dataset, multiplying length by width to get sepal area or petal area definitely makes sense, as does dividing them, to obtain an aspect ratio.

If a metafeature doesn't improve training error, then of course discard it and move on.

  • $\begingroup$ Wouldn’t you expect xgboost to produce splits that effectively did the same thing? I can understand that creating these meta features makes sense with linear regression. $\endgroup$ – cjm2671 Dec 15 '18 at 10:43
  • $\begingroup$ No. Inferring a ratio or a cubic relationship, or a relationship among two or three noisy variables, is not what a technique like XGBoost is good at. For an even harder example, consider a task like classifying positive profit = revenue - cost, or profit = revenue - (rent + tax). $\endgroup$ – J_H Dec 15 '18 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.