Regarding "modification" of feature columns in supervised learning

Question

I have a training set with columns as follows: $x_1,x_2,x_3,x_4, x_3/x_4$. I want to know if I should consider $x_3$,$x_4$ and $x_3/x_4$ as separate features (as $x_3/x_4$ is derived from other features)? Also I found that dropping $x_3$ as a feature increases my k-fold neg-loss score (marginally!).

The underlying question is how would I know if entertaining $1/x_1$ or $x_1/x_2$ or even $\sqrt{x_4}x_3^4/x_1$... (you get the idea) as another feature column will/won't increase my score?

Is there any method/algorithm to know which "modified" features will increase my score?

Answer 1

I agree mostly with what was already said regarding feature engineering and just to provide you with more material this post has a nice analysis regarding different stages in feature engineering, with many links and references to papers and specific challenges. I think it's worth checking out.

Also, to some extend you could automate the task of finding "good" compound features, using kernel methods. You could use some of the standard kernel methods implemented in many libraries (e.g. in sklearn) as feature extractors and feed these higher-level features as input to a random forest model that inherently uses feature-importance while training. Then keep only most-informative of these higher-dimension features, as scored by the trained model, as compound features alongside the raw data.

Answer 2

I think what you're talking about is called compound features, and it's extremely important because sometimes it captures interactions between certain features which are not readily apparent in considering each column vector (or column) independently.

The answer to your question is, it may or may not, there's really no way to know without looking at the data, for example if we have a dataset consisting of the following features $x_1, x_2, x_3, x_4$. You could create many datasets consisting of mathematical operations between these variables, like you have mentioned in the question.

Now for the reason as to why someone would want to do this. Consider a supervised learning scenario where you're trying to classify a nominal feature with respect to some other features. This is a purely hypothetical example, but let's say we're predicting skin color which has $3$ or $4$ distinct classes based on facial features (absolute nose position in a grid, position of left eye, etc). While you may attain low error, it may be worthwhile considering compound features like position of right eye - position of left eye which could give us more information than their absolute positions.

To answer the latter question, there isn't any real algorithm which does this because they are feature combinations which are based on the data and anything that does this would have to be a trial and error mechanism. Because underlying data similarities may not be apparently visible. One way you could solve this is in your elementary data processing phase, consider the actual possible relationships between data, or at least, what you perceive them to be.

Answer 3

If you are considering the information theoretical point of view, given x3and x4 then x3/x4 adds no information.

However, before we rush into conclusions, one must recall that there are more aspects.

The first one is the ability to represent the concept. Let's consider as case in which your features are x3and x4 and the concept is x3/x4. The only problem is that I force you to use a decision tree as the model. How can you represent that? Note that even if I allow you to choose any tree, the representation will be bad.

The second problem is that the concept is not given in advance and therefor the algorithm should search for it. Most supervised learning algorithms are actually looking for correlations between the concept and the features. A relation like 1/x usually cause them problem since the behavior is not linear in x.

So, in case that you have ideas for features that don't add information but ease representation or search, adding them might help.

You are right asking where should I stop? It might be that there is a different combination of features that will lead to a better model. Unfortunately, we don't have a good solution for that.

Consider the problem of building good features, let's say features that reduce the size of the model. We can restrict our self to features that can be represented by trees. However, building a small tree for a data set is an NP-complete problem, which means that we don't have an efficient algorithm for that.

More than that, you can claim that given a feature building problem you can reduce classification into it (e.g., build a feature that represents the concept). Therefore, feature building is as hard as classification.