Ongoing feature selection

Question

If you have a set of n features you have 2^n-1 non-empty feature subsets. As a result, if you pick one of them you are unlikely to have found the best one.

To me, it seems intuitive that as you build your model, you would want to look at the things it does badly and try to find features that would help it to improve, or take out features that don't seem to be helping.

Although I've seen this done in practice and muddled through this way I've never seen any formal theory behind it. How do you know which features to add to the set your training on? and WHich to remove?

Answer 1

There are various features selections techniques. The most common techniques rank individual features by how much information they bring with respect to the target, for example with information gain or conditional entropy.

Techniques based on individual features are efficient (i.e. fast) and usually help to reduce dimensionality and improve performance. But they are not necessarily optimal, because they cannot take into account the contribution of a subset of features together. For example they might select several features which are highly correlated between each other, even though selecting only one of them would be enough.

In order to take into account how features interact, ideally one would train and test a model with every possible subset of features, and then select the best one. However the full exploration of $2^N$ subsets is rarely feasible, but some optimization methods can be used, for example feature selection with genetic learning.

Note that there are also features extraction techniques. In this case the original semantic of the features is not preserved, since the whole set of features are transformed into a new representation.