If you have a set of n features you have 2^n-1 non-empty feature subsets. As a result, if you pick one of them you are unlikely to have found the best one.

To me, it seems intuitive that as you build your model, you would want to look at the things it does badly and try to find features that would help it to improve, or take out features that don't seem to be helping.

Although I've seen this done in practice and muddled through this way I've never seen any formal theory behind it. How do you know which features to add to the set your training on? and WHich to remove?

  • $\begingroup$ "as you build your model, you would want to look at the things it does badly and try to find features that would help it to improve" FYI that makes me think of boosting. It's not about improving by adding more features, but improving by adding more models that take care of what the initial model does badly (by training on the residuals), probably using a different subset of features. See en.m.wikipedia.org/wiki/Boosting_(machine_learning). $\endgroup$
    – asac
    Aug 19, 2021 at 7:20

1 Answer 1


There are various features selections techniques. The most common techniques rank individual features by how much information they bring with respect to the target, for example with information gain or conditional entropy.

Techniques based on individual features are efficient (i.e. fast) and usually help to reduce dimensionality and improve performance. But they are not necessarily optimal, because they cannot take into account the contribution of a subset of features together. For example they might select several features which are highly correlated between each other, even though selecting only one of them would be enough.

In order to take into account how features interact, ideally one would train and test a model with every possible subset of features, and then select the best one. However the full exploration of $2^N$ subsets is rarely feasible, but some optimization methods can be used, for example feature selection with genetic learning.

Note that there are also features extraction techniques. In this case the original semantic of the features is not preserved, since the whole set of features are transformed into a new representation.

  • $\begingroup$ What are the python package you'd recommend for these? $\endgroup$
    – Abijah
    Aug 18, 2021 at 16:55
  • 1
    $\begingroup$ @Abijah sklearn proposes a range of options. What I called "based on individual features" is called "univariate feature selection" in their doc. For genetic feature selection I can see that there are some packages like here and here. I didn't test any of these. $\endgroup$
    – Erwan
    Aug 18, 2021 at 18:27
  • $\begingroup$ What about using PCA ?which technique whould be best ? $\endgroup$
    – Malo
    Sep 5, 2021 at 8:25
  • $\begingroup$ @Malo PCA can be used as a feature extraction method, i.e. to reduce the number of features in an unsupervised way. It's important to distinguish feature selection (supervised, preserves the meaning of the features) and feature extraction (unsupervised, doesn't preserve the meaning). $\endgroup$
    – Erwan
    Sep 5, 2021 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.