About

Feature selection, also called attribute selection or feature reduction, refers to techniques for identifying a subset of features of a data set that are relevant to a given problem. By removing irrelevant and redundant features, successful feature selection can avoid the curse of dimensionality and improve the performance, speed, and interpretability of subsequent models.

Feature selection includes manual methods (such those those based on domain knowledge) and automatic methods. Automatic methods are often categorized into filter, wrapper, and embedded approaches.

Filter approaches perform feature selection as a separate preprocessing step before the learning algorithm. Filter approaches thus look only at the intrinsic properties of the data. Filter methods include Wilcoxon rank sum tests and Correlation based tests.

Wrapper approaches uses performance of a learning algorithm to select features. A search algorithm is “wrapped” around the learning algorithm to ensure the space of feature subsets is adequately searched. As such, wrapper methods can be seen as conducting the model hypothesis search within the feature subset search. Examples of wrapper approaches are simulated annealing and beam search.

Embedded approaches incorporate variable selection as a part of the training process, with feature relevance obtained analytically from the objective of the learning model. Embedded methods can be seen as a search in the combined space of feature subsets and hypotheses. Examples of embedded approaches are boosting and recursive ridge regression.