Feature selection vs Feature extraction. Which to use when?

Question

Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am right.

Which technique would be preferred over the other and when?

I was thinking, since feature selection does not modify the original data and it's properties, I assume that you will use feature selection when it's important that the features you're training on be unchanged. But I can't imagine why you would want something like this..

Answer 1

Adding to The answer given by Toros,

These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember)

• feature extraction and feature engineering: transformation of raw data into features suitable for modeling;

• feature transformation: transformation of data to improve the accuracy of the algorithm;

• feature selection: removing unnecessary features.

Just to add an Example of the same,

Feature Extraction and Engineering(we can extract something from them)

• Texts(ngrams, word2vec, tf-idf etc)
• Images(CNN'S, texts, q&a)
• Geospatial data(lat, long etc)
• Date and time(day, month, week, year, rolling based)
• Time series, web, etc
• Dimensional Reduction Techniques (PCA, SVD, Eigen-Faces etc)
• Maybe we can use Clustering as well (DBSCAN etc)
• .....(And Many Others)

Feature transformations(transforming them to make sense)

• Normalization and changing distribution(Scaling)
• Interactions
• Filling in the missing values(median filling etc)
• .....(And Many Others)

Feature selection(building your model on these selected features)

• Statistical approaches
• Selection by modeling
• Grid search
• Cross Validation
• .....(And Many Others)

Hope this helps...

Do look at the links shared by others. They are Quite Nice...

Answer 2

As Aditya said, there are 3 feature-related terms that sometimes are confused with each other. I will try and give summary explanation to each one of them:

• Feature extraction: Generation of features from data that are in a format that is difficult to analyse directly/are not directly comparable (e.g. images, time-series, etc.) In the example of a time-series, some simple features could be for example: length of time-series, period, mean value, std, etc.
• Feature transformation: Transformation of existing features in order to create new ones based on the old ones. A very popularly used technique for dimensionality reduction is Principal Component Analysis (pca) that uses some orthogonal transformation in order to produce a set of linearly non-correlated variables based on the initial set of variables.
• Feature selection: Selection of the features with the highest "importance"/influence on the target variable, from a set of existing features. This can be done with various techniques: e.g. Linear Regression, Decision Trees, calculation of "importance" weights (e.g. Fisher score, ReliefF)

If the only thing you want to achieve is dimensionality reduction in an existing dataset, you can use either feature transformation or feature selection methods. But if you need to know the physical interpretation of the features you identify as "important" or you are trying to limit the amount of data that need to be collected for your analysis (you need all the initial set of features for feature transformation), then only feature selection can work.

You can find more details on Feature Selection and Dimensionality Reduction in the following links:

• A summary of Dimension Reduction methods

• Classification and Feature Selection: A Review

• Relevant question and answers in Stack Overflow

Answer 3

I think they are 2 different things,

Lets start with Feature Selection:

This technique is used for selecting the features which explain the most of the target variable(has a correlation with the target variable).This test is ran just before the model is applied on the data.

To explain it better let us go by an example: there are 10 feature and 1 target variable, 9 features explain 90% of the target variable and 10 features together explains 91% of the target variable. So the 1 variable is not making much of a difference so you tend to remove that before modelling(It is subjective to the business as well). I can also be called as Predictor Importance.

Now lets talk about Feature Extraction,

Which is used in Unsupervised Learning,extraction of contours in images, extraction of Bi-grams from a text, extraction of phonemes from recording of spoken text. When you don't know anything about the data like no data dictionary, too many features which means the data is not in understandable format. Then you try applying this technique to get some features which explains the most of the data. Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.

You can apply Feature Extraction on the given data to extract features and then apply Feature Selection with respect to the Target Variable to select the subset which can help in making a good model with good results.

you can go through these Link-1,Link-2 for better understanding.

we can implement them in R, Python, SPSS.

let me know if need any more clarification.

Answer 4

The two are very different: Feature Selection indeed reduces dimensions, but feature extraction adds dimensions which are computed from other features.

For panel or time series data, one usually has the datetime variable, and one does not want to train the dependent variable on the date itself as those do not occur in the future. So you should eliminate the datetime: feature elimination.

On the other hand, weekday/weekend day may be very relevant, so we need to compute the weekday status from the datetime: feature extraction.

Answer 5

A critical part of the success of a Machine Learning project is coming up with a good set of features to train on. This process, called feature engineering, involves:

• Feature selection: selecting the most useful features to train on among existing features.
• Feature extraction: combining existing features to produce a more useful one (as we saw earlier, dimensionality reduction algorithms can help).
• Creating new features by gathering new data

Quoting : "A Hands on Machine Learning with SciKit-Learn, Keras & Tensorflow - Aurelien Geron"