What is the difference between feature selection and feature reduction?
When do we use feature selection and what happens when we don't use it? How is this different than feature reduction?
Feature selection refers to deciding on what features to include in model training.
Feature reduction refers to assigning weights to features regarding how important they are for training. In linear regression, this is known as Ridge Regression (shrinking features based on L2 norm), Elastic Net, and Lasso (shrinking based on L1 norm). Lasso can shrink features to zero, so that they are effectively excluded from training (this would be similar to selection iff features are shrunken to zero).
Ridge and Lasso are very well described in the book „Introduction to Statistical Learning“. If you want to know more about Ridge/Lasso, have a look there.
The feature selection is thinking of which subset of features (columns) matter in determining the outcome (y column). Therefore, after analyzing the features in each case (dataset) we would find out some of the features have nothing to do with the output, meaning, they don't have any effect on what the dependent feature (y) is. hence, the features with a higher contribution for predicting the outcome are selected.
One reason that leads to feature selection is the
curse of dimensionality. In fact, if you have many features in your dataset, the number of your required samples (record/ data rows) for having a proper model and anticipation grows exponentially. So, we choose the most relevant features for the outcome.
Now for choosing the best features that have a higher impact on the outcome (y), there are many approaches. one approach can be using heatmap for perceiving the correlation between variables.
Another approach of feature selection is feature reduction. And there are a bunch of techniques for feature reduction like
backward elimination, forward selection, etc. In the backward elimination for instance, first, the model is trained on the whole features of the dataset, then the one with the highest
p-value is omitted, again the model is trained on the remained columns and this process continues until there is no feature with a
p-value higher than the threshold.
Feature selection: -heatmap correlation matrix -feature reduction -backward elimination -forward selection -bidirectional selection -... -...
When you say feature reduction i guess that you mean dimensionality reduction. The difference between feature selection and dimensionality reduction is that the set of features created by feature selection must be a subset of the original set of features, and the set created by dimensionality reduction doesn't have to be a subset of the original set of features (e.g. see PCA algorithm for dimensionality reduction)