A1. What is dimensionality reduction:
If you think of data in a matrix, where rows are instances and columns are attributes (or features), then dimensionality reduction is mapping this data matrix to a new matrix with fewer columns. For visualization, if you think of each matrix-column (attribute) as a dimension in feature space, then dimensionality reduction is projection of instances from the higher dimensional space (more columns) to a lower dimensional sub-space (fewer columns).
Typical objective for this transformation is (1) preserving information in the data matrix, while reducing computational complexity; (2) improving separability of different classes in data.
A2. Dimensionality reduction as feature selection or feature extraction:
I'll use the ubiquitous Iris dataset, which is arguably the 'hello world' of data science. Briefly, the Iris dataset has 3 classes and 4 attributes (columns). I'll illustrate feature selection and extraction for the task of reducing Iris dataset dimensionality from 4 to 2.
I compute pair-wise co-variance of this dataset using library in Python called seaborn. The code is: sns.pairplot(iris, hue="species", markers=["o", "s", "D"]) The figure I get is
I can select the pair of attributes (2 dimensions) that provide me the greatest separation between the 3 classes (species) in the Iris dataset. This would be a case of feature-selection.
Next up is feature extraction. Herein, I am projecting the 4-dimensional feature space of Iris to a new 2-dimensional subspace, which is not axis aligned with the original space. These are new attributes. They are typically based on the distribution in the original high dimensional space.
The most popular method is Principal Component Analysis, which computes Eigenvectors in the original space.
Obviously, we are not restricted to using only a linear and global projection to a subspace based on Eigenvectors. We can use non-linear projection methods as well.
Here is an example of non-linear PCA using neural networks
The attributes (dimensions) in the last example are extracted from the original 4 attributes using neural networks. You can experiment with various flavors of PCA for iris dataset youself using this pca methods code.
While feature extraction methods may appear to be superior in performance to feature selection, the choice is predicated by the application. The attributes from feature extraction typically lose physical interpretation, which may or may not be an issue based on the task at hand. For example, if you are designing a very expensive data collection task with costly sensors and need to economize on the attributes (number of different sensors), you'd want to collect a small pilot sample using all available sensors and then select the ones that are most informative for the big data collection task.