First of sorry for asking a possibly beginner question, but i don't understand pca seems to be the same as feature selection, but when i search online they seems to be talked differently. What people usually say is PCA for reducing dimensionality and feature selection for selecting feature. Didn't those two do the same thing? Reducing number of attributes/features/dimension? Please kindly help me understand. Thank you
This answer emphasizes an intuitive understanding since the OP is a beginner. (1) PCA can be used for Feature Selection, in a special case, when the features are already uncorrelated and the 'relevant' features are embedded in a lower-dimensional sub-space.
(2) PCA can be used for Feature Extraction, when the features are correlated. Based on variance of the data among the transformed features, we may now choose to do Feature Selection from among the transformed features.
Figure 1 should elucidate these idea. There are 5 'original' features in the data (x1,...,x5). First compute a covariance matrix. Note that, if the original features were completely uncorrelated, then the covariance matrix would be a diagonal matrix, where the values on the principal diagonal are equal to the variance in each dimension. See Figure 2.
The next step is Eigenvector analysis of the covariance matrix. The Eigenvectors provide the transformed features. These transformed features have lower correlation (assume uncorrelated for simplicity) than the original features. We can now choose to do Feature Selection but picking the transformed features based on the variance. The Eigenvalues provide this variance. If the data happens to be embedded in a sub-space, which is the case in this hypothetical example, we simply pick the Eigenvector(s)/Transform Feature(s) with distinctly highest Eigenvalue(s)/Variance(s).
To round up your understanding of covariance, correlation, eigenvector analysis, consider the follow hypothetical example of data with 3 original features/traits A, B, and C. The example is illustrated in Figure 3.
In Figure 3:
(A) . Covariance matrix for three traits A , B , and C . The diagonal elements are the variances, and the off-diagonal elements are the covariances. (B) . Correlation matrix for the same three traits. Off-diagonal elements are the product-moment correlations among the traits. (C) . Eigenvalues for the principal components (eigenvectors) of the covariance matrix (diagonal elements). Notice that the covariances among the three principal components are all zero (off-diagonal elements) and that the sum of the eigenvalues is equal to the sum of the variances in A. (D) . Eigenvectors of the covariance matrix. These numbers can be thought of as loadings of each trait on each principal component, or as angle (in radians) that each principal component must be turned to be aligned with the original variable axes. (E) . The percentage of the sum the original variances explained by each principal component. (F) . The spatial relationships between data, trait axes, and principal components. Traits A , B , and C are correlated, so lie in a linear cloud in the 3D space formed by their trait axes. The principal components are the major axes through those data, lying at angles to the original trait axes described by the eigenvectors in D and having variances along each principal component axis described by the eigenvalues in C.
PCA reduces dimensionality by generating orthogonal (i.e. uncorrelated) "new" features from the original features. This can be very useful, e.g. when you do linear regression. So principle components "transform" the original features to orthogonal ones. You can see this as some kind of feature engineering. Usually PCA will reduce the number of features which can be very helpful. See Ch. 10.2 in ISL.
Feature selection basically means to choose features which perform well in some model (e.g. have high explanatory power) or exclude features which are noisy. This can be any kind of features (transformed or original features). So feature selection is a very broad term and may be linked to various different methods.