1
$\begingroup$

My questions are:

  1. Can we use "pca feature selection" for supervised classification? What will happen to labels when we use dimension reduction?
  2. If I understand it right when we use pca for feature extraction after do it we have old features but in new dimension then can we say these are the new features or mixed features?
  3. Is "feature extraction" different with "feature selection" in pca? I mean they are two things to do with different ways.
  4. How can select effective features with pca? There is a lots of references on the internet but I couldn't find any MATLAB sample code or video (on youtube) about pca feature selection, I mean not in theory, I'm looking for sample code in any language(Paython, Matlab,etc).
$\endgroup$
1
$\begingroup$

Let $X\in\mathbb{R}^{n\times m}$ be your data and $Z=XT\in\mathbb{R}^{n\times k}$ be the PCA transformed data.

Can we use "pca feature selection" for supervised classification? What will happen to labels when we use dimension reduction?

Yes. It is commonly used for feature selection and linear dimensionality reduction. Nothing happens to the labels. At the start you have training pairs $(x_i,y_i)$, and after PCA you have $(z_i,y_i)$ instead.

If I understand it right when we use pca for feature extraction after do it we have old features but in new dimension then can we say these are the new features or mixed features?

You have new features which are simply linear combinations of the old features. Another way to see it is that you chose a new set of (orthogonal) axes in the data space (potentially throwing away a few of them), and then projected your data onto this new subspace. So $Z$ is (literally) a mixture of the features from $X$.

Is "feature extraction" different with "feature selection" in pca? I mean they are two things to do with different ways.

It can be both or either. To me, feature extraction implies obtaining latent descriptors from raw data. So if PCA is run on raw data, then it is doing feature extraction. Otherwise, it is not. As for feature selection, I would say whenever you remove any principal components (i.e., performing linear dimensionality reduction), then you are doing feature selection because you are selecting some features but not others.

How can select effective features with pca? There is a lots of references on the internet but I couldn't find any MATLAB sample code or video (on youtube) about pca feature selection, I mean not in theory, I'm looking for sample code in any language(Paython, Matlab,etc).

People tend to either (a) choose a fixed variance to explain based on the normalized singular values (e.g., 90% of the original variance is explained) or (b) treat the number of selected features ($k$) as a hyper-parameter, and cross-validate over a validation set with it. Some discussion on the explained variance and its connection to the reconstruction error are here or here.

As for practical code samples, the scikit-learn library has some simple examples:

from sklearn import decomposition, datasets

k = 3
iris = datasets.load_iris()
X = iris.data
Y = iris.target
pca = decomposition.PCA(n_components=k)
pca.fit(X)
Z = pca.transform(X)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.