1
$\begingroup$

Principal Component Analysis is a means to reduce the dimensionality of data, if I understand correctly.

So if I have a 1000 sample point 12 dimensional matrix and reduce it to a 1000 sample point 2 dimensional one, then are the values of the sample points themselves changed in some way? or are simply 10 dimensions/columns thrown out and 2 of the original remain?

$\endgroup$
  • 2
    $\begingroup$ Welcome to DataScience.SE! You are running into the difference between feature selection (selecting a subset of the original features) and feature extraction (deriving new ones). $\endgroup$ – Emre Jul 7 '16 at 17:33
  • $\begingroup$ Glad to be here! So PCA falls into feature selection right? Because feature extraction varies from field by field,application to application, if I'm not wrong? $\endgroup$ – MyBushisaNeonJungle Jul 8 '16 at 23:46
  • 1
    $\begingroup$ No, feature extraction, because the principal components returned by PCA are functions of the input features, not usually a subset of them. I think this will become more clear if you apply PCA to some data and see what the principal components are. $\endgroup$ – Emre Jul 8 '16 at 23:48
  • $\begingroup$ @Emre Ok. So PCA is feaure extraction? Wow, I guess you learn something new everyday! $\endgroup$ – MyBushisaNeonJungle Jul 9 '16 at 8:41
  • $\begingroup$ Related: Does dimension reduction always lose some information? $\endgroup$ – smci Dec 13 '16 at 0:36
2
$\begingroup$

Yes, the new 2 dimensional values will be a projection of original 12 dimensional points onto the two principle components (vectors).

please refer to the first figure in this clear tutorial: http://lazyprogrammer.me/tutorial-principal-components-analysis-pca/

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have a bad habit, but I like to confirm things, so you're saying PCA does in fact changes values of data right? Does this ever cause any significant loss ever? $\endgroup$ – MyBushisaNeonJungle Jul 8 '16 at 23:48
  • 1
    $\begingroup$ Of course it can cause significant loss. Any dimensionality reduction can cause loss, or we would always reduce data to 0 dimensions. $\endgroup$ – Has QUIT--Anony-Mousse Jul 9 '16 at 1:42
  • $\begingroup$ @Anony-Mousse if we did that then there would be no data to learn from, and reducing too many dimensions may lead to subsequent overfitting. $\endgroup$ – MyBushisaNeonJungle Jul 9 '16 at 8:38
  • 1
    $\begingroup$ Therefore, every dimensionality reduction will be lossy at some point (or not reduce dimensionality anymore). Proof complete. $\endgroup$ – Has QUIT--Anony-Mousse Jul 9 '16 at 9:39
1
$\begingroup$

PCA is a transform: it creates new (transformed) features from the original data. In general if you choose fewer dimensions (e.g. you chose to reduce m=12 -> n=2 dimensions), it's lossy and will throw away some of in the information content of the original data. The higher n is, the less you lose, and for m=n, you preserve all the original information (although you still do a vector transform on the data, so the extracted features are != the original data).

It was your (arbitrary) decision to choose the parameter n=2 (number of Principal Components), you could try other values or explore a range. You could have chosen n=5, n=9, or even the maximum possible: n=12.

For standard rules-of-thumb on how to choose n, see e.g. Choosing number of principal components to retain

(Scree plot, Proportion of total variance explained, Average eigenvalue rule, Log-eigenvalue diagram, etc.)

where a Scree Plot is a simple line-segment plot that shows the fraction of total variance in the data as explained or represented by each PC. Usually the scree plot will have a knee where the number of PCs explains most of the variance, and if so that might suggest you an upper bound on n.

There are other rules-of-thumb discussed there too. You can find tons of articles on this subject.

See also e.g. How many principal components to take?

| improve this answer | |
$\endgroup$
0
$\begingroup$

Yes. PCA changes the values of the data. It transforms the data and projects it into a new dimension.

This video is good to learn about PCA

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.