2
$\begingroup$

Principal Component Analysis is a means to reduce the dimensionality of data, if I understand correctly.

So if I have a 1000 sample point 12 dimensional matrix and reduce it to a 1000 sample point 2 dimensional one, then are the values of the sample points themselves changed in some way? or are simply 10 dimensions/columns thrown out and 2 of the original remain?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Welcome to DataScience.SE! You are running into the difference between feature selection (selecting a subset of the original features) and feature extraction (deriving new ones). $\endgroup$
    – Emre
    Commented Jul 7, 2016 at 17:33
  • $\begingroup$ Glad to be here! So PCA falls into feature selection right? Because feature extraction varies from field by field,application to application, if I'm not wrong? $\endgroup$
    – MyBushisaNeonJungle
    Commented Jul 8, 2016 at 23:46
  • 1
    $\begingroup$ No, feature extraction, because the principal components returned by PCA are functions of the input features, not usually a subset of them. I think this will become more clear if you apply PCA to some data and see what the principal components are. $\endgroup$
    – Emre
    Commented Jul 8, 2016 at 23:48
  • $\begingroup$ @Emre Ok. So PCA is feaure extraction? Wow, I guess you learn something new everyday! $\endgroup$
    – MyBushisaNeonJungle
    Commented Jul 9, 2016 at 8:41
  • $\begingroup$ Related: Does dimension reduction always lose some information? $\endgroup$
    – smci
    Commented Dec 13, 2016 at 0:36

4 Answers 4

Reset to default
2
$\begingroup$

Yes, the new 2 dimensional values will be a projection of original 12 dimensional points onto the two principle components (vectors).

please refer to the first figure in this clear tutorial: http://lazyprogrammer.me/tutorial-principal-components-analysis-pca/

Improve this answer
$\endgroup$
4
  • $\begingroup$ I have a bad habit, but I like to confirm things, so you're saying PCA does in fact changes values of data right? Does this ever cause any significant loss ever? $\endgroup$
    – MyBushisaNeonJungle
    Commented Jul 8, 2016 at 23:48
  • 1
    $\begingroup$ Of course it can cause significant loss. Any dimensionality reduction can cause loss, or we would always reduce data to 0 dimensions. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jul 9, 2016 at 1:42
  • $\begingroup$ @Anony-Mousse if we did that then there would be no data to learn from, and reducing too many dimensions may lead to subsequent overfitting. $\endgroup$
    – MyBushisaNeonJungle
    Commented Jul 9, 2016 at 8:38
  • 1
    $\begingroup$ Therefore, every dimensionality reduction will be lossy at some point (or not reduce dimensionality anymore). Proof complete. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jul 9, 2016 at 9:39
1
$\begingroup$

PCA is a transform: it creates new (transformed) features from the original data. In general if you choose fewer dimensions (e.g. you chose to reduce m=12 -> n=2 dimensions), it's lossy and will throw away some of in the information content of the original data. The higher n is, the less you lose, and for m=n, you preserve all the original information (although you still do a vector transform on the data, so the extracted features are != the original data).

It was your (arbitrary) decision to choose the parameter n=2 (number of Principal Components), you could try other values or explore a range. You could have chosen n=5, n=9, or even the maximum possible: n=12.

For standard rules-of-thumb on how to choose n, see e.g. Choosing number of principal components to retain

(Scree plot, Proportion of total variance explained, Average eigenvalue rule, Log-eigenvalue diagram, etc.)

where a Scree Plot is a simple line-segment plot that shows the fraction of total variance in the data as explained or represented by each PC. Usually the scree plot will have a knee where the number of PCs explains most of the variance, and if so that might suggest you an upper bound on n.

There are other rules-of-thumb discussed there too. You can find tons of articles on this subject.

See also e.g. How many principal components to take?

Improve this answer
$\endgroup$
0
$\begingroup$

Yes. PCA changes the values of the data. It transforms the data and projects it into a new dimension.

This video is good to learn about PCA

Improve this answer
$\endgroup$
0
$\begingroup$

PCA is used to find the variables that contain the most information, and to remove the ones that have little information or sometimes just noise.

If only 2 dimensions out of the initial 10 dimensions carry the most information, then only those 2 dimensions need to be used. Hence the PCA has helped in reducing the dataset's dimensions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.