0
$\begingroup$

In the following code run after PCA i can see that X number of components explain Y % of cumulative explained variance (CEV). I would like to know

1- What percentage of the CEV is typically acceptable e.g. 95% or 99%? (Or is it a case by case basis?)

2- If 20 out of 200 components explain 95% of the CEV, what does this say about my data, what about when 200 out of 200 explain 95% ?

pca = PCA().fit(X_train)
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of components')
plt.ylabel('cumulative explained variance');
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I always think that Principal component analysis is a very interesting tool. Many know its applications, few knows exactly what is going on. If you care about the math, you need to study eigen values and eigen vectors, but here let me explain pca in a very simple way. Imagine that you have a point in a 2-d space (x, y), this same point can be represented by using another two values, let us say (r, theta), where r is the distance from the origin and theta is the angle with respect to the +v x-axis. Given any point (x,y), you can find the point (r, theta) corresponding to the point (x,y). You should be able to move between the two coordinates easily. Now let us ask the following question: If I decide to ignore the theta dimension (for example set it to zero) how many points in the (x,y) you still can recover? All the points across the +ve x-axis can be recovered correctly. Now what about the r dimension, if we set it to zero, we can recover only one point which is the origin point.

Principal component analysis helps find this new coordinate system depending on the data, number of dimensions and variation across each dimension. The higher variation across dimension the more important to reserve this dimension. Very important to know that pca proposes both new coordinates and a measure of how much each dimension is important to recover the points in the original coordinate.

For example, let us take these points, (1,1) (2,2) (3,3)

Both x and y dimensions are important and they experience the same variation. If we transform, these data points into a new coordinate system (k,m), such that K=x. And. M=0 Then. K is very important 100% and m is not important at all 0% You can recover the original points by using, x=k. And y=k. Note that the variation across the m dimension is zero and it is high across the k dimension.

How much you want to reserve depends mainly on the application, usually > 95% is good. For the second question, it is possible to recover less than 100% when you use less dimensions, but when you use 200 of 200 you should be able to recover 100%.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Just a comment (please correct me if I'm wrong): the standard (not kernelized) PCA cannot actually come up with an $(x,y) \rightarrow (r, \theta)$ transformation, it only performs linear transformations. $\endgroup$
    – Romain Reboulleau
    Oct 11, 2018 at 18:39
  • $\begingroup$ Also, in your small example, the first principal component eigen vector would actually be $(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$, because components must preserve the total variance and be orthogonal. $\endgroup$
    – Romain Reboulleau
    Oct 11, 2018 at 18:43
  • $\begingroup$ Yes, In the example I gave, I did not really find the pca for the data points, I am just trying to explain the idea with synthetic numbers $\endgroup$
    – Bashar Haddad
    Oct 12, 2018 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.