How to choose K for PCA? K is the number of dimensions to project down to. The only requirement is to not lose too much information. I understand it depends on the data, but I'm looking more for a simple general overview about what characteristics to consider when choosing K.
2
-
$\begingroup$ Depends on the tolerable data loss, and also on the problem statement too! $\endgroup$– Dawny33Mar 16, 2016 at 4:50
-
$\begingroup$ I agree with the two answers below. However, do you know there is a simple way to quantify the information loss, i.e., using the diagonal of SVD of the covariance matrix? $\endgroup$– yuqianMar 17, 2016 at 23:24
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components removed and see if it makes an 'elbow' where it makes sense. A lot of this depends on your use case though.
answered Mar 16, 2016 at 5:32
$\begingroup$
$\endgroup$
1
I normally check for percentage of the information held by K value. Let's say out of 8 fields, 2 of them hold 90% of the information. Then there is no point in including the other 6 or 5 fields. If u know mnist data, out of 768 input, I only used 250, which bumped my accuracy from 83 to 96%. The fact is more dimensionality brings more problem. So cut them off. I usually only take K who only hold 90% of the info, and it works for me.
-
$\begingroup$ Hi.. I have a similar problem where I'd like to use x% of information and unsure how to do this? I intend to use the IPCA to do this I can leave the n_components=None but how do I then Decide what are the features that have x% of the data ? $\endgroup$ Aug 11, 2016 at 15:14