12
$\begingroup$

How to choose K for PCA? K is the number of dimensions to project down to. The only requirement is to not lose too much information. I understand it depends on the data, but I'm looking more for a simple general overview about what characteristics to consider when choosing K.

$\endgroup$
  • $\begingroup$ Depends on the tolerable data loss, and also on the problem statement too! $\endgroup$ – Dawny33 Mar 16 '16 at 4:50
  • $\begingroup$ I agree with the two answers below. However, do you know there is a simple way to quantify the information loss, i.e., using the diagonal of SVD of the covariance matrix? $\endgroup$ – yuqian Mar 17 '16 at 23:24
13
$\begingroup$

After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components removed and see if it makes an 'elbow' where it makes sense. A lot of this depends on your use case though.

$\endgroup$
  • $\begingroup$ (+1) Yeah, as simple as that :) $\endgroup$ – Dawny33 Mar 16 '16 at 14:00
3
$\begingroup$

I normally check for percentage of the information held by K value. Let's say out of 8 fields, 2 of them hold 90% of the information. Then there is no point in including the other 6 or 5 fields. If u know mnist data, out of 768 input, I only used 250, which bumped my accuracy from 83 to 96%. The fact is more dimensionality brings more problem. So cut them off. I usually only take K who only hold 90% of the info, and it works for me.

$\endgroup$
  • $\begingroup$ Hi.. I have a similar problem where I'd like to use x% of information and unsure how to do this? I intend to use the IPCA to do this I can leave the n_components=None but how do I then Decide what are the features that have x% of the data ? $\endgroup$ – Arsenal Fanatic Aug 11 '16 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.