How to choose K for PCA? K is the number of dimensions to project down to. The only requirement is to not lose too much information. I understand it depends on the data, but I'm looking more for a simple general overview about what characteristics to consider when choosing K.
After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components removed and see if it makes an 'elbow' where it makes sense. A lot of this depends on your use case though.
I normally check for percentage of the information held by K value. Let's say out of 8 fields, 2 of them hold 90% of the information. Then there is no point in including the other 6 or 5 fields. If u know mnist data, out of 768 input, I only used 250, which bumped my accuracy from 83 to 96%. The fact is more dimensionality brings more problem. So cut them off. I usually only take K who only hold 90% of the info, and it works for me.