0
$\begingroup$

I applied PCA on MNIST data and found that the first 64 components are able to retain 86% of variance.

Is there any problem while applying pca to a big dataset like MNIST. Because in most of the papers I have read to take the components which can explain upto 99% of variance. But it would be pointless to apply pca if such variance comes at 120 pca components.

$\endgroup$
1
$\begingroup$

It is entirely correct to apply PCA to a dataset like MNIST. Intuitively, corner pixels should almost never contain any information as to what digit is contained in the center of the image. So we should disregard them. You should expect similar results as with other datasets. PCA lowers the dimensionality of your data, thus allowing for a less complex model, however, this is at the cost of some information that is rejected when retaining only $n$ components.

When data is limited a less complex model will result in a lower bias and thus better accuracy when applied to novel data. However, the MNIST dataset is plentiful, thus the benefit of removing some features and losing even minimal information may cause your performance to degrade.

$\endgroup$
  • $\begingroup$ Yes as a few of the feature columns in mnist is zero $\endgroup$ – Boris Mar 23 '18 at 9:58
  • $\begingroup$ Yeah exactly, but honesty I would just leave them since your model's weights will just zero out as a consequence. A few additional features shouldn't hurt your performance when you have so much data available to you. $\endgroup$ – JahKnows Mar 23 '18 at 9:59
  • 1
    $\begingroup$ Yes but so after doing dimensionality reduction the algorithms like svm run fast $\endgroup$ – Boris Mar 23 '18 at 10:01
  • 1
    $\begingroup$ Oh yes! If you are not using deep learning, by all means PCA on MNIST is highly recommended. Precisely because of the high impact the number of features plays on the complexity of these algorithms. $\endgroup$ – JahKnows Mar 23 '18 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.