I'm rather new to PCA and was hoping to have some confusion cleared up. Lets say for example we have a feature matrix that's nx100 and I want to get it down to something a bit smaller, p-dimensions, without losing too much variance.

After applying PCA and receiving and new feature matrix nxp, I would use x_reduced to predict some target variable y.

My question is, after the transformation, the new reduced feature matrix has been rotated by the eigenvectors and is sitting on a new basis. Yet, our y has not changed relative to X_reduced.

I'm unsure about how y_original and x_reduced can be used for training since y has not changed with respect to x_reduced.

Is there a way to correct for this or am I not thinking about it correctly?


The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced. While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format. You lose a lot of interpretability as far as what the actual numbers mean which is why it may seem confusing, but it's just a transformed representation of the x_original that (hopefully) contains enough of the x_original variability to make it useful.

| improve this answer | |
  • 1
    $\begingroup$ No Problem! On a side note if you want to get a test error, you should create the PCA rotations on the x_original_training, then use those same rotations on x_original_test. If you split up your data into train/test after you transform everything, then you're using data from your training set to influence your test set before the prediction occurs, which will make your test error overly-optimistic. $\endgroup$ – TBSRounder Feb 19 '16 at 15:48
  • $\begingroup$ You can also check out supervised methods, such as LDA (Linear Discriminant Analysis) $\endgroup$ – Omri374 Feb 22 '16 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.