1
$\begingroup$

After reading this article I have got a question about PCA.

Author was talking about whether to use test set while computing PCA.

But, few important points to understand:

1) We should not combine the train and test set to obtain PCA components of whole data at once. Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’. Eventually, this will hammer down the generalization capability of the model.

2) We should not perform PCA on test and train data sets separately. Because, the resultant vectors from train and test PCAs will have different directions ( due to unequal variance). Due to this, we’ll end up comparing data registered on different axes. Therefore, the resulting vectors from train and test data should have same axes.

Author mentioned "Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’."

As I understand it... test data can only affect which PC(s) we will choose ( because of changed variance) but data can't leak really( let's say only metadata leak). For each observation we still have data only from it's own previous variables(predictors) just in lower dimensionality(projected). Am I right? or during calculations of matrixes data from test set can leak into?

$\endgroup$
1
$\begingroup$

The author is right. You are wrong. There's no such thing as "only metadata leak" or "can't leak really". It's like saying "not pregnant really" -- either you're pregnant, or you're not. Same here -- either data can leak, or it can't. In this case, data can leak. Maybe only partial data, but it's hard to know just how partial and just how bad the impact of that might be. It might be very bad, or it might not; but since you can't know, testing with a methodology where you know the results might be meaningless isn't a good idea.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.