Could data from a test set 'leak' into predictor during PCA?

Question

After reading this article I have got a question about PCA.

Author was talking about whether to use test set while computing PCA.

But, few important points to understand:

1) We should not combine the train and test set to obtain PCA components of whole data at once. Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’. Eventually, this will hammer down the generalization capability of the model.

2) We should not perform PCA on test and train data sets separately. Because, the resultant vectors from train and test PCAs will have different directions ( due to unequal variance). Due to this, we’ll end up comparing data registered on different axes. Therefore, the resulting vectors from train and test data should have same axes.

Author mentioned "Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’."

As I understand it... test data can only affect which PC(s) we will choose ( because of changed variance) but data can't leak really( let's say only metadata leak). For each observation we still have data only from it's own previous variables(predictors) just in lower dimensionality(projected). Am I right? or during calculations of matrixes data from test set can leak into?

Answer 1

The author is right. You are wrong. There's no such thing as "only metadata leak" or "can't leak really". It's like saying "not pregnant really" -- either you're pregnant, or you're not. Same here -- either data can leak, or it can't. In this case, data can leak. Maybe only partial data, but it's hard to know just how partial and just how bad the impact of that might be. It might be very bad, or it might not; but since you can't know, testing with a methodology where you know the results might be meaningless isn't a good idea.

Answer 2

The author is right. Data Leakage is when you used additional information not available during training for training your ML model. In PCA if you use test data you are using some insights from test data as part of your training process hence data leakage.

Machine learning mastery explains data leakage as below :

Data leakage can cause you to create overly optimistic if not completely invalid predictive models.

Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in turn invalidate the estimated performance of the mode being constructed.

if any other feature whose value would not actually be available in practice at the time you’d want to use the model to make a prediction, is a feature that can introduce leakage to your model