Is it correct to use non-target values of test set to engineer new features for train set?

Question

Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value.

Question: Can I (1) use feature_1 value of test set to calculate mean or (2) should I use only the train set values?

If (1) is correct than I can use the same mean for test set and train set by calculating the mean of feature_1 for all dataset. I'm not sure it's legal, because here we use information from the test set in the train set. On the other hand, we don't use target value, so it might be ok.

If (2) is correct, then, I suppose, we can't use test to calculate the mean for train set, but we can use train set feature_1 values to calculate mean for test set. But then train and test sets' means might be different and influence the correctness of the model for test set. I could use the mean of train set for test set, but again I'm not sure it's correct.

It might be irrelevant for a large dataset, because influence of each value for mean is negligible, so I suppose (1) is ok her. But if I have very small dataset of, say, less than 30 samples or, if e.g. I want to generate new feature by calculating relative value of feature_1 for each category in some categorical feature_2 by calculating mean of all the samples belonging to the same category. Then, it might turn out to be just a few samples in some category of feature_2, so that each sample would influence mean greatly.

Answer 1

Definitely (2). You should not include the testing values when calculating features over the data (or normalizing or scaling the data, etc).

Let's take a step back for a second. Why are we holding out a test set at all? Well, we're training a model on a dataset so that in the future, when we collect more data from the same domain, we can use our model to predict the target for that new data. We don't just want to do as well as possible on the data that we have, we already have the labels for it. So we hold out a test set to measure the performance on, so that we can get a sense of how well the model's predictions generalize to new data.

If we give the model information about the test set (which putting the test samples into a mean calculation would do), we'd improve the model's performance against that particular test set -- but presumably not against any new data that was collected later, when the model was actually deployed and in use! All we'd be doing is making our performance metrics less effective measures of whether or not the model generalizes.

So even while the performance would appear better, it would both possibly not reflect an actual improvement (depending on the actual distributions of train, test, and post-deployment collected data), and would harm our ability to say for sure whether our choices in model design improved our generalization performance at all.

I actually talked about this a little bit with respect to feature scaling in this answer.