Information leakage when train/test are truly i.i.d.?

Question

I am well aware that to avoid information leakage, it is recommended to fit any transformation (e.g., standardization or imputation based on the median value) on the training dataset and applying it to the test datasets. However. I am not clear what is the risk of applying these transformations to the entire dataset prior to train/test split if the the data is iid and the train/test split is indeed random?

For example, if the original data set has certain statistical characteristics(e.g., mean, median, and std) then I would expect a random data spilt with generate a train and test datasets that have the same statistical characteristics. Therefore, standardizing the entire datasets and then splitting should produce the same results as splitting the dataset, standardizing based on the train database and transforming the test dataset. The same argument can be made for imputation based on the median value.

Am I missing something?

Answer 1

Well, keep in mind that when you standardize/impute data you're estimating parameters. Given the conditions that you've defined and having enough data such that the estimates are good, then I don't think it should matter to use the training data or all the data (as a matter of fact, the estimate of the parameter using training data should be very similar to the parameter using all the data).

However, when the datasets are small these estimates may have high variance, and if you want your cross-validation to be reliable you might want to take it into account. In general, I think for scaling and missing imputation this should not be an issue, but if you're doing some things that are more prone to overfitting like target encoding, or imputing using a model instead of a single parameter, then you should be more careful.

In order to not even think about having enough data or not, the cross-validation framework tries to treat the test data as what the serving data should be when putting the model into production, and then you'll always be alright.