Doing correctly paired-trial validation

Question

In paired-trial validation, a statistical (ML) models are trained on $n$ datasets separately and then applied to other datasets, as a way of estimating the generalization of the models obtained. Typically, when training a model on a single dataset, we would separate the dataset in a train and test sets, so that we can test the model on the data coming from the same source. In case of paired-trial validation this leaves us with possibility of splitting every dataset into train and test parts, and testing the models only on the test parts of the other datasets, or testing on the full datasets, and even training on full datasets.

I wonder what is the consistent way of doing it - i.e., what potential pitfalls or typical errors one should avoid.

Similar questions arise in the context of the leave-one-dataset-out approach, where the model is trained on all the datasets except one, and then applied to the dataset withheld.

Suggestions for solid background reading on the subject would be greatly appreciated as well.

Answer 1

You are describing variants of the well known k-fold CV.

When presented with training folds, a specified model learns parameters.

Part of that specification is the hyperparameters we choose, either per model or across models. Picking e.g. SVM vs. Decision Tree is a similar kind of choice, limiting the model's behavior to a certain hypothesis class.

Suppose we have already performed data cleaning and ETL to our satisfaction, and we do not return to such steps, keeping a single frozen input dataset.

Typically we will iterate through model evaluation and choosing new hyperparameters, either manually or via a technique like grid search. It is critically important to avoid data leakage at this point, since what we really care about is not historic statistics but rather the performance on tomorrow's unseen data.

potential pitfalls or typical errors

Abu-Mostafa et al. in Learning from Data warn against such leakage, or Data Snooping (slides, pp 18 & 21).

Moral of the story: Retain a "hold-out" fold of data which you never consult during the model development process. Produce a model you're happy with. Evaluate it against the hold-out fold at the last moment, just before submitting your paper for publication. That's the accuracy measure that really matters.

As a practical matter you may wish to begin the development process by creating multiple hold-out folds, since after a poor evaluation result you may decide to return for another development cycle.