I have a methodology question: are hold-out and CV generalization-optimization techniques mutually exclusive? It gets really confusing to me at times, because in the most recent project I have been doing something as follows:

  • I have split the dataset into TRAIN and TEST sets (with stratified distribution)
  • I have applied gridsearchCV on the TRAIN set with cv = 10, so effectively the model was splitting TRAIN set into TRAIN' and VALIDATION sets at each fold
  • I have used the optimized trained model with the TEST set, and ended up with results 0.97 (TRAIN set) vs 0.68 (TEST set).

Normally, when I see results like that my immediate reaction would be to assume that I am overfitting to the TRAIN set, and that the model does not generalize well enough. However, I have already used CV with the TRAIN set to make sure I was tuning the model to generalize as well as possible. I do not have any immediate tools (nor I should touch the already trained model) to improve this score now, since as I understand Hold-out technique, it just gives me the final score of how my model performs on previously unseen data.

At the same time, it feels kind of strange, since with using both CV and Hold-out, I am guaranteed to get some outliers in the hold-out set (TEST in my text above), which will fail.

Any suggestions or ideas?


You are correct in the sense that when tuning your model via. grid search you are technically not leaking any data. But, recall that tuning your model (via. a specific procedure such as grid search) is one of only many steps you probably took in fitting your model pipeline. In particular, areas such as pre-processing, feature engineering, imputation, model tuning, data aggregations, etc. The point of the test set is to capture the entire model building process and not just the process of model tuning.

Furthermore, it is highly known that validation scores reported during model tuning tend to be optimistically biased (and this bias tends to be worse with smaller datasets). This is because the probability of finding a set of hyper parameters that coincidentally minimizes the error for the validation set but not to the overall population (i.e., overfitting to the validation set) becomes higher the finer your grid is. Imagine theoretically tuning your model to one million different hyper parameter combinations. The probability of selecting a bogus set of hyper parameters (i.e. that are only optimal for the validation set) is now quite large due to the sheer number of possible candidates you have elected to try.

With a test set, this still won't prevent overfitting to a validation set. However, it will allow you to detect the problem and give you the true unbiased measure of model performance.

At the same time, it feels kind of strange, since with using both CV and Hold-out, I am guaranteed to get some outliers in the hold-out set (TEST in my text above), which will fail.

Indeed, which is a major drawback of cross validation and data splitting especially for smaller datasets (with a lot of outliers/noise). Basically, the performance measure you observe tends to be highly variable with how you split the data in the first place (that is, the seed you choose when splitting your data can lead to large changes in estimated model performance depending on where your outliers fall). The solution is unfortunately, not very glamorous and time consuming. In order to gain more certainty in our estimate, we need more than just a single estimate of model performance. Thus, simply repeat the entire model building process again with a different data partition (a different test set). Repeat this however many times, and average over all repeats. Possibly, form a confidence interval that allows you to see for yourself how variable your model's performance is.

I will also note that there are other ways to get around this problem, such as "optimism adjusted bootstrap", but recent issues have arisen with this method which have potentially shown that for high dimensional data this method does not do well, despite being more efficient than cross validation. Since high dimensional data is the norm these days, I have my doubts but perhaps it may be of use to you.

  • $\begingroup$ Thank you arangol - I have just noticed your reply. Indeed, my dataset is rather small (~100 targets with 2000 samples), so I have decided to go with a full-on CV approach. As my sample size will keep on growing, I might change my approach, but for now it was a choice between constantly low TEST scores, or a working model where I can use prediction probability scores to distinguish likely good predictions from the rubbish ones. $\endgroup$ – Greem666 Jul 3 '19 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.