What is the point of final test set in K-fold cross-validation?

Question

I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, though: can't the performance of the model be evaluated based on the kfold validation (i.e. the metrics from the 10 resulting folds?). Because isn't the whole point of kfold that the model doesn't actually "see" the test data? And so a final test set seems redundant, but it also seems as if it is common practice, and I am wondering why? I've seen some mention of hyperparameter overtuning/overfitting, etc., but logistic regression doesn't have many hyperparameters to tune, anyway.

So what is the point of the final test set, and do I need it, or are the performance metrics given from each of the 10 folds sufficient to evaluate the model's performance?

Answer 1

ChatGPT already may give you a very good answer:

While k-fold cross-validation provides a good estimate of a model's performance on unseen data, it does not completely replace the need for a final test set. The reason for this is that during the model development process, it is possible to make choices that inadvertently overfit the model to the validation data, even when k-fold cross-validation is used.

For example, you may try different models or hyperparameters on the k-fold validation set until you find the best combination, and you may select the model that performs best on the validation set without realizing that the model has been overfitted to the validation set. The final test set is then used to evaluate the performance of the model on completely unseen data, which provides a more accurate estimate of how well the model will perform in the real world.

Moreover, the final test set can also be used to compare the performance of different models that have been tuned using k-fold cross-validation. This is because the models may perform similarly on the validation set but have different performance on the test set, which can reveal important differences in their generalisation ability.

Nonetheless I would add some fact that I have almost never been taken into account during the validation stage and that from my experience should be done. (My little advantage over ChatGPT lol)

Usual train test split and CV uses random split which is supposed to measure the model-s ability to predict on unseen data. Nonetheless, this approach loses important information, which is the time componen of the data generation.

In practically every supervised problem, you have data generated at different times, so the complete model evaluation should be done so that you can guarantee that the model is capable of predicting unseen samples but also unseen and more recent ones.

Below is shown a picture that depicts how the data should be split so that we can test the point mentioned previously.

So for having such splits, you should

1. Fix a training end date
2. Create a random split only on the data that is inside the training period so that you have an in-time test set
3. All the data after the training period is also a test set, but this is an out time test set.

In that way you are evaluating non only your model performance but how resilient your model is to the natural change of the distribution of the features over time.

Hopefully, you see that the test set was not as redundant as initially seemed.

I hope it helps!