ChatGPT already may give you a very good answer:
While k-fold cross-validation provides a good estimate of a model's
performance on unseen data, it does not completely replace the need
for a final test set. The reason for this is that during the model
development process, it is possible to make choices that inadvertently
overfit the model to the validation data, even when k-fold
cross-validation is used.
For example, you may try different models or hyperparameters on the
k-fold validation set until you find the best combination, and you may
select the model that performs best on the validation set without
realizing that the model has been overfitted to the validation set.
The final test set is then used to evaluate the performance of the
model on completely unseen data, which provides a more accurate
estimate of how well the model will perform in the real world.
Moreover, the final test set can also be used to compare the
performance of different models that have been tuned using k-fold
cross-validation. This is because the models may perform similarly on
the validation set but have different performance on the test set,
which can reveal important differences in their generalisation
Nonetheless I would add some fact that I have almost never been taken into account during the validation stage and that from my experience should be done. (My little advantage over ChatGPT lol)
Usual train test split and CV uses random split which is supposed to measure the model-s ability to predict on unseen data. Nonetheless, this approach loses important information, which is the time componen of the data generation.
In practically every supervised problem, you have data generated at different times, so the complete model evaluation should be done so that you can guarantee that the model is capable of predicting unseen samples but also unseen and more recent ones.
Below is shown a picture that depicts how the data should be split so that we can test the point mentioned previously.
So for having such splits, you should
- Fix a training end date
- Create a random split only on the data that is inside the training period so that you have an
in-time test set
- All the data after the training period is also a test set, but this is an
out time test set.
In that way you are evaluating non only your model performance but how resilient your model is to the natural change of the distribution of the features over time.
Hopefully, you see that the test set was not as redundant as initially seemed.
I hope it helps!