2
$\begingroup$

I am carrying out logistic regression for my binary classification problem, and I have validated the model with kfold cross-validation (k=10). I don't understand why I need to have a final test set, though: can't the performance of the model be evaluated based on the kfold validation (i.e. the metrics from the 10 resulting folds?). Because isn't the whole point of kfold that the model doesn't actually "see" the test data? And so a final test set seems redundant, but it also seems as if it is common practice, and I am wondering why? I've seen some mention of hyperparameter overtuning/overfitting, etc., but logistic regression doesn't have many hyperparameters to tune, anyway.

So what is the point of the final test set, and do I need it, or are the performance metrics given from each of the 10 folds sufficient to evaluate the model's performance?

$\endgroup$

1 Answer 1

1
$\begingroup$

ChatGPT already may give you a very good answer:

While k-fold cross-validation provides a good estimate of a model's performance on unseen data, it does not completely replace the need for a final test set. The reason for this is that during the model development process, it is possible to make choices that inadvertently overfit the model to the validation data, even when k-fold cross-validation is used.

For example, you may try different models or hyperparameters on the k-fold validation set until you find the best combination, and you may select the model that performs best on the validation set without realizing that the model has been overfitted to the validation set. The final test set is then used to evaluate the performance of the model on completely unseen data, which provides a more accurate estimate of how well the model will perform in the real world.

Moreover, the final test set can also be used to compare the performance of different models that have been tuned using k-fold cross-validation. This is because the models may perform similarly on the validation set but have different performance on the test set, which can reveal important differences in their generalisation ability.

Nonetheless I would add some fact that I have almost never been taken into account during the validation stage and that from my experience should be done. (My little advantage over ChatGPT lol)

Usual train test split and CV uses random split which is supposed to measure the model-s ability to predict on unseen data. Nonetheless, this approach loses important information, which is the time componen of the data generation.

In practically every supervised problem, you have data generated at different times, so the complete model evaluation should be done so that you can guarantee that the model is capable of predicting unseen samples but also unseen and more recent ones.

Below is shown a picture that depicts how the data should be split so that we can test the point mentioned previously.

enter image description here

So for having such splits, you should

  1. Fix a training end date
  2. Create a random split only on the data that is inside the training period so that you have an in-time test set
  3. All the data after the training period is also a test set, but this is an out time test set.

In that way you are evaluating non only your model performance but how resilient your model is to the natural change of the distribution of the features over time.

Hopefully, you see that the test set was not as redundant as initially seemed.

I hope it helps!

$\endgroup$
5
  • $\begingroup$ Sorry I am really not convinced by chat GPT's part of the answer: if the data partition process is random, I agree with OP's post that testing again on a final test set sounds completely redundant. For your other argument, what do you mean by "the natural change of the distribution of the features over time"? One could argue that you need to have an explanation/theory as to why the data generation process changes over time. If so, shouldn't that be an additional feature to you data? Add that as a feature and still use the cross validation without the need for a final test set. $\endgroup$
    – CyG
    Jan 31 at 11:55
  • $\begingroup$ Do you have a scientific reference for suggesting the use of a final test set? $\endgroup$
    – CyG
    Jan 31 at 11:58
  • $\begingroup$ How ironic that an (machine) AI/ML model answers questions on a (human) AI/ML forum.. $\endgroup$
    – Nikos M.
    Mar 1 at 10:34
  • $\begingroup$ @NikosM It is Ironic indeed, that two humans are discussing an AI/ML model's answer about AI/ML on a human-based AI/ML forum. What a time to be alive! $\endgroup$
    – Multivac
    Apr 5 at 3:24
  • $\begingroup$ "What a time to be alive", I find it rather depressing than what I had initially imagined or hoped for. Regardless, @Multivac $\endgroup$
    – Nikos M.
    May 24 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.