2
$\begingroup$

May the training set and validation set overlap?

Similarly, may the testing set and validation set overlap?

$\endgroup$
7
$\begingroup$

Definitions, so we are on the same page:

  • Training set: the data points used to train the model.
  • Validation set: the data points to keep checking the performance of your model in order to know when to stop training.
  • Testing set: the data points used to check the performance once training is finished.

May training and validation sets overlap?

They should not.

The validation set is used to know when to stop training your model. The idea is that you check your model performance often and when there seems to be no more improvement, you stop.

Take a look at the plot below. It is plotting the loss of the model. If the loss is still decreasing means that you can keep on improving your model, but if the loss stops decreasing you stop training.

Notice how the valid loss stops decreasing before the train loss. That is because the model could keep improving to improve the performance on the training set, but if you do that, you will get overfitting. So, by having an unseen validation set, you will stop training earlier and not overfit your mode, otherwise, if you keep on training, the accuracy on the validation set will start to decrease.

That means that if the two sets overlap, the validation loss will become more similar to the training loss, so you model will keep on training and you will overfit your model.

enter image description here


May validation and testing sets overlap?

They should not.

You already used your validation set to stop training your model. That means that you already know the performance of your model again the validation set.

Now your model is trained and you want to test your model with unseen data points, i.e. with the testing set. If your sets overlap, you are biasing your test results towards the performance which you already know your model has.

$\endgroup$
  • $\begingroup$ This really helps me. $\endgroup$ – Muhammad Usman Aug 11 '18 at 7:28
  • $\begingroup$ Happy to hear :) $\endgroup$ – BrunoGL Aug 11 '18 at 7:29
  • $\begingroup$ @BrunoGL How about for the case of cross-validation? If I understand correctly, the training data and validation data overlap during the rotation. Is that considered ok? Thanks. $\endgroup$ – yoyostein Jul 30 at 0:53
  • $\begingroup$ @yoystein yes that is ok, and you want that as well. One strategy you can have is: use training and validation set during cross-validation and hyper-parameter optimization. Then, once you figure out your best hyper-parameters and stoping critirea, retrain and use the test set for final testing. $\endgroup$ – BrunoGL Jul 31 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.