May the training set and validation set overlap?

Question

May the training set and validation set overlap?

Similarly, may the testing set and validation set overlap?

Answer 1

Definitions, so we are on the same page:

• Training set: the data points used to train the model.
• Validation set: the data points to keep checking the performance of your model in order to know when to stop training.
• Testing set: the data points used to check the performance once training is finished.

---

May training and validation sets overlap?

They should not.

The validation set is used to know when to stop training your model. The idea is that you check your model performance often and when there seems to be no more improvement, you stop.

Take a look at the plot below. It is plotting the loss of the model. If the loss is still decreasing means that you can keep on improving your model, but if the loss stops decreasing you stop training.

Notice how the valid loss stops decreasing before the train loss. That is because the model could keep improving to improve the performance on the training set, but if you do that, you will get overfitting. So, by having an unseen validation set, you will stop training earlier and not overfit your mode, otherwise, if you keep on training, the accuracy on the validation set will start to decrease.

That means that if the two sets overlap, the validation loss will become more similar to the training loss, so you model will keep on training and you will overfit your model.

---

May validation and testing sets overlap?

They should not.

You already used your validation set to stop training your model. That means that you already know the performance of your model again the validation set.

Now your model is trained and you want to test your model with unseen data points, i.e. with the testing set. If your sets overlap, you are biasing your test results towards the performance which you already know your model has.