Let say I have train set and validation set
if 'A' included in the train set. 'A' should not be include in the validation set? or some is ok?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Validation set is basically to understand how your model is behaving in terms of over-fitting, and under-fitting, and also to find the best set of hyper-parameters for your algorithm. If you use some parts of the same training data on your validation set, then this hypothesis would not be justified. Hence, it is suggested to split your data set into train/validation/test set without any overlapping of the data.
-
$\begingroup$ Yes...It is recommond to repeat very little portion of training also in order to check consistency of output. $\endgroup$ Commented Jul 11, 2020 at 7:59
$\begingroup$
$\endgroup$
The problem with having overlapping examples between Validation set and training set is the purpose of using a validation set is to ideally tune hyperparameters for your model and to have overlapping examples would mean that since your model has already been trained on this overlapping data, it would ideally have a greater probability of predicting the correct output as compared to an ideal scenario where it will be seeing completely unseen data in validation set.
It's always best to split your data into completely disjoint training, validation and test sets. It is best to treat the validation set as a kind of a test set to tune hyperparameters. And finally, once you are satisfied with the performance on training and validation sets, you should check the performance on the test set only once to understand how it behaves with completely unseen data.
However in certain scenarios you can deviate from this standard format, for example while working with time-series data.
When you are working with time-series data, the most recent data captures the most relevant information possible, so it is more prudent to include them in training data. So a more prudent decision would be to opt for Roll-Forward Partitioning.
Roll-Forward Partitioning: We start with a short training period and we gradually increase it, at each iteration of training, we train it on the current training period and make it forecast the next interval of data. It will require more training time, but it mimics what we would do during deployment where we would want to keep training our model at regular intervals to keep it up to date.
You can find more about splitting data for time-series models, in this question here.