Should I include the validation file in the training process after finishing the tuning process (e.g. searching for params using the validation file)?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
It depends on the distribution of the train, valid and holdout/test set.
There are a couple of possibilities (basically permutations). In general any different distribution=covariate shift is bad and you should repair it. If this is the case, including valid is the least of your problems (but you should include it in this case to make corrections) and you should worry about covarite shift.
If distributions are the same between the sets, it wont make any negative difference and it could only help if you add the valid-hyperparam tuning dataset to the train.
answered Jan 9, 2020 at 13:24
-
$\begingroup$ Thanks @Noah, my dataset consists of text and I'll use word embeddings for classification. Thus, how can I check the cases you mentioned? $\endgroup$– MinionsCommented Jan 9, 2020 at 13:44
-
$\begingroup$ by looking at the dataset, domain knowledge, trial and error.... $\endgroup$ Commented Jan 9, 2020 at 13:50