0
$\begingroup$

I'm trying to understand if the test dataset can be used to select a final trained model. Let's assume this scenario:

I first split the whole dataset: 70% training, 30% test. Then I fit several models (let's say NN, RandomForest, AdaBoost,..) on the training dataset with cross-validation and tune the hyperparameters to get the best performance on the train data. I know that these scores are biased, since I was tuning the hyperparameters on this data.

Then I use the test dataset to get the true performance on the unbiased data and select which model performs the best.

Is this a correct way to use the test dataset? Some confusion comes from the internet definition of the test dataset:

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

It seems like it should only be used to get the performance of the one final trained model. My teacher told me I can't select a trained model based on the scores of the test dataset and quoted the definition above. I'm struggling to believe that she is correct. Which dataset should be used to select the model then? Thanks in advance.

$\endgroup$
2
$\begingroup$

Your teacher is correct. The test dataset is unseen data. You could not select the final model using the test set. In competitions, to be fair, the test dataset is not revealed until you submit the final trained model.

To select the hyperparameters of the final model (e.g., activation functions, the number of hidden layers, the number of units, learning rate, dropout, ...), you should use the validation set.

$\endgroup$
  • $\begingroup$ Sure, the hyperparameters are selected on the validation set. It just seems like a waste of data to use the test dataset only on 1 model only to get the score of the one selected model. So which dataset should I use to select which of the trained models to use? If I use the validation dataset, it will be biased since I set the hyperparameters on the very same data. $\endgroup$ – MSKL Dec 27 '18 at 16:00
  • 1
    $\begingroup$ It is not waste of data. Let's assume that you want to predict the future financial data. The test data is in the future, i.e., you don't know the test set. $\endgroup$ – Nga Dao Dec 27 '18 at 16:22
  • $\begingroup$ Also you should randomly divide the known data into the training and validation sets. Make sure the validation set is not very similar to the training set. $\endgroup$ – Nga Dao Dec 27 '18 at 16:24
  • 1
    $\begingroup$ So in my case the final model should be selected on the cross-validation scores? $\endgroup$ – MSKL Dec 27 '18 at 16:26
1
$\begingroup$

Your process is OK. By using k-fold cross-validation, you're also (repeatedly) dividing the training set further into training and cross-validation sets. The test set may be used to estimate the actual generalization error.

You aren't using the test data set to select a model, as I understand.

If you weren't using k-fold cross-validation, and using the 'test' set to select a model, it would still be correct, but the 'test' set would be your cross-validation set. What you wouldn't be able to do is estimate the true generalization error.

BTW 70-30 seems like an aggressive split; "it depends," but 90-10 in this case leaves plenty of test data.

$\endgroup$
  • $\begingroup$ No, i'm using the test data to get the scores of the (already) trained models. And based on these scores I select the final model to be trained on the whole data. So I do. I only had a dataset of about 1000 records (and the split was actually 75-25) so I felt that less would make the test sample too small. $\endgroup$ – MSKL Dec 27 '18 at 15:20
0
$\begingroup$

Thanks for the answers. I consulted it with some more people and I think that I have settled for an explanation that makes sense for me:

The truth is that although the "test dataset" can be used to select the model as I do, it is not a "test dataset" in it's true meaning. Since I am using the dataset to select a model, the score is no longer unbiased and so it no longer represents the true accuracy, but instead an unbiased relative score between multiple models. I shouldn't call it a "test dataset" then, because it does not show the real final accuracy. It could perhaps be called "validation 2 dataset".

TLDR: "Test dataset" works to select a model, but I should not call it "test dataset" because it shows relative scores instead of the true final model accuracy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.