Can I use the test dataset to select a model?

Question

I'm trying to understand if the test dataset can be used to select a final trained model. Let's assume this scenario:

I first split the whole dataset: 70% training, 30% test. Then I fit several models (let's say NN, RandomForest, AdaBoost,..) on the training dataset with cross-validation and tune the hyperparameters to get the best performance on the train data. I know that these scores are biased, since I was tuning the hyperparameters on this data.

Then I use the test dataset to get the true performance on the unbiased data and select which model performs the best.

Is this a correct way to use the test dataset? Some confusion comes from the internet definition of the test dataset:

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

It seems like it should only be used to get the performance of the one final trained model. My teacher told me I can't select a trained model based on the scores of the test dataset and quoted the definition above. I'm struggling to believe that she is correct. Which dataset should be used to select the model then?

Answer 1

Your teacher is correct. The test dataset is unseen data. You could not select the final model using the test set. In competitions, to be fair, the test dataset is not revealed until you submit the final trained model.

To select the hyperparameters of the final model (e.g., activation functions, the number of hidden layers, the number of units, learning rate, dropout, ...), you should use the validation set.

Answer 2

Thanks for the answers. I consulted it with some more people and I think that I have settled for an explanation that makes sense for me:

The truth is that although the "test dataset" can be used to select the model as I do, it is not a "test dataset" in it's true meaning. Since I am using the dataset to select a model, the score is no longer unbiased and so it no longer represents the true accuracy, but instead an unbiased relative score between multiple models. I shouldn't call it a "test dataset" then, because it does not show the real final accuracy. It could perhaps be called "validation 2 dataset".

TLDR: "Test dataset" works to select a model, but I should not call it "test dataset" because it shows relative scores instead of the true final model accuracy.

Answer 3

Your process is OK. By using k-fold cross-validation, you're also (repeatedly) dividing the training set further into training and cross-validation sets. The test set may be used to estimate the actual generalization error.

You aren't using the test data set to select a model, as I understand.

If you weren't using k-fold cross-validation, and using the 'test' set to select a model, it would still be correct, but the 'test' set would be your cross-validation set. What you wouldn't be able to do is estimate the true generalization error.

BTW 70-30 seems like an aggressive split; "it depends," but 90-10 in this case leaves plenty of test data.