0
$\begingroup$

I am training a CNN model for a three-class classification problem. To do this, I'm gradually unfreezing more convolutional blocks of a pre-trained Resnet-18 network. The thing is that after unfreezing a block (let's say block 3, and preceding blocks), the performance on the validation set did improve but the performance on the test dataset did not improve (as opposed to block 2, and preceding blocks).

I'm now wondering whether or not it is 'justified' (so to speak) to keep unfreezing blocks to see how this affects performance on the test dataset. I feel like I'm just capitalising on some kind of luck that the model fits the test dataset better just by chance.

In short, I guess my question is whether I should choose my models on validation performance or test performance?

$\endgroup$
1
$\begingroup$

Don't optimize your model for the test set during development. You shouldn't even consider using your model on the test data until you're done tweaking your it, since the point of the test set is to have a ready example of "unseen" data for your model.

The idea with a train/validation/test split is that you could evaluate all the different iterations your model went through (that you feel like testing) against this completely unseen set of data and not have to worry about whether you were optimizing for fit to this data because it's completely new to your evaluation process.

No decisions were made during development to ensure that any of the tested models would perform well on this set because you're only checking how well you did after you're done fussing with your models. It's as clean a reference point for model performance as you can get.

If the only constraint for your final model choice is performance, then you would pick the model that performed the best on your test data. But I'd highly recommend using the train/validation/test split as intended.

$\endgroup$
0
$\begingroup$

The test set is there to help you understand how well your model generalizes.

It acts as a checking point (i.e., is my model valid for data it has not seen before) rather than a reference point.

If your model performs a lot better than in your validation set than in your test data set, your model could be overfitting, etc.

Agreed with @tm121 above; you shouldn't be tweaking the model based on your test set performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.