What is the best practice for tuning hyperparameters using validation data?

Question

I'm building a binary classifier, using task-transfer from resnet and a total training set of 300 images. Initially I put aside 100 images as validation, and tuned the hyperparameters, each time training on 200 and testing on 100, until I got a validation accuracy of 93%.

Happy with this accuracy I tried the same parameters on the test set (another 170 images) and got really bad accuracy (around 65%).

What did I do wrong?

1. Should I have used cross validation?
2. What is the best practice here? How should I got about tuning my hyperparameters?
3. Can I repeat my process and check on the test set again? If so, how many times can I do this before it's "cheating"

Answer 1

If you're seeing performance that is much better on the validation than the unseen test data, then that is suggestive of some sort of overfitting or, if not, that the data do not come from the same distribution. That could mean that your test images are very different from the training and validation data, for example.

First, I'd double check the data to make sure the train, validation and test sets are definitely distinct, and that all three sets have roughly the same number of positive and negative examples. If this isn't obviously the problem, then most likely I'd guess that you are overfitting the hyperparameters to the validation dataset.

Using cross-validation would most likely help you to see whether this is the case, as you'd see more dramatic variation between each fold if the parameters were overfit towards one specific validation fold.

If you repeat the process on the test dataset, then that data is no longer "unseen" and is part of the model, even if you just use it to tune hyperparameters. You should keep the test dataset completely separate from the model building process so that when you come to measuring model performance, you get a true estimation of what would happen on unseen data.