I'm building a binary classifier, using task-transfer from resnet and a total training set of 300 images. Initially I put aside 100 images as validation, and tuned the hyperparameters, each time training on 200 and testing on 100, until I got a validation accuracy of 93%.
Happy with this accuracy I tried the same parameters on the test set (another 170 images) and got really bad accuracy (around 65%).
What did I do wrong?
- Should I have used cross validation?
- What is the best practice here? How should I got about tuning my hyperparameters?
- Can I repeat my process and check on the test set again? If so, how many times can I do this before it's "cheating"