I'm a little unclear about the expected use/value of a test set in machine learning. Here is a story that explains my confusion, assuming you're using a train/dev/test split:
You use your dev set to choose the best hyperparameters and make various tweaks, and when you're finally "done" you evaluate it on your test set. Your test performance comes back much worse than your dev performance.
So now you conclude, "My dev set must be too small, causing me to overfit my hyperparameters." So you make your dev set bigger, find new hyperparameters, and evaluate on your test set again. Now your dev and test performances are close to each other.
But note that you used your test set twice in that case. So in some sense you were fitting your hyperparameters to your test set, and it became a second development set.
To try to answer my own question: I guess you could say that the value of the test set is that without it, you would have never known you were overfitting your hyperparameters. And as long as we aren't using the test set "too much" in the above way (increasing the dev set size in a cycle again and again), it is still "mostly" unbiased. However we would have to concede that the test set is only truly unbiased if we only need it once.
Do you think this is an accurate take?
Incidentally, I'm unsure if there's anything else you can do (other than increasing dev set size) if your test performance comes back much worse. Well, I suppose you can cry ;). But are there any other options?