I like your question, it is somewhat philosophical in nature.
We know that a test set should not affect the model, otherwise it acts as a validation set. Therefore, even if there is enough time, if we act on a bad test result and change the model, the test set becomes a validation set, although, it is not as involved as a validation set that is used for early stopping or parameter tuning.
In other words, a test set must be useless just the way you have described it! The moment it is useful, it becomes a validation set. Although, to be more precise, a test set is not THAT useless because it probably lowers your (and your boss's) expectation about the later performance of the model in production, so lower risk of heart failure there.
As an example, in a Kaggle competition, the final set is a "test set" since it does not affect the submitted models, however as soon as the final leaderboard is announced, that test set becomes a validation set; e.g., it affects which algorithms we later choose, i.e. those of top competitors.
In summary, it seems that most of the time we are using less-involved validation sets to double check more-involved validation sets.
P.S.: as of writing this answer, @aranglol came up with similar notes and examples :) (+1)