Every thread on stackexchange that I've found says that you can only use the test set once and thats it. So for instance, if you used a linear regression model and got poor results on the test set, you cannot change the the model to say a random forest and evaluate this model again on the test set.
This doesnt make sense to me. Example, we all know the MNIST dataset well. Lets say I download the data and split it into train, validation and test. Also, say I used a linear regression model and clearly I will do poorly on the test set. Now, what's stopping anyone else from downloading the same MNIST data set, splitting into train, validation, test and using a different model (e.g. neural network) and reporting their test result?
According to our understanding of only being able to use the test set once, their test results are invalid because they have somehow "learnt" that a linear regression model was not good. This does not seem right to me.