# Does using your test set ultimately burn your data set in case of failure?

Given a data set I want to train a machine learning algorithm on. The data is split into training, validation, and test data.

I now successfully trained my algorithm to work well with the training data and validating using the validation set is also promising. However, when applying the test data the model underperformes.

I am now stuck with two options:

• Throw everything away and start new with the same data set. This however has been likened to p-hacking.
• Throw the data set away as it is now burned. This could kill my project or be really expensive as I need to recollect data, this might even be impossible.

Is my data set ultimately burned when applying the trained model unsuccessfully on my test set?

Related Bonus: Is there some form of Bonferroni Correction I could apply to keep reusing the data set in case I would have burned the data set?

• though this personal suggestion is appreciated I am rather looking for a more authorative source - i.e., publication, best-practice guidelines, math$^{tm}$. Given that p-hacking is becoming more and more a issue of public/academic debate I want to have a firm understanding.