Given a data set I want to train a machine learning algorithm on. The data is split into training, validation, and test data.
I now successfully trained my algorithm to work well with the training data and validating using the validation set is also promising. However, when applying the test data the model underperformes.
I am now stuck with two options:
- Throw everything away and start new with the same data set. This however has been likened to p-hacking.
- Throw the data set away as it is now burned. This could kill my project or be really expensive as I need to recollect data, this might even be impossible.
Is my data set ultimately burned when applying the trained model unsuccessfully on my test set?
Related Bonus: Is there some form of Bonferroni Correction I could apply to keep reusing the data set in case I would have burned the data set?