Given a data set I want to train a machine learning algorithm on. The data is split into training, validation, and test data.

I now successfully trained my algorithm to work well with the training data and validating using the validation set is also promising. However, when applying the test data the model underperformes.

I am now stuck with two options:

  • Throw everything away and start new with the same data set. This however has been likened to p-hacking.
  • Throw the data set away as it is now burned. This could kill my project or be really expensive as I need to recollect data, this might even be impossible.

Is my data set ultimately burned when applying the trained model unsuccessfully on my test set?

Related Bonus: Is there some form of Bonferroni Correction I could apply to keep reusing the data set in case I would have burned the data set?


I would try one more time to split the data & train a new algo. This is similar to p-hacking as you've stated so try to avoid doing this more than once but if the alternative is the project stops there doesn't seem to be much more you can do.

This time when you split the data I would try stratifying it by some feature (the target? Some category of the input variables?). This ensures the training/validation/testing sets have the same proportion of some criteria which might help your performance be a little more consistent when comparing the validation set to the testing set.

  • $\begingroup$ though this personal suggestion is appreciated I am rather looking for a more authorative source - i.e., publication, best-practice guidelines, math$^{tm}$. Given that p-hacking is becoming more and more a issue of public/academic debate I want to have a firm understanding. $\endgroup$
    – Sim
    Oct 16 '20 at 13:26
  • $\begingroup$ Fair enough! If you learn of such a source please post it here as I'd be interested as well. $\endgroup$ Oct 16 '20 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.