The standard setup when training a neural network seems to be to split the data into train and test sets, and keep running until the scores stop improving on the test set. Now, the problem: there is a certain amount of noise in the test scores, so the single best score may not correspond to the state of the network which is most likely to be best on new data.
I've seen a few papers point to a specific epoch or iteration in the training as being "best by cross-validation" but I have no idea how that is determined (and the papers do not provide any details). The "best by cross-validation" point is not the one with the best test score.
How would one go about doing this type of cross validation? Would it be by doing k-fold on the test set? Okay, that gives k different test scores instead of one, but then what?