6
$\begingroup$

The standard setup when training a neural network seems to be to split the data into train and test sets, and keep running until the scores stop improving on the test set. Now, the problem: there is a certain amount of noise in the test scores, so the single best score may not correspond to the state of the network which is most likely to be best on new data.

I've seen a few papers point to a specific epoch or iteration in the training as being "best by cross-validation" but I have no idea how that is determined (and the papers do not provide any details). The "best by cross-validation" point is not the one with the best test score.

How would one go about doing this type of cross validation? Would it be by doing k-fold on the test set? Okay, that gives k different test scores instead of one, but then what?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

I couldn't say what the authors refer to by

best by cross-validation

but I'll mention a simple and general procedure that's out there:

You are correct that analyzing one estimate of the generalization performance using one training and one test set is quite simplistic. Cross-validation can help us understand how this performance varies across datasets, instead of wonder whether we got lucky/unlucky with our choice of train/test datasets.

Split the whole dataset into k folds (or partitions), and train/test the model k times using different folds. When you're done, you can compute the mean performance and a variance that will be of utmost importance assessing confidence in the generalization performance estimate.

Improve this answer
$\endgroup$
0
2
$\begingroup$

Best by cross validation The whole data can be divided into training and testing. You can not touch the testing data set for any kind of training. Keep it away!!

For the training data you can use it freely to specify the best classification model. But how to know if the current setting is the best one, you do cross validation by using only training data set, dividing the training data set into k folds. Based on this cross validation result you select the setting that gives the best result. Best by cross validation

Improve this answer
$\endgroup$
2
  • $\begingroup$ I get dividing the training data into k folds and repeating the training of the neural network k times. I've done this, okay, it produces k values for loss, accuracy etc for each iteration/epoch. What I don't get is how do you know which epoch is the "best" one? In real life, the results are such that in many cases we can't tell whether one epoch has a higher or lower loss/accuracy using a suitable statistical test such as Wilcoxon signed-rank or Mann Whitney U. $\endgroup$
    – Alex I
    May 11, 2016 at 8:52
  • $\begingroup$ I agree with you ! It is not significant to make decisions based on one test. But, always there is no guarantee that the model you generate is the best for real life. $\endgroup$
    – Bashar Haddad
    May 11, 2016 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.