Reporting test result for cross-validation with Neural Network

Question

I have a small dataset, so I have to use cross validation to report the test result to get a better estimate of the classification result. For some reason, I have to use neural networks to do this.

Because neural networks have their unique quirks e.g finding hyper-parameters, I am using a nested cross-validation. I am dividing up my dataset into 10 folds for cross-validation. Then I am dividing the 9 folds that are for training, again in 10 folds. From those 10 folds, I am using 9 folds to train with different hyper-parameters(in my case it is number of hidden units and dropout rate), and using the other fold to to get the accuracy with different hyper-parameters (kind of like a validation set in the deep learning literature).

Then I am training my model again on all of the 9 folds of the first division of data with the best hyper-parameters I found. Because I am missing out on some data from the initial 9 folds for using as a validation set. Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result? Just to make it clear, I am not tuning any hyper-parameters at this stage. I am just setting the network to stop training on the training data when it reaches the best test result. I think this is a really subtle problem, if it is a problem at all. That is why I am confused.

I am doing this whole thing 5-6 times with different seeds for different divisions of data, and I am only reporting the mean of all of these runs.

Answer 1

Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result?

Technically yes this is incorrect process for reporting an unbiased test metric. This could be a bad over-estimation of performance if cv results are noisy and vary randomly epoch-to-epoch. You should in theory treat the early stopping epoch number same as your other hyper-parameters, discover a good value from the lower-level cross-validation and stick with it.

There is a problem with my suggestion though - the early stopping epoch number is sensitive to training data set size, and you just increased the size when you changed from lower level cross-validation to the higher level one. So you might get an unbiased measure at the expense of significantly worse results.

First, take a look at your learning curves. Just how sensitive are the cv results to epoch number? If they are not sensitive - no obvious over-fit over a reasonable range of epoch numbers - then just pick a mid-range fixed number of epochs that applies to all folds. That way you will have your unbiased estimate, and probably not compromised on model quality.

Alternatively, you may just have to be happy knowing that your test estimate is biased, but you have reduced the variance significantly by using 10-fold CV, and have only searched one hyper-parameter at the top level (not all three). It may only be slightly biased, and still a good estimate. The smoother and less jittery the learning curves are, the more you can get away with this - but sadly you won't get a measure of the bias, so you'll never be 100% sure.