1
$\begingroup$

I have a neural network that Im evaluating using 10 -Fold cross validation. The validation accuracy for a fold changes alot during training in the range of -+10%

So for example the validation accuracy of a fold would range between 80% and 70%.

My question is which number should I consider to be this fold's accuracy.

Should I just take the maximum validation accuracy reached while training or should I just run the training for a certain number of epochs and take the last number (The second approach's result will depend on luck)?

Thanks

$\endgroup$
2
$\begingroup$

For some context, the entire point of running K-Fold cross-validation to compute an estimate of the performance. If the obtained values vary a lot, it just means that your estimate is less precise (i.e. has a larger standard deviation)

In a neural network setting, your network evolves, so obviously, averaging all folds might give you a less accurate estimate if the network learned a lot in the last few iterations for instance.

Typically, you would always take the fold at the end of training, since this is the performance you currently have.

However, it is good to note that analyzing the variance of your performance within a short window can help you understand whether or not your model has converged or whether it is still "exploring".

$\endgroup$
  • $\begingroup$ Thanks @Valentin. I understand that I should average the results of all folds. Im just wondering what should I consider the result of 1 fold to be? Should it be the mean of the validation accuracy's achieved during training? $\endgroup$ – Mohamed Ahmed Nabil Dec 4 '19 at 20:46
  • $\begingroup$ I'm not sure what you're confused about. The result of 1 fold is whatever performance you get from that fold. Am I understanding your question wrong? $\endgroup$ – Valentin Calomme Dec 4 '19 at 21:36
  • $\begingroup$ My problem is the result of my 1 fold varies and fluctuates a lot during training. Example if I stop at epoch X i get validation accuracy of 71 and If I stop at epoch Y I get validation accuracy of 65. So this high variance and fluctuation is what I am asking about. How should I consider the result of this fold (validation acc) to be if it is constantly changing and fluctuating a lot during training $\endgroup$ – Mohamed Ahmed Nabil Dec 4 '19 at 22:43
  • $\begingroup$ take the last. There should be some variance in between epochs because you're still training. The value after your last epoch is your current accuracy. If it fluctuates much this just means that your algorithm doesn't really converge below a certain standard deviation as @ValentinCalomme said $\endgroup$ – Philipp Dec 5 '19 at 16:42
  • $\begingroup$ I've updated my answer in lights of the comments $\endgroup$ – Valentin Calomme Dec 7 '19 at 9:42
1
$\begingroup$

The value you take as result should always be the last one you obtain. However, you might want to set up some early stopping procedure, in order to avoid overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.