I'm training a CNN with 5-fold stratified cross-validation. On the first fold, my accuracy is ~80%, on each subsequent fold the accuracy is ~50%. Finally, upon fitting the entire training set my accuracy jumps back to ~80%. I can see that on the folds with bad performance the accuracy and loss never really change. Is it simply that one fold received all/most of the signal from the dataset or is there something else going on here?
Based on the description my guess is that any or all of the following problems occur:
- the data is not randomized, which could explain why the first fold get a very different performance than the others
- perhaps the dataset is very small, causing massive variations in performance and overfitting.
- in any case the model is very unstable, possibly due to overfitting caused by the model being too complex and/or dataset too small.