I am training a neural network and doing 10-fold cross validation to measure performance. I have read lots of documentation and forums telling that the set of weights that should be saved or checkpointed are the ones that results to lowest
val_loss and not highest
val_accuracy, since the former usually results to higher testing accuracy.
Out of curiosity, I checkpointed both highest
val_accuracy and lowest
val_loss during my training. However, I found out that for some folds, I am getting better testing accuracy when I use the set of weights with highest
val_accuracy compared to lowest
val_loss. So during my cross-validation, I chose the set of weights that resulted to higher testing accuracy regardless of whether it came from the highest
val_accuracy or lowest
val_loss, and then just averaged the resulting testing accuracy across the 10 folds.
Is my methodology valid?