How to properly set up neural network training for stable accuracy and loss

Question

I have a DenseNet121 implemented in Pytorch for image classification. For now, the training set-up is pretty straightforward:

• the data is loaded. An important characteristic here is that the validation data is fixed from the outset and can never change. The rest of the data I split into training and testing. None of the data sets overlap!
• for every epoch iterate through the training data loader, calculating loss, optimizing etc.
• every 100 batches evaluate the loss using the validation data loader
• at the end of an epoch compare the current state’s loss on validation data with the state that previously had the best validation loss (for the first epoch just compare this with a random high number like 1e5) and save the current state if it better or keep the older state.
• after all epochs are finished save the state with the lowest validation error as the best model
• the best model is then applied to test data. Accuracies are calculated and ROC curves drawn

I was wondering how to extend my set-up to make it a proper statistical experiment set-up, with the goal of getting stable accuracy and loss results ie I want to be able to say that my model gives consistently the same results. I was thinking along the lines of running the testing step say 10 times and averaging the error? Or do you see some deficiency in the training that I could improve to improve stability?

Thanks in advance

Answer 1

The learning rate is one of those first and most important parameters of a model, and one that you need to start thinking about pretty much immediately upon starting to build a model. It controls how big the jumps your model makes, and from there, how quickly it learns.

There are learning rate technique called Cyclical Learning Rates.

Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. Check this out : Cyclical Learning Rates for Training Neural Networks. And implementation in pytorch and details blog : Adaptive - and Cyclical Learning Rates using PyTorch.

By this small trick, you can build a stable version of your model. Best of luck.