I'm training a CNN for a 3-class image classification problem. My training loss decreased smoothly, which is the expected behaviour. However, my validation loss shows a lot of fluctuation.

enter image description here

Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?

Additional info: I'm fine-tuning the last layer of a Resnet-18 that was pre-trained on ImageNet data in PyTorch. I must note that I'm using a weighted loss function for the training phase since my data is highly unbalanced. To plot the losses, however, I use the unweighted loss as to be able to compare the validation and training loss. I would use the unweighted loss, were it not that the distributions of the training dataset and validation dataset are somewhat different (they are both highly unbalanced however).


2 Answers 2


From simple inspection of your plot, I could make a few conclusions and list things to try. (This is without knowing any more about your setup: training parameters and model hyperparameters).

It looks like the loss is decreasing (put a line of best fit through the validation loss). It also looks like you might be able to train for longer to improve results, as the curve is still headed downwards.

First I will try answer your title question:

what is the cause of the fluctation in the validation loss?

I can think of three possibilities:

  1. Regularisation - to help smoothing the learning process and make the model weights more robust. Adding/increasing your regularisation will prevent large updates to weights being introduced.
  2. Batch size - is it relatively small (e.g. < 20?). This would mean that the measured mean error at the end of the network is computed using only a few samples. With a batch size of, say 8, then getting 4/8 correct and compared to getting 6/8 correct has a large relative difference when looking at the loss. Taking the mean of the errors with such small batches will lead to a not-so-smooth loss curve. If you have enough GPU memory/RAM, try increasing batch size.
  3. Learning Rate - might be too large. This is similar to the first point regarding regularisation. To make smoother improvements, you might need to slow down the pace of learning as you approach a loss-minimum. You can make this perhaps run on a schedule, whereby is is reduce by some factor (e.g. multiply it by 0.5) every time the validation loss has not improved after, say 6 epochs. This will prevent you from taking big steps and then maybe overshooting a minumum and just bouncing around it.

Specific to your task, I would also suggest trying to perhaps unfreeze another layer, to increase the scope of your fine-tuning. This will give the Resnet-18 a little more freedom to learn, based on your data.

Regarding your last question:

Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?

Should you be worried? In short, no. A validation loss curve like yours can be perfectly fine and deliver reasonable results; however, I would try some of the steps I mentioned above before settling for it.

Should you just pick the best performing model? If you mean taking the model at its point with best validation loss (validation accuracy), then I would say to be more careful. On your plot above, this might equate to around epoch 30, but I would personally take a point that has trained a little more, where the curve gets a little less volatile. Again, after having tried some of the steps outlined above.

  • $\begingroup$ Thanks for your answer Dexter! I have indeed noticed that decaying the learning rate over time makes the validation loss curve a lot less volatile in later epochs. Now I've one follow-up question about the unfreezing of another layer. I was already thinking about doing this but don't really know how to go about this. Should I first train the classifier for a number of epochs and then both the classifier AND convolutional layer, or could I immediately start training both of them (perhaps with differing learning rates)? I don't want to waste the small amount of computational recourses I have. $\endgroup$
    – Josh
    Commented May 4, 2018 at 8:45
  • $\begingroup$ @Tim - If possible it is usually preferable to train end-to-end, meaning both at the same time. So I would unfreeze another layer (the one closest to the output that is still frozen) and then train everything. If this doesn't give you the boost you're after, you could perhaps re-initialise the weights of the unfrozen layers (not all layers!) and train for longer, but this would have other risks and perhaps make things worse. Consider it a last resort. $\endgroup$
    – n1k31t4
    Commented May 9, 2018 at 13:55

I would also recommend that you use methods for data augmentation and oversampling to compensate the unbalance of the classes. This standford paper explains some of the ideas that you can implement The Effectiveness of Data Augmentation in Image Classification using Deep Learning.

I hope this helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.