So I designed my own CNN with 10 layers of convolutions and no maxpoolings or any other connections. When I ran it on a dataset I got the following loss curve (blue) the other one is accuracy vs number of epochs. What do you think might be the problem here? And how to rectify it? enter image description here

NOTE: My data-set of size 20 due to lack of computational resources, so the oscillation is probably looking more prominent or happening in the first place.

  • $\begingroup$ Can you give more details about how you train it? Do you mean batch-size of 20 or total dataset? are you performing classification? $\endgroup$
    – n1k31t4
    Jun 26 '18 at 13:20
  • $\begingroup$ @n1k31t4 total size of 20 $\endgroup$
    – DuttaA
    Jun 26 '18 at 14:25
  • $\begingroup$ Adam is not a good choice in this case, just reduce the learning rate that will help $\endgroup$
    – Perry Shao
    Nov 22 '18 at 23:43

Oscillating loss can be attributed to either of the following:

  1. Learning rate: Reduce the learning rate so that the gradient descent doesn't overshoot the minima.
  2. Optimizer: Choose ADAM optimizer over the others like SGD. It works well.
  • $\begingroup$ Yeah but I don't have computational power to run that many epochs to compensate for learning rate..As for backprop algorithm I will change that $\endgroup$
    – DuttaA
    Jun 26 '18 at 16:26
  • 1
    $\begingroup$ You can run it on AWS or on google cloud where you can get enough compute power. $\endgroup$
    – varsh
    Jun 28 '18 at 4:45
  • $\begingroup$ I generally agree that Adam is a good choice of optimizer. I often use Nadam, which incoporates both Nesterov acceleration and adaptive moments. Could you explain more about how adaptive moments relates to oscillating loss? $\endgroup$
    – user120547
    Nov 23 '21 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.