So my dataset image before and after balancing looks like this:

enter image description here enter image description here

But when I train with Adam(0.0001) and SGD(0.0001), the results are very different. Why? What is going on under the hood?

This is DenseNet121 run with Adam(0.0001)

enter image description here

This is DenseNet121 ran with SGD 0.001

enter image description here

  • 1
    $\begingroup$ Have you tried increasing the learning rate for the SGD optimizer? Generally the SGD optimizer uses a higher learning rate than the Adam optimizer, see for example the defaults for tensorflow (0.01 for SGD versus 0.001 for Adam). $\endgroup$
    – Oxbowerce
    Sep 11, 2022 at 17:40
  • $\begingroup$ Thank you so much for your comment. I've edited. The last graph is of SGD. Could you tell what is going on? $\endgroup$ Sep 11, 2022 at 18:09

2 Answers 2


First of all, your learning curves look really bad. Take a look at this post on How to use Learning Curves to Diagnose Machine Learning Model Performance.

However, since this is an interesting topic, I think some notes are worth mentioning:

One interesting and dominant argument about optimizers in papers is that SGD better generalizes than Adam. These papers argue that although Adam converges faster, SGD generalizes better than Adam and thus results in an improved final performance. One of said papers argues that minimizing training time can decrease generalization error. This is because the model will not see the same data several times and wouldn't memorize the data without losing its generalization capability. Interestingly, this can directly correlate to your problem: you are randomly oversampling your dataset. Thus, the model sees some data over and over again. So, add this to what I mentioned before; You get a model that learns nothing.

  • In summary, ADAM isn't always better.
  • Use regularization techniques, such as Dropout, Batch Normalization, etc.
  • Fine-tune your model. If you don't know how, try different learning rates and compare the results. There are also numerous automatic hyperparameter tuning methods such as Hyperband, Bayesian Optimization, etc., that can basically fine-tune any hyperparameter.

Adam is considered the easiest optimizer to tune, though other optimizers can achieve higher performance with more hyperparameter tuning effort. It often works out of the box once you figure out the right learning rate. So it would be surprising if Adam would continue to flat-line after you tune the learning rate (assuming that there aren't bugs with your setup).

Looking at your plots, I think it's possible that your Adam-trained classifier is always predicting the same class (maybe class zero).

Evidence: Classifier converges very quickly to 25% training accuracy, which is unusual, and corresponds to the class balanced proportion of any one label. ~43% validation accuracy, corresponding approximately to the unbalanced proportion of class 0 or class 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.