SGD performing better than Adam in Random minority oversampling, I don't know what is the reason. Help

Question

So my dataset image before and after balancing looks like this:

But when I train with Adam(0.0001) and SGD(0.0001), the results are very different. Why? What is going on under the hood?

This is DenseNet121 run with Adam(0.0001)

This is DenseNet121 ran with SGD 0.001

Answer 1

First of all, your learning curves look really bad. Take a look at this post on How to use Learning Curves to Diagnose Machine Learning Model Performance.

However, since this is an interesting topic, I think some notes are worth mentioning:

One interesting and dominant argument about optimizers in papers is that SGD better generalizes than Adam. These papers argue that although Adam converges faster, SGD generalizes better than Adam and thus results in an improved final performance. One of said papers argues that minimizing training time can decrease generalization error. This is because the model will not see the same data several times and wouldn't memorize the data without losing its generalization capability. Interestingly, this can directly correlate to your problem: you are randomly oversampling your dataset. Thus, the model sees some data over and over again. So, add this to what I mentioned before; You get a model that learns nothing.

• In summary, ADAM isn't always better.
• Use regularization techniques, such as Dropout, Batch Normalization, etc.
• Fine-tune your model. If you don't know how, try different learning rates and compare the results. There are also numerous automatic hyperparameter tuning methods such as Hyperband, Bayesian Optimization, etc., that can basically fine-tune any hyperparameter.

Answer 2

Adam is considered the easiest optimizer to tune, though other optimizers can achieve higher performance with more hyperparameter tuning effort. It often works out of the box once you figure out the right learning rate. So it would be surprising if Adam would continue to flat-line after you tune the learning rate (assuming that there aren't bugs with your setup).

Looking at your plots, I think it's possible that your Adam-trained classifier is always predicting the same class (maybe class zero).

Evidence: Classifier converges very quickly to 25% training accuracy, which is unusual, and corresponds to the class balanced proportion of any one label. ~43% validation accuracy, corresponding approximately to the unbalanced proportion of class 0 or class 1.