As I understood, SGDW and SGD + momentum is two different optimizer techniques and SGDWR is SGDW + scheduler in a form of cosine annealing with warm restart. Am I right? If not, please correct me. So, another question would be, can SGDW and SGD + momentum be merged together or it will be pointless in terms of results? And which one out of three show better results (even if it is individual for each model, but asking in average) in terms of better generalization, peak accuracy and times it get to this peak? Thanks in advance!
I am late but anyways.
To answer second Question,
SGDW is usually defined as below (given in this paper Decoupled Weight Decay Regularization)
So, SGDW has momentum term in itself. It is just that the weight decay term is separately added. But it should be noted that if Loss function contains L2 regularization then SGDW will be same as SGD except you can choose the decay rate and learning rate without affecting each other.
Hence we need not merge them, since SGDW has all the characteristics of SGD+momentum.
To answer the first question,
Yes, SGDW and SGD + momentum is two different optimizer techniques. As far I understand SGDWR is SGDW with warm restart, the scheduler can be of any form.
To answer your last question,
This is really problem dependent. But I use warm restarts most of the time because initially since the weights are randomly initialized the gradients of each of the weights will be of different magnitude (and usually high). I find SGDWR to give better results in terms of accuracy.