As I understood, SGDW and SGD + momentum is two different optimizer techniques and SGDWR is SGDW + scheduler in a form of cosine annealing with warm restart. Am I right? If not, please correct me. So, another question would be, can SGDW and SGD + momentum be merged together or it will be pointless in terms of results? And which one out of three show better results (even if it is individual for each model, but asking in average) in terms of better generalization, peak accuracy and times it get to this peak? Thanks in advance!