Optimization algorithms such as gradient descent or particle swarm can find a minima in a function.
On the other hand, learning methods such as back-prop define learning as an optimization problem and are used to learn weights in deep neural networks and etc.
We know that DL models can learn/memorize basically anything in the training data , even random noise, which brings us to the question of then how we can achieve generalization (as defined in ) *.
In [7, 1, 3] authors tried to show the connection between generalization and the loss landscape (sharp minima/flat minima) plus its effect from the batchsize.
The effect of training data on generalization studied in  shows that models that have high influence from more data points generalize better than models that rely/influenced on less data points.
On the other hand  sheds some light on the role of the model's parameters from a neuroscience point of view in terms of effect of neurons in generalization (Networks that generalize better are harder to break with neuron deletion.).
Although all of these papers investigated generalization up to a certain point, the effect of optimization used in the learning algorithm on the generalization is not yet clear.
Is there any study done in this direction? What are the possible related work on this? Or do you have an answer to this question?
Let me point out that although many believe explicit regularization is crucial for generalization,  already explains explicit regularization (l1/l2/dropout) does not play a big role in generalization. Many tricks that were known as a generalizer, are shown to be a myth. They also show that interestingly, SGD works as an implicit regularizer which might be a connection to the effect of optimization alg. to generalization.
: Li, Hao, et al. "Visualizing the Loss Landscape of Neural Nets." arXiv preprint arXiv:1712.09913 (2017).
: Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).
: Keskar, Nitish Shirish, et al. "On large-batch training for deep learning: Generalization gap and sharp minima." arXiv preprint arXiv:1609.04836 (2016).
: Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." arXiv preprint arXiv:1703.04730 (2017).
: Kawaguchi, Kenji, Leslie Pack Kaelbling, and Yoshua Bengio. "Generalization in deep learning." arXiv preprint arXiv:1710.05468 (2017).
: Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, Matthew Botvinick, "On the importance of single directions for generalization." arXiv preprint arXiv:1803.06959 (2018). https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/
: Dinh, Laurent, et al. "Sharp minima can generalize for deep nets." arXiv preprint arXiv:1703.04933 (2017).