In reading about machine learning (ML), and working through some basic examples, it appears to me most learning algorithms use generic optimizers. I am using the word "optimizer" to describe the technique the learner uses to minimize the loss function. Gradient decent, and it's variants, seems to be the most common. But the general idea in ML seems to be to continually iterate a learning algorithm, each time adjusting various things to try to improve the loss.
Gradient decent, and similar techniques, are generic in that they work on a wide variety of learning models. This makes intuitive sense, all you need is the predicted value and the known target value.
But, often the trade-off with generic techniques is that while they work everywhere, they are slow. In other words, in theory you could construct a learner-specific optimizer that exploits characteristics of the learning algorithm to avoid some of the work that something like SGD might have to do. So if you know you're going to use learning algorithm XYZ, then maybe you can speed up computation considerably by using a loss-minimizing technique designed explicitly for learner XYZ.
As a rough analogy, since my background is in programming, consider sorting algorithms. Quicksort is often cited as the best general sorting algorithm. But if you know in advance that your data is constrained in certain ways, other algorithms will be faster, given a specific use-case (even good old bubblesort can be fast if conditions are right!).
So, that's the question: have learner-specific optimizers been developed? (And a related question, is "optimizer" the right word to use here?)
(FWIW, I posted this same question to Kaggle Q&A, but didn't get any responses there.)