Your intuition that the learning rate related to the shape of the error gradient is correct when the loss is differentiable and convex in relation to its parameters (Theorem 6.1 https://www.stat.cmu.edu/~ryantibs/convexopt-F13/scribes/lec6.pdf).
Generally your loss is high dimensional in its parameter space and therefore non-convex. Moreover, the loss itself is usually Lipschitz continuous with a large constant $L$, which leads to the existence of not only local but also sharp minima.
Choosing a learning rate then necessarily depends on both $L$ and the initialization. For a very large $L$, you could still converge with a large learning rate (as opposed to the convex case), as long as the initialization is within a neighbor of a decent flat minima.