Your intuition is on point, and shrinking the learning rate like this is often referred to as "annealing". But linking the learning rate to error magnitude neglects certain problematic error surface topologies. An excellent motivating example is the Rosenbrock "Banana" Function, which is often used as a test case for optimization algorithms. The "banana" is a low error valley which hides the global minimum. If an optimization path finds its way into this valley, the path to the global minimum is along a nearly flat gradient.
If you use an optimization algorithm that naively shrinks the learning rate relative to the error, you're going to get stuck as soon as you hit the valley. On the one hand: congrats! You've achieved a low error solution. But you're not necessarily anywhere near the global minimum. So how can we do better?
An approach used by modern gradient-based methods like Adagrad, RMSProp, and Adam is to separately assign learning rates to each parameter, and tie the learning rate to the magnitude of the respective parameter's update. The Stanford CS231n lecture notes explains:
Adagrad is an adaptive learning rate method originally proposed by Duchi et al..
# Assume the gradient dx and parameter vector x
cache += dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)
Notice that the variable cache has size equal to the size of the gradient, and keeps track of per-parameter sum of squared gradients. This is then used to normalize the parameter update step, element-wise. Notice that the weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased.