neural networks error function: is global minimum desirable?

Question

In "Elements of statistical learning" page 395 the authors state that, relative to R(θ), the regression/classification error function in a neural network such as a multi layer perceptron:

Typically we don’t want the global minimizer of R(θ), as this is likely to be an overfit solution. Instead some regularization is needed: this is achieved directly through a penalty term, or indirectly by early stopping. Details are given in the next section.

However in Backpropagation, when momentum regularization is described the figure 9 shows how the back propagation process should steer the error function towards its global minimum, avoiding any local minimum found along the way.

How come these 2 reasonings are compatible? The only thing I can think of is that we want to get the global minimum for each coefficient error (as computed during the backprop process) while not adjusting for the "global" error function. Am I interpreting this correctly?

Answer 1

The figure describing momentum is a bit misleading because it only considers a very simple case. Usually momentum will help you not to get stuck too early in a local minimum, so you end up in a better but still local minimum.

Momentum can help you converge faster, but there are no guarantees to end up in a global minimum. In popular use cases of neural networks like image classification etc. the error function will be extremely complicated, so it will always be infeasible to reach a global minimum using gradient based methods. But practice has shown that driving down the error to a local minimum still leads to good results.

Also, momentum is compatible with other forms of regularization. You can apply momentum to any regularized cost function. The statement in Elements of Statistical Learning is more of general nature, meaning that even if you could achieve the global minimum it wouldn't be desirable because it is most likely not the solution that will give you the best generalization (even though it has the best training(!) error).