Does a neural network continue to change after SGD stops improving?

Question

I would like to understand the behaviour of stochastic gradient descent (SGD) over long time periods for a fixed learning rate.

Let us assume that after a number of epochs we expect no further improvement to our cost function. If we continue to train over more epochs then one of two things can happen:

1. The solution keeps changing but remains close to a local minimum; in the space of weights it remains in a local region. Or,
2. The solution keeps changing but explores a large region of the space of weights.

Let's make this precise. Take a fixed shape of neural network and use the Euclidean distance metric on the space of weights. Now train this network using SGD on a dataset, e.g. MNIST, and for a fixed choice of learning rate. After a large number of epochs, N start recording the weights of the network, $$ {\bf w_N}, {\bf w_{N+1}} \ldots $$ How does $d({\bf w_N}, {\bf w_M})$ behave? To gauge whether these distances are large or small, they can be compared to the distances you get by comparing ${\bf w_N}$ with equivalent networks given by permuting the neurons within a layer.

Ofcourse the answer may vary depending on the neural network, the data, the learning rate, etc. I suppose I would like to know the answer for typical "real world" networks. Unfortunately I lack the processing power (and expertise) to adequately investigate this myself, and am confident this will be known to experts.

Answer 1

The evolution of the improvement of the performance of the network if you keep training it depends on:

1) the surface of the loss function with respect to the network parameters.

2) the learning rate, which is the scale factor you apply to the gradient in order to compute the next set of weights.

If the learning rate is too high (with respect to the shape of the loss surface) and you keep training, you may escape a local minimum you just reached.

The problem is that you don't know the loss surface and so we cannot choose the perfect learning rate for it. There are a lot of heuristic tricks to try to improve the exploration behaviour of SGD, like momentum.

So the answer is: it depends. The network could stabilize around a local minimum and so the weights would only suffer small oscillations, or it might ruin everything by escaping the local minimum.