I would like to understand the behaviour of stochastic gradient descent (SGD) over long time periods for a fixed learning rate.
Let us assume that after a number of epochs we expect no further improvement to our cost function. If we continue to train over more epochs then one of two things can happen:
- The solution keeps changing but remains close to a local minimum; in the space of weights it remains in a local region. Or,
- The solution keeps changing but explores a large region of the space of weights.
Let's make this precise. Take a fixed shape of neural network and use the Euclidean distance metric on the space of weights. Now train this network using SGD on a dataset, e.g. MNIST, and for a fixed choice of learning rate. After a large number of epochs, N start recording the weights of the network, $$ {\bf w_N}, {\bf w_{N+1}} \ldots $$ How does $d({\bf w_N}, {\bf w_M})$ behave? To gauge whether these distances are large or small, they can be compared to the distances you get by comparing ${\bf w_N}$ with equivalent networks given by permuting the neurons within a layer.
Ofcourse the answer may vary depending on the neural network, the data, the learning rate, etc. I suppose I would like to know the answer for typical "real world" networks. Unfortunately I lack the processing power (and expertise) to adequately investigate this myself, and am confident this will be known to experts.