Gradient descent in a noisy environment

Question

How to know the right direction in a noisy environment?

In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and moves in that direction and somehow it works.

I imagine that if there are lots of neurons, that there is a large space of possibilities.

I am electrical engineer, so I am used to encounter noise. I am also new to the topic of neural networks, so pardon me if this is a beginner's question.

I fear that the space of possibilities looks very noisy if I take too close a look. Gradient descent knows only the close look: You get the gradient at THIS micropoint.

Add noise, your derived vector probably points anywhere it wants.

How can I achieve a less fine grain approach, to find a global minimum in a noisy plane, then go down to a better resolution in order to find the local minimum inside the global minimum?

Yes, one probable solution would be to train the network on some points in the near neighbourhood and to estimate the noise from there plus make an average... but that takes a lot of trainings, and those are expensive.

Am I thinking too complicated here?

Answer 1

If I understand correctly, your question is whether if you cannot compute the gradient of the function exactly, but only within some errors, then gradient descent method still works in that it gives you convergence to a minimum point?

The answer to this is Yes, if you use Backtracking gradient descent. (And not the standard version of gradient descent, that version works only if you have some strong assumptions on the cost function, such as being convex or having Lipschitz continuous gradients, together with some other assumptions such as the function has compact sub level sets. The point is that if you consider a regularisation technique, as mentioned in the answer by Victor, then even your original function has Lipschitz continuous gradients and has compact sub level sets, after you do regularisation, the new function won't satisfy these assumptions any more.) You can find more details in the paper I mention in my answer in the following link:

Does gradient descent always converge to an optimum?

Answer 2

Have you considered using some regularization / optimization techniques? You could have a look to the Adam Optimizer for gradient descent: https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/

And to some regularization techniques like Dropout: https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/

Of particular interest for you, a commonly used technique is to train the neural network with batches of examples, instead of one-by-one. This is called mini-batch gradient descent, and helps avoiding local minima: https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/

and Batch Normalization: https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c

I hope these information can help you. If you are working in python with a framework like PyTorch or TensorFlow, you will just have to declare a couple of things to use these techniques.

Victor Huerlimann

Answer 3

When training noisy data with stochastic gradient descent (SGD), there are a variety of strategies to increase the chance of reaching a global optimal:

• Increase batchsize. By taking larger samples, each updated estimate is more accurate because it uses more of the data.

• Increase momentum. Momentum accelerates updates in consistent directions. The effect is that the gradient updates will "skip" over noisy sections.