How to know the right direction in a noisy environment?
In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and moves in that direction and somehow it works.
I imagine that if there are lots of neurons, that there is a large space of possibilities.
I am electrical engineer, so I am used to encounter noise. I am also new to the topic of neural networks, so pardon me if this is a beginner's question.
I fear that the space of possibilities looks very noisy if I take too close a look. Gradient descent knows only the close look: You get the gradient at THIS micropoint.
Add noise, your derived vector probably points anywhere it wants.
How can I achieve a less fine grain approach, to find a global minimum in a noisy plane, then go down to a better resolution in order to find the local minimum inside the global minimum?
Yes, one probable solution would be to train the network on some points in the near neighbourhood and to estimate the noise from there plus make an average... but that takes a lot of trainings, and those are expensive.
Am I thinking too complicated here?