Conflicting directions of weights gradients in gradient descent?

Question

In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w₁, w₂, w₃ to eventually update them.

We calculate ∂Err/∂w₁, ∂Err/∂w₂, ∂Err/∂w₃ and update the weights as w_i = w_i + ∂Err / ∂w_i for each of the weights so that we move towards the direction where the loss function reduces in measure.

The problem that I see is that there might be situations all the time when some of the weight delta directions conflict in terms of the loss function. That is, maybe Err reduces when w₁ goes towards ∂Err/∂w₁ alone, but it might well be the case that Err actually increases when w₁ is updated along with w₂, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err. Isn't this the case? What am I missing?

Answer 1

That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err.

Isn't this the case?

Not exactly. Using back propagation, the weight gradients may be calculated precisely (for the given training data). It doesn't matter that there are many of them updating at the same time. The gradients don't "conflict" as such, they are 100% compatible. But they are only valid locally to the current values of $w_i$

An update to the all the weights at once in the opposite direction to the gradient is guaranteed to reduce the error value for that training data, with an important caveat that it is only fully guaranteed to make an infinitesimal small improvement, when the step size is also infinitesimal.

If you make an update step that is too large (for some value of "too large" which varies a lot depending on context), then the curve may not remain consistent over the step and your error value could increase.

This problem is not directly related to updating multiple weights at once. It can occur even when you have a single weight. However, when there is a more complex function, with more weights all changing at once, there can be more places where the function does this.

In practice, infinitesimal updates would take too long to train, so you need to find a larger value - but not so large it causes problems - as a compromise.

In addition:

• The training data you have will usually allow you at best to create a rough approximation to the "true" function you are searching for. So you don't really want to find an actual global minimum in the error function for your training data set, as it would overfit.

• Typically, using mini-batches, the gradient is also only a rough approximation, so updates only go roughly in the right direction (this can sometimes be good as it can help escape from local minima and saddle points)