3
$\begingroup$

In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w1, w2, w3 to eventually update them.

We calculate ∂Err/∂w1, ∂Err/∂w2, ∂Err/∂w3 and update the weights as wi = wi + ∂Err / ∂wi for each of the weights so that we move towards the direction where the loss function reduces in measure.

The problem that I see is that there might be situations all the time when some of the weight delta directions conflict in terms of the loss function. That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err. Isn't this the case? What am I missing?

$\endgroup$

1 Answer 1

2
$\begingroup$

That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err.

Isn't this the case?

Not exactly. Using back propagation, the weight gradients may be calculated precisely (for the given training data). It doesn't matter that there are many of them updating at the same time. The gradients don't "conflict" as such, they are 100% compatible. But they are only valid locally to the current values of $w_i$

An update to the all the weights at once in the opposite direction to the gradient is guaranteed to reduce the error value for that training data, with an important caveat that it is only fully guaranteed to make an infinitesimal small improvement, when the step size is also infinitesimal.

If you make an update step that is too large (for some value of "too large" which varies a lot depending on context), then the curve may not remain consistent over the step and your error value could increase.

This problem is not directly related to updating multiple weights at once. It can occur even when you have a single weight. However, when there is a more complex function, with more weights all changing at once, there can be more places where the function does this.

In practice, infinitesimal updates would take too long to train, so you need to find a larger value - but not so large it causes problems - as a compromise.

In addition:

  • The training data you have will usually allow you at best to create a rough approximation to the "true" function you are searching for. So you don't really want to find an actual global minimum in the error function for your training data set, as it would overfit.

  • Typically, using mini-batches, the gradient is also only a rough approximation, so updates only go roughly in the right direction (this can sometimes be good as it can help escape from local minima and saddle points)

$\endgroup$
1
  • $\begingroup$ yeah, I realized that the partial derivatives are independent of each other, and we will always go down the slope in each dimension independently. Thanks for your answer, gives a clear and well rounded picture $\endgroup$
    – Vaibhav
    Jun 29, 2018 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.