# Why averaging the gradient works in Gradient Descent?

In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally use it to correct the network, at once.

But why does averaging the gathered gradient work? Each training sample ends up in a distant, completely separate location on the error-surface

Each sample would thus have its direction of a steepest descent pointing in different directions compared to other training examples. Averaging those directions should not make sense? Yet it works so well. In fact, the more examples we average, the more precise the correction will be: full-batch vs mini-batch approach

• This is a good question. There are many explanation articles about gradient descent optimisation but few talk about the relation between data points and the optimisation clearly. – chichi Nov 28 '20 at 10:18

## 2 Answers

Each training sample ends up in a distant, completely separate location on the error-surface

That is not a correct visualisation of what is going on. The error surface plot is tied to the value of the network parameters, not to the values of the data inputs. During back-propagation of an individual item in a mini-batch or full batch, each example gives an estimate of the gradient in the same location in parameter space. The more examples you use, the better the estimate will be (more on that below).

A more accurate representation of what is going on would be this:

Your question here is still valid though:

But why does averaging the gathered gradient work?

In other words, why do you expect that taking all these individual gradients from separate examples should combine into a better approximation of the average gradient over the error surface?

This is entirely to do with how the error surface is itself constructed as the average of individual loss functions. If we note cost function for the error surface as $C$, then

$$C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} L(x, \theta)$$

Where $X$ represents the whole dataset, $\theta$ are your model's trainable parameters, $L$ is an individual loss function for $x$. Note I have rolled labels into $X$ here, it doesn't matter for this argument whether loss is due to comparison of model output with some part of the training data - all we care about is finding a gradient to the error surface.

The error gradient that you want to calculate for gradient descent is $\nabla_{\theta} C(X, \theta)$, which you can therefore write as:

$$\nabla_{\theta} C(X, \theta) = \nabla_{\theta}(\frac{1}{|X|}\sum_{x \in X} L(x, \theta))$$

The derivative of the sum of any two functions is the sum of the derivatives, i.e.

$$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$$

In addition, any fixed multiplier that doesn't depend on the parameters you are taking the gradient with (in this case, the size of the dataset) can just be treated as an external factor:

$$\nabla_{\theta} C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} \nabla_{\theta} L(x, \theta)$$

So . . . the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately. Taking any completely random subset of $X$ will result in an unbiased estimate of the mean gradient, same as taking a random subset of any variable and taking its mean will give you an unbiased estimate of the population's mean.

This will not work if your samples are somehow correlated, hence why you will often see recommendations to shuffle the data prior to training, in order to make it i.i.d.

This will also not work if your cost function combines examples in any way other than addition. However, it would be an unusual cost function that combined separate training examples by multiplying their loss functions, or some other non-linear combination.

• Oh, so although the Cost is working via two arguments (the X set and the weights of network θ), any time we mention "gradient" we automatically imply its the gradient for θ. We never bother computing the second parameter - for the set of examples. And also, thanks for "the error surface is itself constructed as the average of individual loss functions" – Kari Jun 22 '18 at 18:46
• So thus my image doesn't depict the full situation. Because on one axis should be the set of all examples, and on the other - the set of all current weights. And since they are sets, there is no way each of them could fit on its own single, separate axis in a 2D graph. – Kari Jun 22 '18 at 18:52
• So when we talk about the error surface, we assume working with 2-weights network, and have one axis as the θ1, the other axis as θ2, and height is the average of errors from all examples. All of this while the θ set remains constant as we are iterating over examples. We never get a chance to show the set of examples X on such a graph - is my understanding correct? – Kari Jun 22 '18 at 19:03
• @Kari: Yes your last comment is right. The error surface is for the parameters that you want to optimise due to the gradient, and you do not (usually) want to change X, so you don't need its gradient, and you don't take update steps in X. The surface plotted is implied to be for the cost function across the whole data set normally, or if you are generalising then conceptually it could be for the whole population (and even a whole batch is an approximation). You could plot each $L()$ separately and they may look very different . . . but you are not interested in optimising $L()$ separately – Neil Slater Jun 22 '18 at 20:29

When we correct the network using the "high-quality "gradient we are trying to get that one unified weight vector (or mininma of the error surface) which would fit the entire data i.e. all the training examples.

The gradient from each separate example (though having different direction on the error surface initially) aims at achieving the minima on error surface. Hence, averaging them makes sense as it will give average direction towards that minima which is better that the individual gradients.

I am not an expert in the field but according to me this is the probable answer. Hope it helps.

EDIT : As learned from the comments you diagram is wrong as "The error surface is always over model parameters, not over examples". So, if you meant gradients in different directions the above explanation will help you.

• The key to this is "different direction" whilst OP is showing the gradients at different locations on the error surface. The OP may need some guidance as to what the error surface represents in order to understand why averaging those directions works. – Neil Slater Jun 22 '18 at 7:09
• @Neil Slater Those are the gradients for separate examples, so aren't both the same thing ? – bkshi Jun 22 '18 at 7:13
• The OP's diagram shows gradients in different locations on the error surface. That is not the same thing as gradient estimates in different directions but at the same location. Your answer correctly identifies direction as the key property, whilst location is not changing as OP shows it. – Neil Slater Jun 22 '18 at 7:15
• But the location here can be interpreted as the magnitude of gradient which still works while averaging. I have updated my answer regarding the same. – bkshi Jun 22 '18 at 7:34
• "i'm assuming the gradients you have shown are for separate examples, hence the different locations." That is wrong in the same way as OP is wrong. The error surface is always over model parameters, not over examples. It makes no sense to plot an error surface over examples, and none of the literature showing error surfaces does so - all examples you can find are over the parameters to optimise. – Neil Slater Jun 22 '18 at 8:28