Backpropagation: Relevance of the error signal of a neuron

Question

During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer: \begin{eqnarray} \delta^l_j \equiv \frac{\partial C}{\partial z^l_j} \tag{1}\end{eqnarray}

Basically, $\delta^l_j$ measures how much the total error changes when the input sum of the neuron is changed and is used for calculating the weights and biases of the neural network as follows:

\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \tag{2}\end{eqnarray}

\begin{eqnarray} \frac{\partial C}{\partial b^l_j} = \delta^l_j. \tag{3}\end{eqnarray}

Besides being useful for the calculation of the weights and the biases in the sense it makes it possible to reuse its value several times, is there any other reason why this definition is always brought up when addressing back propagation?

Answer 1

Let me explain one thing very clearly "NEURAL NETWORK ARE VERY SIMPLE STRUCTURES AND YET VERY EFFICIENT ALGORITHM WHICH GIVES AMAZINGLY GREAT RESULTS AND IF YOU SEE CLOSELY NEURAL NETWORKS ARE ACTUALLY BIG COMPOSITE FUNCTIONS WHICH REQUIRE YOU TO CALCULATE GRADIENTS SO THAT YOU CAN OBSERVE HOW THE OUTPUT VARIES ACCORDING TO YOUR INPUT WEIGHTS(That is what is backpropagation achieved using gradient descent) THAT IS SIMPLY THE GOAL I.E. TO CALCULATE DERIVATIVES OF THE TWO EQUATIONS YOU HAVE MENTIONED (2) AND (3)".

Now as you stated you can calculate its value and reuse it i.e. eq(1) because eq (1) here gives you the error at a neuron j in layer 'l' with respect to that neuron's weighted sum of inputs it is useful and necessary for calculation of weights when the loss is calculated at the'Lth layer(last layer)' it is then backpropagated to the L-1 th layer where gradient is calculated and then weights are updated it is the eq(1) that allows you to compute the gradients of a neuron j in layer 'l' and then the weight matrix of that neuron j is updated so essentially eq(1) allows you to use eq(2) and eq(3) to optimize your weights in order to minimize your loss function.

One more thing if you calculate its derivative that is of eq(1) its partial derivative it will be: first-rate of change of cost function w.r.t to activation function * rate of change of activation function w.r.t to weighted sum of inputs * rate of change of weighted sum of inputs w.r.t to weights of your neuron so this is essentially a chain in itself which gets longer and longer as the loss is backpropagated through l-1,l-2,l-3.... layers and so on, thus eq(1) is actually calculated using the first two in the chain stated above which is then used to calculate the rate of change of weights w.r.t to your loss function. This is why whenever you read about backprop this equation is bound to be seen as it is kind of connection mechanism that allows you to relate your loss with neurons in the subsequent layers during backpropagation and subsequently update their weights. Hence according to me, eq(1) in itself is the true essence of Backpropagation using gradient descent.

Answer 2

The goal of back-propagation is to find the weights that minimize the cost function, that is, the weights for which the cost function calculates a cost that is as low as possible.

A cost function of a neural net has a complex shape (in all cases except trivial examples). Therefore, you cannot calculate its minimum directly using algebra. You have to find it iteratively, by each time finding a set of weights that lowers the cost function a bit. This is done by increasing the weights if the derivative (1) is negative and decreasing them if it is positive. This numerical procedure is called back-propagation, and because the derivatives are such a central part of it, they are always mentioned in the explanation.

Storing and reusing values of the derivatives is not needed, but makes the computation faster. It’s what is called an “implementation detail”.

Answer 3

As far as I can understand, you know the derivation and just want to get why we use error signal whenever we talk about backpropagation. You said correctly that it is helpful for calculation of weights and biases and that is exactly the main reason why it is standardised. Here is a neat derivation that uses the error signal:

The process of backpropagating the error signal can iterate all the way back to the input layer by successively projecting $\delta_k$(error signal for output layer) back through $w_{jk}$, then through the activation function for the hidden layer via $g'_j$ to give the error signal $\delta_j$, and so on. This backpropagation concept is central to training neural networks with more than one layer.

Also, it is useful to save us from computing the same values again and again. If we do not standardize this concept, we would be calculating the same values repetitively and as we go deeper into the network, calculating the gradients will become more cumbersome.

Using the concept of error signal to calculate the weight gradients at any layer $l$ in an arbitrarily-deep neural network, we simply need to calculate the backpropagated error signal that reaches that layer $\delta_l$ and weight it by the feed-forward signal $a_{l-1}$ feeding into that layer.

Hence, using the error signal is an integral part of the backpropagation technique and forms the basis of it's derivation!

Answer 4

I guess you have fellowed this post CHAPTER 2 How the backpropagation algorithm works.

$$ \begin{eqnarray} \delta^L = (a^L-y) \odot \sigma'(z^L). \tag{30}\end{eqnarray} $$,

$$ \begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l), \tag{BP2}\end{eqnarray} $$

including your formula $2$ and $3$ is called The four fundamental equations behind backpropagation in that great post.

And from the $\text{BP2}$ formula we can tell that $\delta^{l+1}$ drive to $\delta^{l}$ directly, so there is no need to calculate the gradient of $z^l$ from cost function again.

What's more, once we got $\delta^{l}$, we could calculate $\delta^{l-1}$, $\frac{\partial C}{\partial w^l}$ and $\frac{\partial C}{\partial w^l}$ according to formula $\text{BP2}$ and your formula $2, 3$. Then we keep going to get gradient of all parameters.