# Basic backpropagation question

I'm attempting to create my own neural network optimization model from scratch. I'm getting hung up on backpropagation.

I have just a few basic questions:

1. When adjusting a given weight by the learning rate, $\alpha$, the weight is selected by changing the weight with the highest $\delta$. Are these deltas all calculated simultaneously and independently? Or is it calculated one layer at a time, or some alternative method?

2. How is the delta calculated? I am not following any of the calculations since it seems to leave delta unsolved. Maybe I'm misunderstanding. Is it common to use finite differencing to estimate deltas, or is this very slow/inaccurate?

3. When optimizing on time series training data, is the total error the sum of the output errors of all the observations? If so, is the delta the change in the total error? What is the most efficient way to approach this problem?

If this is inappropriate for this board, apologies in advance.

# Edit: Clarifying my question:

Suppose we have 3 inputs, two layers of hidden neurons with Sigmoid activation functions, and 3 neurons per layer, and one output neuron.

I will denote weights as $w_{i,j,k}$ where $i =$ target layer, $j =$ target node, and $k =$ preceding node. Nodes will be denoted as $n_{x,y}$ where $x =$ layer and $y =$ node.

The error for the output is given as $E = \Sigma \cfrac{1}{2} (Target - Out)^2$.

I want to clarify that I am doing this properly. Calculating $\delta$ is simple if the weight targets the output layer. However, if I want to find $\cfrac{\partial E}{\partial w_{2,1,1}}$, is it expanded as:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} \cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}}$ where $Out = \cfrac{1}{1+e^{-net_O}}$

$\cfrac{\partial E}{\partial Out} = (Out - Target)$

$\cfrac{\partial Out}{\partial net_O} = (Out)(1-Out)$

$\cfrac{\partial net_O}{\partial n_{2,1}} = w_{3,1,1}$

$\cfrac{\partial n_{2,1}}{\partial net_{2,1}} = n_{2,1}(1-n_{2,1})$

$\cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = n_{1,1}$

Putting it all together:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} \cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} =$

$(Out - Target)(Out)(1-Out)(w_{3,1,1})(n_{2,1})(1-n_{2,1})(n_{1,1})$

In essence, to calculate $\delta$ for each layer of weights further back in the back propagation, you just add two more terms, being the derivative of the sigmoid for that node, and the node previous, as well as another weight term? Is this correct?

• I suggest start with a simpler, more focused question. Your assertion in first question "When adjusting a given weight by the learning rate, α, the weight is selected by changing the weight with the highest δ." does not seem correct to me. Unpicking all your misunderstandings and figuring out an answer to the whole lot looks like a lot of work, and risks going off in wrong direction to help you. There are loads of tutorials for neural networks and understanding backprop. Perhaps pick one, link/summarise it, then explain at what point it stops making sense and you'd like some help. – Neil Slater Jan 23 '17 at 22:23
• Here is a simpler question: Based on the tutorial I'm reading, supposing just one layer of neurons and a Sigmoid activation function, for node $k$, the $\Delta_k = E(O_k) - O_k$ , and $\delta_k = \Delta_k O_k (1-O_k)$. How are you supposed to calculate $\delta_k$ if there is no clear $E(O_k)$ for node $k$, and you can only calculate $\Delta$ for the output variable at the last "layer"? That is the origin of my question 1. Do you just substitute $\Delta_{output}$ for $\Delta_k$? Otherwise, how do I determine $E(O_k)$? – milkmotel Jan 23 '17 at 23:49
• @milkmotel Please edit your question, not in the comment. – SmallChess Jan 24 '17 at 3:04
• I've edited with a more fleshed out question – milkmotel Jan 24 '17 at 5:24

To make our life easier, let's omit optimization methods here (e.g. Learning Rate, Momentum) and focus on vanilla Backpropagation. As you know, backprop consists of computing the partial derivative with respect to each weight in order to penalize each weight for its role in the global output error; such that:

\begin{equation} \frac{\partial E}{\partial W_{ij}} = e_{i}y_{j} \end{equation}

With $E$ the global output error to minimize, $W$ the weight to update, $y$ the output value, and $e$ the local error. The local error $e$ is the unknown we want to compute! :)

Using the chain rule, the gradient can be expressed as: \begin{equation} \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} \end{equation}

With $x_{i}$ the input value to neuron $i$.

First, the partial derivative with respect to $W_{ij}$ can be computed as follows:

\begin{equation} \frac{\partial x_{i}}{\partial W_{ij}} = y_{j} \end{equation}

Which is simply equal to the output value $y_{j}$!

Secondly, the partial derivative with respect to $x_{i}$ is:

\begin{equation} \frac{\partial y_{i}}{\partial x_{i}} = \frac{\partial \phi(x_i)}{\partial x_{i}} \end{equation}

Which is simply $\frac{\partial \phi(x_i)}{\partial x_{i}}$, namely the derivative of the activation function (i.e. Sigmoid, Tanh, ReLU...).

Thirdly, the partial derivative with respect to $y_{i}$ can be computed as follows:

\begin{equation} \frac{\partial E}{\partial y_{i}} = \begin{cases} \begin{aligned} \frac{\partial}{\partial y_{i}} (T_i - y_i) \end{aligned} & \text{if $i \in$ output layer,} \\ \begin{aligned} \frac{\partial}{\partial y_{i}} \left(\sum\limits^{n}_{j=1}W_{ij}\frac{\partial E}{\partial y_{j}}\right) \end{aligned} & \text{otherwise;} \end{cases} \end{equation}

With $T$ the target expected output (i.e. the label). The local error $e$ is computed differently depending on the layer position. For the output layer, the error is proportional to the difference between the predicted value (i.e. $y_{i}$) and the expected output (i.e. $T_{i}$). Otherwise, the error in hidden layers is proportional to the weighted sum of errors from connected layers.

Fourthly, combining the partial derivatives allow the computation of the error at a given neuron $i$ such as:

\begin{equation} \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} = e_{i} = \begin{cases} \begin{aligned} \frac{\partial \phi(x_i)}{\partial x_{i}} (T_i - y_i) \end{aligned} & \text{if $i \in$ output layer,} \\ \begin{aligned} \frac{\partial \phi(x_i)}{\partial x_{i}} \left(\sum\limits^{n}_{j=1}W_{ij} e_{j}\right) \end{aligned} & \text{otherwise;} \end{cases} \end{equation}

As you can see here, you need to recursively compute the local error $e$ all the way to the input layer, that's why it's called backprop! ;) I think that what you missed!

If we now simplify:

\begin{equation} \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} = e_{i}y_{j} \end{equation}

We already know $y_{j}$, and we now know how to compute $e_{i}$ all the way to the input layer. So the weight can finally be updated from the gradient as follows:

\begin{equation} W_{ij} = W_{ij} - \,e_{i}y_{j} \end{equation}

Sources:

• I see. So I should calculate the error of each node separately from the $\delta$. If the cost function is, $C = \Sigma \frac{1}{2}(Target - Out)^2$ then the error of each output node is $(Out - Target)$, and the error of each node in the previous layers is the weighted sum of the errors in the layer following? Then you just multiply $\frac{\partial x_i}{\partial W_{ij}} = y_j$ for whatever node you are on to get the $\delta$ for that node? – milkmotel Jan 24 '17 at 14:59
• @milkmotel: I think this answer has re-visited how backprop works in detail. Two things 1) You don't actually work with the error directly at any point, instead you always use the partial derivative of the error w.r.t. parameters, including all activations and weights (the weights are of course the things you change, but activation deltas are needed along the way). 2) At one point I put together a tutorial as part of NN that I was implementing, it covers the same ground and might help along with links provided by tony: github.com/neilslater/ru_ne_ne/blob/master/tutorials/… – Neil Slater Jan 24 '17 at 18:50
• @NeilSlater: Apologies for the poor wording. I think the confusion was that I misunderstood $\delta$ to be $\cfrac {\partial E}{\partial W_k}$, when it is rather $\cfrac {\partial E }{\partial n_{k+1}}$, so the actual correction to a weight is not $W_k - \alpha \delta$ but rather $W_k - \alpha \delta n_k$ – milkmotel Jan 24 '17 at 21:18
• So the actual calculation is very simple. You can easily calculate $\delta$ for each node by calculating the $\delta$ for the output neurons and then doing the weighted sums multiplied by the derivative of the activation function of the target node into the network until you arrive at the first layer. And then you can solve for $\cfrac {\partial E}{\partial W_k}$ by multiplying by the value of the preceding node? – milkmotel Jan 24 '17 at 21:28
• @milkmotel: The symbol/variable $\delta$ does not actually appear in tony's answer here, and you don't define or use it in your question, just refer to it in isolation. So I cannot tell if that was your mistake. – Neil Slater Jan 24 '17 at 21:59