Basic backpropagation question

Question

I'm attempting to create my own neural network optimization model from scratch. I'm getting hung up on backpropagation.

I have just a few basic questions:

1. When adjusting a given weight by the learning rate, $\alpha$, the weight is selected by changing the weight with the highest $\delta$. Are these deltas all calculated simultaneously and independently? Or is it calculated one layer at a time, or some alternative method?

2. How is the delta calculated? I am not following any of the calculations since it seems to leave delta unsolved. Maybe I'm misunderstanding. Is it common to use finite differencing to estimate deltas, or is this very slow/inaccurate?

3. When optimizing on time series training data, is the total error the sum of the output errors of all the observations? If so, is the delta the change in the total error? What is the most efficient way to approach this problem?

If this is inappropriate for this board, apologies in advance.

Edit: Clarifying my question:

Suppose we have 3 inputs, two layers of hidden neurons with Sigmoid activation functions, and 3 neurons per layer, and one output neuron.

I will denote weights as $w_{i,j,k}$ where $i =$ target layer, $j =$ target node, and $k =$ preceding node. Nodes will be denoted as $n_{x,y}$ where $x = $ layer and $y =$ node.

The error for the output is given as $E = \Sigma \cfrac{1}{2} (Target - Out)^2$.

I want to clarify that I am doing this properly. Calculating $\delta$ is simple if the weight targets the output layer. However, if I want to find $\cfrac{\partial E}{\partial w_{2,1,1}}$, is it expanded as:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} \cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}}$ where $Out = \cfrac{1}{1+e^{-net_O}}$

$\cfrac{\partial E}{\partial Out} = (Out - Target)$

$\cfrac{\partial Out}{\partial net_O} = (Out)(1-Out)$

$\cfrac{\partial net_O}{\partial n_{2,1}} = w_{3,1,1}$

$\cfrac{\partial n_{2,1}}{\partial net_{2,1}} = n_{2,1}(1-n_{2,1})$

$\cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = n_{1,1}$

Putting it all together:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} \cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = $

$(Out - Target)(Out)(1-Out)(w_{3,1,1})(n_{2,1})(1-n_{2,1})(n_{1,1})$

In essence, to calculate $\delta$ for each layer of weights further back in the back propagation, you just add two more terms, being the derivative of the sigmoid for that node, and the node previous, as well as another weight term? Is this correct?

Answer 1

To make our life easier, let's omit optimization methods here (e.g. Learning Rate, Momentum) and focus on vanilla Backpropagation. As you know, backprop consists of computing the partial derivative with respect to each weight in order to penalize each weight for its role in the global output error; such that:

\begin{equation} \frac{\partial E}{\partial W_{ij}} = e_{i}y_{j} \end{equation}

With $E$ the global output error to minimize, $W$ the weight to update, $y$ the output value, and $e$ the local error. The local error $e$ is the unknown we want to compute! :)

Using the chain rule, the gradient can be expressed as: \begin{equation} \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} \end{equation}

With $x_{i}$ the input value to neuron $i$.

First, the partial derivative with respect to $W_{ij}$ can be computed as follows:

\begin{equation} \frac{\partial x_{i}}{\partial W_{ij}} = y_{j} \end{equation}

Which is simply equal to the output value $y_{j}$!

Secondly, the partial derivative with respect to $x_{i}$ is:

\begin{equation} \frac{\partial y_{i}}{\partial x_{i}} = \frac{\partial \phi(x_i)}{\partial x_{i}} \end{equation}

Which is simply $\frac{\partial \phi(x_i)}{\partial x_{i}}$, namely the derivative of the activation function (i.e. Sigmoid, Tanh, ReLU...).

Thirdly, the partial derivative with respect to $y_{i}$ can be computed as follows:

\begin{equation} \frac{\partial E}{\partial y_{i}} = \begin{cases} \begin{aligned} \frac{\partial}{\partial y_{i}} (T_i - y_i) \end{aligned} & \text{if $i \in$ output layer,} \\ \begin{aligned} \frac{\partial}{\partial y_{i}} \left(\sum\limits^{n}_{j=1}W_{ij}\frac{\partial E}{\partial y_{j}}\right) \end{aligned} & \text{otherwise;} \end{cases} \end{equation}

With $T$ the target expected output (i.e. the label). The local error $e$ is computed differently depending on the layer position. For the output layer, the error is proportional to the difference between the predicted value (i.e. $y_{i}$) and the expected output (i.e. $T_{i}$). Otherwise, the error in hidden layers is proportional to the weighted sum of errors from connected layers.

Fourthly, combining the partial derivatives allow the computation of the error at a given neuron $i$ such as:

\begin{equation} \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} = e_{i} = \begin{cases} \begin{aligned} \frac{\partial \phi(x_i)}{\partial x_{i}} (T_i - y_i) \end{aligned} & \text{if $i \in$ output layer,} \\ \begin{aligned} \frac{\partial \phi(x_i)}{\partial x_{i}} \left(\sum\limits^{n}_{j=1}W_{ij} e_{j}\right) \end{aligned} & \text{otherwise;} \end{cases} \end{equation}

As you can see here, you need to recursively compute the local error $e$ all the way to the input layer, that's why it's called backprop! ;) I think that what you missed!

If we now simplify:

\begin{equation} \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} = e_{i}y_{j} \end{equation}

We already know $y_{j}$, and we now know how to compute $e_{i}$ all the way to the input layer. So the weight can finally be updated from the gradient as follows:

\begin{equation} W_{ij} = W_{ij} - \,e_{i}y_{j} \end{equation}

Sources:

• https://www.youtube.com/watch?v=i94OvYb6noo
• Backpropagation, Intuitions: http://cs231n.github.io/optimization-2/
• Chapter 9.2 Classification by Backpropagation in Data mining: concepts and techniques: concepts and techniques. Jiawei Han, Micheline Kamber, and Jian Pei. 2011.