I'm attempting to create my own neural network optimization model from scratch. I'm getting hung up on backpropagation.
I have just a few basic questions:
When adjusting a given weight by the learning rate, $\alpha$, the weight is selected by changing the weight with the highest $\delta$. Are these deltas all calculated simultaneously and independently? Or is it calculated one layer at a time, or some alternative method?
How is the delta calculated? I am not following any of the calculations since it seems to leave delta unsolved. Maybe I'm misunderstanding. Is it common to use finite differencing to estimate deltas, or is this very slow/inaccurate?
When optimizing on time series training data, is the total error the sum of the output errors of all the observations? If so, is the delta the change in the total error? What is the most efficient way to approach this problem?
If this is inappropriate for this board, apologies in advance.
Edit: Clarifying my question:
Suppose we have 3 inputs, two layers of hidden neurons with Sigmoid activation functions, and 3 neurons per layer, and one output neuron.
I will denote weights as $w_{i,j,k}$ where $i =$ target layer, $j =$ target node, and $k =$ preceding node. Nodes will be denoted as $n_{x,y}$ where $x = $ layer and $y =$ node.
The error for the output is given as $E = \Sigma \cfrac{1}{2} (Target - Out)^2$.
I want to clarify that I am doing this properly. Calculating $\delta$ is simple if the weight targets the output layer. However, if I want to find $\cfrac{\partial E}{\partial w_{2,1,1}}$, is it expanded as:
$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} \cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}}$ where $Out = \cfrac{1}{1+e^{-net_O}}$
$\cfrac{\partial E}{\partial Out} = (Out - Target)$
$\cfrac{\partial Out}{\partial net_O} = (Out)(1-Out)$
$\cfrac{\partial net_O}{\partial n_{2,1}} = w_{3,1,1}$
$\cfrac{\partial n_{2,1}}{\partial net_{2,1}} = n_{2,1}(1-n_{2,1})$
$\cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = n_{1,1}$
Putting it all together:
$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} \cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = $
$(Out - Target)(Out)(1-Out)(w_{3,1,1})(n_{2,1})(1-n_{2,1})(n_{1,1})$
In essence, to calculate $\delta$ for each layer of weights further back in the back propagation, you just add two more terms, being the derivative of the sigmoid for that node, and the node previous, as well as another weight term? Is this correct?