# A good reference for the back propagation algorithm?

I'm trying to learn more about the fundamentals of neural networks. I feel like I understand the basics of back propagation, but I want to solidify the details in my mind.

I was working through Ian Goodfellow's famous Deep Learning text. However, I found their exposition of back propagation unsatisfactory. They model the computational graph the wrong way, making the weights into the vertices and the operators into the edges. But this makes no sense, as it is the operations (not the variables) which must have an in-degree and out-degree to them.

I was wondering if there is a better reference, either in a textbook, a paper, or a blog that would rigorously outline the details of back propagation in its full generality.

## 3 Answers

Perhaps we can forget the graph idea to start and just go through the mathematics of it and then we will map our computations to a type of graph just so that it makes it easier to scale.

# One neuron network is called the perceptron

The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output. A common function used is the sigmoid function

$\sigma(z) = \frac{1}{1+exp(z)}$

$\sigma(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$

where $w$ is the associated weight for each input $x$ and we have a bias $b$.

# How do we train these weights

We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$.

$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$

$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.

So we can then update the weights through gradient descent as

$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$

where $\eta$ is the learning rate.

# What if we have 2 layers?

Take a look at the following answer: Compute backpropagation

# Why model neural networks as a graph

In graph theory nodes are connected by vertices. Each neuron takes in some inputs and produces an output. The inputs to this node are not the same as the outputs of the previous layer. They are affected by some weight. Thus we can consider these weights to be the vertices of the graph that link nodes together.

I made little search some days ago to get familiar with some Backpropagation related thing and came across to find this pdf.

In the beginning the author says that the approach is there taken so that it is most understandable.

Directly from Author:

In this chapter we present a proof of the backpropagation algorithm based on a graphical approach in which the algorithm reduces to a graph labeling problem. This method is not only more general than the usual analytical derivations, which handle only the case of special network topologies, but also much easier to follow.

(Author: R. Rojas, 1996)

I find the explanation in this video to be extremely helpful. Hopefully, the first 30 minutes are enough for clarifying some of the most important basics of backpropagation.