Purpose of backpropagation in neural networks

Question

I've just finished conceptually studying linear and logistic regression functions and their optimization as preparation for neural networks.

---

For example, say we are performing binary classification with logistic regression, let's define variables:

$x$ - vector containing all inputs.

$y$ - vector containing all outputs.

$w_{0}$ - bias weight variable.

$W=(w_1,...,w_{2})$ - vector containing all weight variables.

$f(x_i)=w_{0}+\sum_{i=1}x_{i}w_{i}=w_{0}+x^{T}W$ - summation of all weight variables.

$p(x_{i})=\frac{1}{1+e^{-f(x_i)}}$ - logistic activation function (sigmoid), representing conditional probability that $y_i$ will be 1 given observed values in $x_i$.

$L=-\frac{1}{N} \sum^{N}_{i=0} y_i*ln(p(x_i))+(1-y_i)*ln(1-p(x_i))$ - binary cross entropy loss function (Kullback-Leibler divergence of Bernoulli random variables plus entropy of activation function representing probability)

$L$ is multi-dimensional function, so it must be differentiated with partial derivative, being:

$$\frac{\partial{L}}{\partial{w}}$$

Then, the chain rule gives:

$$\frac{\partial{L}}{\partial{w_1}}=\frac{\partial{L}}{\partial{p_i}} \frac{\partial{p_i}}{\partial{w_1}}$$

After doing few calculations, derivative of the loss function is:

$$(y_i-p_i)*x_i$$

---

So we got derivative of the loss function, and all weights are trained separately with gradient descent.

What does backpropagation have to do with this? To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?

In short

We already have derivative calculated, so what's the point of calculating them on every step when we can just plug in the variables?

Is backpropagation just fancy term for weights being optimized on every iteration?

Answer 1

Is backpropagation just fancy term for weights being optimized on every iteration?

Almost. Backpropagation is a fancy term for using the chain rule.

It becomes more useful to think of it as a separate thing when you have multiple layers, as unlike your example where you apply the chain rule once, you do need to apply it multiple times, and it is most convenient to apply it layer-by-layer in reverse order to the feed forward steps.

For instance, if you have two layers, $l$ and $l-1$ with weight matrix $W^{(l)}$ linking them, non-activated sum for a neuron in each layer $z_i^{(l)}$ and activation function $f()$, then you can link the gradients at the sums (often called logits as they may be passed to logistic activation function) between layers with a general equation:

$$ \frac{\partial L}{\partial z^{(l-1)}_j} = f'(z^{(l-1)}_j) \sum_{i=1}^{N^{(l)}} W_{ij}^{(l)} \frac{\partial L}{\partial z^{(l)}_i}$$

This is just two steps of the chain rule applied to generic equations of the feed-forward network. It does not provide the gradients of the weights, which is what you eventually need - there is a separate step for that - but it does link together layers, and is a necessary step to eventually obtain the weights. This equation can be turned into an algorithm that progressively works back through layers - that is back propagation.

To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?

That is exactly what automatic differentiation is doing. Essentially "automatic differentiation" = "the chain rule", applied to function labels in a directed graph of functions.