2
$\begingroup$

I've just finished conceptually studying linear and logistic regression functions and their optimization as preparation for neural networks.


For example, say we are performing binary classification with logistic regression, let's define variables:

$x$ - vector containing all inputs.

$y$ - vector containing all outputs.

$w_{0}$ - bias weight variable.

$W=(w_1,...,w_{2})$ - vector containing all weight variables.

$f(x_i)=w_{0}+\sum_{i=1}x_{i}w_{i}=w_{0}+x^{T}W$ - summation of all weight variables.

$p(x_{i})=\frac{1}{1+e^{-f(x_i)}}$ - logistic activation function (sigmoid), representing conditional probability that $y_i$ will be 1 given observed values in $x_i$.

$L=-\frac{1}{N} \sum^{N}_{i=0} y_i*ln(p(x_i))+(1-y_i)*ln(1-p(x_i))$ - binary cross entropy loss function (Kullback-Leibler divergence of Bernoulli random variables plus entropy of activation function representing probability)

$L$ is multi-dimensional function, so it must be differentiated with partial derivative, being:

$$\frac{\partial{L}}{\partial{w}}$$

Then, the chain rule gives:

$$\frac{\partial{L}}{\partial{w_1}}=\frac{\partial{L}}{\partial{p_i}} \frac{\partial{p_i}}{\partial{w_1}}$$

After doing few calculations, derivative of the loss function is:

$$(y_i-p_i)*x_i$$


So we got derivative of the loss function, and all weights are trained separately with gradient descent.

What does backpropagation have to do with this? To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?

In short

We already have derivative calculated, so what's the point of calculating them on every step when we can just plug in the variables?

Is backpropagation just fancy term for weights being optimized on every iteration?

$\endgroup$
  • 1
    $\begingroup$ Each iteration, you take a small step, you have to take so many steps to reach to an optimal point. $\endgroup$ – Media Sep 5 '18 at 5:54
  • $\begingroup$ @Media Yes, I understand how cost function is weights are optimized so that it eventually converges at local minima, but partial derivative (roadmap, gradient) is already calculated, what does backpropagation do with this? Why is automatic differentiation necessary in this? Thank you $\endgroup$ – ShellRox Sep 5 '18 at 6:36
  • $\begingroup$ Basically, in backpropagation algorithm, you find the derivatives of each parameter with respect to the cost function and after updating each parameter, you can take next steps. You use learning rate for taking controlled steps. $\endgroup$ – Media Sep 5 '18 at 6:43
  • 1
    $\begingroup$ @Media Oh, so partial derivative must be re-calculated with respect of each weight, but isn't minimization of each parameter done in different process, so if there are say 5 parameters, 5 minimization processes for each. I assume when weights are changing, gradient must be alsp re-calculated during that change with respect to weight. So backpropagation is basicay re-calculation of gradient on each step? $\endgroup$ – ShellRox Sep 5 '18 at 6:55
  • 2
    $\begingroup$ I guess you have not understand the math behind it. Suppose you have two parameters and the cost function which changes with respect to the weights is a hill. Each time you change each weight, your cost changes. In gradeint descent based approaches, you find the derivative alongside each parameter and change the current value of each parameter simoltaneously. Next you take another step. $\endgroup$ – Media Sep 5 '18 at 7:08
4
$\begingroup$

Is backpropagation just fancy term for weights being optimized on every iteration?

Almost. Backpropagation is a fancy term for using the chain rule.

It becomes more useful to think of it as a separate thing when you have multiple layers, as unlike your example where you apply the chain rule once, you do need to apply it multiple times, and it is most convenient to apply it layer-by-layer in reverse order to the feed forward steps.

For instance, if you have two layers, $l$ and $l-1$ with weight matrix $W^{(l)}$ linking them, non-activated sum for a neuron in each layer $z_i^{(l)}$ and activation function $f()$, then you can link the gradients at the sums (often called logits as they may be passed to logistic activation function) between layers with a general equation:

$$ \frac{\partial L}{\partial z^{(l-1)}_j} = f'(z^{(l-1)}_j) \sum_{i=1}^{N^{(l)}} W_{ij}^{(l)} \frac{\partial L}{\partial z^{(l)}_i}$$

This is just two steps of the chain rule applied to generic equations of the feed-forward network. It does not provide the gradients of the weights, which is what you eventually need - there is a separate step for that - but it does link together layers, and is a necessary step to eventually obtain the weights. This equation can be turned into an algorithm that progressively works back through layers - that is back propagation.

To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?

That is exactly what automatic differentiation is doing. Essentially "automatic differentiation" = "the chain rule", applied to function labels in a directed graph of functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.