I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: enter image description here Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $W^{[2]}$ is weight matrix of layer 2,
b = bias of a layer. eg: $b^{[2]}$ is bias of layer 2,
Z = Linear function of a layer. eg: $Z^{[2]}$ is linear output of layer 2,
A = Post-activation output of a layer. eg: $A^{[2]}$ is Activation of layer 2,
and Loss/Cost = Cross entropy cost after a Gradient Descent Iteration,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, our calculations will be:

at first layer:
$Z^{[1]} = W^{[1]} . X + b^{[1]}$
$A^{[1]} = relu(Z^{[1]})$

at second layer:
$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$
$A^{[2]} = relu(Z^{[2]})$

at third and output layer:
$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$
$A^{[3]} = \sigma(Z^{[3]})$

Now the cost:
$\mathcal{J} = \mathcal{L} = -\frac{1}{m} \sum\limits_{i = 1}^{m} (y^{(i)}\log\left(a^{[L] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[L](i)}\right))$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at every layer:
$l$ = a given layer, = 1,2,3
$\partial Z^{[l]} = \partial A^{[l]} * g'(Z^{[l]})$
$\partial W^{[l]} = \frac{\partial \mathcal{J} }{\partial W^{[l]}} = \frac{1}{m} \partial Z^{[l]} A^{[l-1] T}$
$\partial b^{[l]} = \frac{\partial \mathcal{J} }{\partial b^{[l]}} = \frac{1}{m} \sum_{i = 1}^{m} \partial Z^{[l](i)}$
$\partial A^{[l-1]} = \frac{\partial \mathcal{L} }{\partial A^{[l-1]}} = W^{[l] T} \partial Z^{[l]}$

And now we use dW and db at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. My primary aim from this question is to know, how to start backpropagation after having computed the cost? What do I compute first and the final layer? And then compute what?

And Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

This is more of a clarification or doubt than a question. Please do not downvote this. I am a beginner trying to grasp concepts of neural networks.

  • $\begingroup$ Backpropagation does not mean to apply all operations in reverse order. It means that you apply the chain rule. I would suggest you take a look at that. Also you are supposed to compute the derivative of the loss with respect to the parameters. $\endgroup$ – matthiaw91 Jun 17 '20 at 10:37
  • $\begingroup$ Yes, I am supposed to compute derivative of loss with respect to weights and bias at all layers. And I am confused in doing that. $\endgroup$ – Naveen Kumar Jun 17 '20 at 10:38
  • $\begingroup$ Are you using any short-hand notation? Because it's hard to understand what your notation means. $\endgroup$ – matthiaw91 Jun 17 '20 at 12:18
  • 1
    $\begingroup$ Okay, so what do you mean when you write e.g. $\partial W$? What is differentiated with respect to what? In the output layer it seems as if you are taking the partial derivative of A with respect to the total number of layers. $\endgroup$ – matthiaw91 Jun 17 '20 at 13:03
  • 1
    $\begingroup$ Okay cool, I will take a swing at a proper answer later today. $\endgroup$ – matthiaw91 Jun 17 '20 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.