# clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $$W^{[2]}$$ is weight matrix of layer 2,
b = bias of a layer. eg: $$b^{[2]}$$ is bias of layer 2,
Z = Linear function of a layer. eg: $$Z^{[2]}$$ is linear output of layer 2,
A = Post-activation output of a layer. $$A^{[2]}$$ is Activation of layer 2,
$$^{T}$$ = transpose of a matrix. eg: if $$A$$ is a matrix, $$A^{T}$$ is transpose of this matrix, and Loss/Cost = Cross entropy cost after a Gradient Descent Iteration,
sigma = mathematical sigma used for summation,
relu = relu activation function,
$$\sigma$$= sigmoid activation function,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, our calculations will be:

at first layer:
$$Z^{[1]} = W^{[1]} . X + b^{[1]}$$
$$A^{[1]} = relu(Z^{[1]})$$

at second layer:
$$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$$
$$A^{[2]} = relu(Z^{[2]})$$

at third and output layer:
$$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$$
$$A^{[3]} = \sigma(Z^{[3]})$$

Now the cost:
$$\mathcal{J} = \mathcal{L} = -\frac{1}{m} \sum\limits_{i = 1}^{m} (y^{(i)}\log\left(a^{[L] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[L](i)}\right))$$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at every layer:
$$l$$ = a given layer, = 1,2,3
$$\partial Z^{[l]} = \partial A^{[l]} * g'(Z^{[l]})$$
$$\partial W^{[l]} = \frac{\partial \mathcal{J} }{\partial W^{[l]}} = \frac{1}{m} \partial Z^{[l]} A^{[l-1] T}$$
$$\partial b^{[l]} = \frac{\partial \mathcal{J} }{\partial b^{[l]}} = \frac{1}{m} \sum_{i = 1}^{m} \partial Z^{[l](i)}$$
$$\partial A^{[l-1]} = \frac{\partial \mathcal{L} }{\partial A^{[l-1]}} = W^{[l] T} \partial Z^{[l]}$$

And now we use dW and db at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. My primary aim from this question is to know, how to start backpropagation after having computed the cost? What do I compute first and the final layer? And then compute what?

And Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

This is more of a clarification or doubt than a question. Please do not downvote this. I am a beginner trying to grasp concepts of neural networks.

• Backpropagation does not mean to apply all operations in reverse order. It means that you apply the chain rule. I would suggest you take a look at that. Also you are supposed to compute the derivative of the loss with respect to the parameters. – matthiaw91 Jun 17 at 10:37
• Yes, I am supposed to compute derivative of loss with respect to weights and bias at all layers. And I am confused in doing that. – Naveen Kumar Jun 17 at 10:38
• Are you using any short-hand notation? Because it's hard to understand what your notation means. – matthiaw91 Jun 17 at 12:18
• Okay, so what do you mean when you write e.g. $\partial W$? What is differentiated with respect to what? In the output layer it seems as if you are taking the partial derivative of A with respect to the total number of layers. – matthiaw91 Jun 17 at 13:03
• Okay cool, I will take a swing at a proper answer later today. – matthiaw91 Jun 17 at 13:18