I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).

Suppose that the network is simple like so (forward pass):

$$\begin{aligned}z_1 &= xW_1 + b_1 & (1) \\ a_1 &= \tanh{(z_1)} & (2) \\ z_2 & = a_1W_2 + b_2 & (3)\\ a_2 & = \text{softmax}(z_2)=\hat{y}_n & (4) \\ l_n & = \log{(\hat{y}_n)} & (5) \\ PL_n & = l_n\dot{y}_n^T & (6) \\ L & = -\frac{1}{N} \sum_{n \in N} PL_n & (7) \end{aligned} $$

So only one hidden layer. Let the dimensions be the following:

$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.

Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):

$$\begin{aligned} dPL_n & = -1/N & (7) \\ d\dot{y}^T_n & = l_n^T dPL_n & (6) \\ dl_n & = \dot{y}^T_n dPL_n & (6) \\ d\hat{y}_n & = 1/\hat{y}_n & (5) \end{aligned} $$

Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?

(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).

Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(

I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.

I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2 = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &= \frac{\partial l_n}{\partial z_2} dl_n \\ &= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$

Also, same question on Maths.SE.


1 Answer 1


The partial derivative of the cost function with respect to the neuron $i$ ,in the layer $Z_2$ defined in your questions above, is $\frac{\partial L}{\partial z_i}$, where $L$ is the cost function.

please note that the derivation will be on the neuron level not at the layer level, where $z_i$ is the neuron number $i$ in the output layer/softmax layer/ $Z_2$ layer. In other words $Z_2$ layer consists of 2 elements $z_0$ and $z_1$ and the derivation will be w.r.t the neurons/elements.

I re-arrange your equations to keep things as simple as possible: combining equations 5,6,7 yields:

$L = -\ \sum_{n} \dot{y}_n$ x $ log(y_n)$

  • Here is a brief derivation

$$\frac{\partial L}{\partial z_i}=-\sum_n\dot{y}_n\frac{\partial \log y_n}{\partial z_i}=-\sum_n\dot{y}_n\frac{1}{y_n}\frac{\partial y_n}{\partial z_i}$$

The softmax function $y_n$ is defined as $$y_n = \frac{e^{z_n}}{\sum_ke^{z_k}}$$

when n=i $$\frac{\partial y_n}{\partial z_i} = \color{blue}{y_i(1-y_i)}$$ when n <> i $$ \frac{\partial y_n}{\partial z_i} = \color{red}{-y_ny_i}$$

now let's substitute $\frac{\partial y_n}{\partial z_i}$

$$\frac{\partial L}{\partial z_i}=-\dot{y}_i\frac{1}{y_i}\color{blue}{y_i(1-y_i)}-\sum_{n\neq i}\dot{y}_n\frac{1}{y_n}\color{red}{({{-y_ny_i}})}$$ $$ \\=-\dot{y}_i(1-y_i)+\sum_{n\neq i}\dot{y}_n({{y_i}})\\=-\dot{y}_i+{\dot{y}_iy_i+\sum_{n\neq i}\dot{y}_n({y_i})} \\={y_i\left(\sum_n\dot{y}_n\right)}-\dot{y}_i=y_i-\dot{y}_i$$ where $\sum_n\dot{y}_n = 1$

  • $\begingroup$ hi @ahmed, can you elaborate a bit more how you went from the RHS term in the first line to the second line, I can see it has something to do with same indices and different indices. Thanks $\endgroup$
    – nz_
    Commented Jun 12, 2018 at 12:53
  • $\begingroup$ Also, I think the answer should not be dependent on the index of $z_2$, because $2$ here is arbitrary. i.e. $\frac{\partial L }{\partial z_2} \neq y_2 - \dot{y}_2$ $\endgroup$
    – nz_
    Commented Jun 12, 2018 at 13:07
  • $\begingroup$ even though $z_2$ is $1 \times 2$, it looks like $\frac{\partial L }{\partial z_2}$ is actually $2 \times 2$... because $\frac{\partial \log{y_n}}{\partial z_2}$ is a derivative of a two dimensional vector with respect to a two dimensional vector.. Oh wait, I may be forgetting about the multiplication by $\dot{y}_n$, which can actually turn it back to be $1 \times 2$ $\endgroup$
    – nz_
    Commented Jun 12, 2018 at 13:15
  • $\begingroup$ I will have a look at your link. $\endgroup$
    – nz_
    Commented Jun 12, 2018 at 13:22
  • 1
    $\begingroup$ Thanks :) Just one thing that I would add here probably. Your $n$ is the class value, whereas in my notation, $n$ represents a training sample. This is very useful. Thanks a bunch again! $\endgroup$
    – nz_
    Commented Jun 12, 2018 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.