Backpropagation - softmax derivative

Question

I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).

Suppose that the network is simple like so (forward pass):

$$\begin{aligned}z_1 &= xW_1 + b_1 & (1) \\ a_1 &= \tanh{(z_1)} & (2) \\ z_2 & = a_1W_2 + b_2 & (3)\\ a_2 & = \text{softmax}(z_2)=\hat{y}_n & (4) \\ l_n & = \log{(\hat{y}_n)} & (5) \\ PL_n & = l_n\dot{y}_n^T & (6) \\ L & = -\frac{1}{N} \sum_{n \in N} PL_n & (7) \end{aligned} $$

So only one hidden layer. Let the dimensions be the following:

$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.

Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):

$$\begin{aligned} dPL_n & = -1/N & (7) \\ d\dot{y}^T_n & = l_n^T dPL_n & (6) \\ dl_n & = \dot{y}^T_n dPL_n & (6) \\ d\hat{y}_n & = 1/\hat{y}_n & (5) \end{aligned} $$

Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?

(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).

Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(

I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.

I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2 = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &= \frac{\partial l_n}{\partial z_2} dl_n \\ &= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$

Also, same question on Maths.SE.

Answer 1

The partial derivative of the cost function with respect to the neuron $i$ ,in the layer $Z_2$ defined in your questions above, is $\frac{\partial L}{\partial z_i}$, where $L$ is the cost function.

please note that the derivation will be on the neuron level not at the layer level, where $z_i$ is the neuron number $i$ in the output layer/softmax layer/ $Z_2$ layer. In other words $Z_2$ layer consists of 2 elements $z_0$ and $z_1$ and the derivation will be w.r.t the neurons/elements.

I re-arrange your equations to keep things as simple as possible: combining equations 5,6,7 yields:

$L = -\ \sum_{n} \dot{y}_n$ x $ log(y_n)$

• Here is a brief derivation

$$\frac{\partial L}{\partial z_i}=-\sum_n\dot{y}_n\frac{\partial \log y_n}{\partial z_i}=-\sum_n\dot{y}_n\frac{1}{y_n}\frac{\partial y_n}{\partial z_i}$$

The softmax function $y_n$ is defined as $$y_n = \frac{e^{z_n}}{\sum_ke^{z_k}}$$

when n=i $$\frac{\partial y_n}{\partial z_i} = \color{blue}{y_i(1-y_i)}$$ when n <> i $$ \frac{\partial y_n}{\partial z_i} = \color{red}{-y_ny_i}$$

now let's substitute $\frac{\partial y_n}{\partial z_i}$

$$\frac{\partial L}{\partial z_i}=-\dot{y}_i\frac{1}{y_i}\color{blue}{y_i(1-y_i)}-\sum_{n\neq i}\dot{y}_n\frac{1}{y_n}\color{red}{({{-y_ny_i}})}$$ $$ \\=-\dot{y}_i(1-y_i)+\sum_{n\neq i}\dot{y}_n({{y_i}})\\=-\dot{y}_i+{\dot{y}_iy_i+\sum_{n\neq i}\dot{y}_n({y_i})} \\={y_i\left(\sum_n\dot{y}_n\right)}-\dot{y}_i=y_i-\dot{y}_i$$ where $\sum_n\dot{y}_n = 1$

• For detailed derivation and explanations please see https://deepnotes.io/softmax-crossentropy