I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).
Suppose that the network is simple like so (forward pass):
$$\begin{aligned}z_1 &= xW_1 + b_1 & (1) \\ a_1 &= \tanh{(z_1)} & (2) \\ z_2 & = a_1W_2 + b_2 & (3)\\ a_2 & = \text{softmax}(z_2)=\hat{y}_n & (4) \\ l_n & = \log{(\hat{y}_n)} & (5) \\ PL_n & = l_n\dot{y}_n^T & (6) \\ L & = -\frac{1}{N} \sum_{n \in N} PL_n & (7) \end{aligned} $$
So only one hidden layer. Let the dimensions be the following:
$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.
Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):
$$\begin{aligned} dPL_n & = -1/N & (7) \\ d\dot{y}^T_n & = l_n^T dPL_n & (6) \\ dl_n & = \dot{y}^T_n dPL_n & (6) \\ d\hat{y}_n & = 1/\hat{y}_n & (5) \end{aligned} $$
Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?
(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).
Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(
I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.
I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2 = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &= \frac{\partial l_n}{\partial z_2} dl_n \\ &= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$
Also, same question on Maths.SE.