# Neural network back propagation gradient descent calculus

So I've drawn a neural network diagram below:

where $$x_1, x_2,\ldots,x_m$$ are the input layer, $$h_1, h_2$$ are the hidden layer and $$\hat y_1, \hat y_2,\ldots \hat y_k$$ are the output layer. In the $$W^h_{im}$$ notation, it represents the weight, where $$i$$ is representing to which node it's pointing to in the hidden layer, which is either $$h_1$$ or $$h_2$$ and $$m$$ is representing from which input node.

For example, if $$W^h_{11}$$, this means it is the weight represented by the line connecting $$x_1$$ node to $$h_1$$ node. $$W^o_{ki}$$, where $$k$$ is the output node and $$i$$ is the hidden layer node. Therefore, $$W^o_{11}$$ is the arrow connecting $$h_1$$ node with $$\hat y_1$$ node.

I'm currently working on the back propagation process of updating the gradient descent equation for $$W^h_{11}$$. Before this, I've only learnt to deal with a diagram with only one output node, but now it's multiple output nodes instead.

So I've attempted the beginning part of the calculus, where I'm not entirely sure if $$\hat y$$ equation i wrote below is correct and also for the loss function?

For $$W^h_{11}$$:

$$Input: (x,y)$$

$$\hat{y} = forward(x)$$

$$\hat{y} = \sum ^K_{k=1} h_1W^o_{k1} + \sum ^K_{k=1} h_2W^o_{k2}$$ *am i doing this correctly?

$$h_1 = \sigma(\bar{h}_1)$$ where $$\bar{h}_1 = \sum_{j=1}^{M} x_jW^h_{1j}$$

$$Loss function: J_t(w) = \frac{1}{2}\sum(\hat{y}-y)^2$$ where $$\hat y$$ is a vector of $$\hat y_1,\hat y_2,..\hat y_k$$ *would the loss function be included with a summation?

Would like to know if I've done the beginning part right.

We have $$\hat{y}_{\color{blue}k}=h_1W_{k1}^o + h_2W_{k2}^o$$
If we let $$\hat{y} = (\hat{y}_1, \ldots, \hat{y}_K)^T$$, $$W_1^o=(W_{11}, \ldots, W_{K1})^T$$, and $$W_2^o=(W_{12}, \ldots, W_{K2})^T$$
Then we have $$\hat{y}=h_1W_1^o+h_2W_2^o$$
I believe you are performing a regression, $$J(w) = \frac12 \|\hat{y}-y\|^2=\frac12\sum_{k=1}^K(\hat{y}_k-y_k)^2$$
• if I were to differentiate $\frac {\partial J}{\partial \hat y}$ would it be $(\hat y_k - y_k)$ in this case? – Maxxx Mar 31 '19 at 4:37
• If $J=\frac12 \| \hat{y}-y\|^2$, then $\frac{\partial{J}}{\partial{\hat{y}}}= \hat{y}-y$, it is a vector. – Siong Thye Goh Mar 31 '19 at 4:41