Neural network back propagation gradient descent calculus

Question

So I've drawn a neural network diagram below:

where $x_1, x_2,\ldots,x_m$ are the input layer, $h_1, h_2$ are the hidden layer and $\hat y_1, \hat y_2,\ldots \hat y_k$ are the output layer. In the $W^h_{im}$ notation, it represents the weight, where $i$ is representing to which node it's pointing to in the hidden layer, which is either $h_1$ or $h_2$ and $m$ is representing from which input node.

For example, if $W^h_{11}$, this means it is the weight represented by the line connecting $x_1$ node to $h_1$ node. $W^o_{ki}$, where $k$ is the output node and $i$ is the hidden layer node. Therefore, $W^o_{11}$ is the arrow connecting $h_1$ node with $\hat y_1$ node.

I'm currently working on the back propagation process of updating the gradient descent equation for $W^h_{11}$. Before this, I've only learnt to deal with a diagram with only one output node, but now it's multiple output nodes instead.

So I've attempted the beginning part of the calculus, where I'm not entirely sure if $\hat y$ equation i wrote below is correct and also for the loss function?

For $W^h_{11}$:

$Input: (x,y)$

$\hat{y} = forward(x)$

$\hat{y} = \sum ^K_{k=1} h_1W^o_{k1} + \sum ^K_{k=1} h_2W^o_{k2}$ *am i doing this correctly?

$h_1 = \sigma(\bar{h}_1)$ where $\bar{h}_1 = \sum_{j=1}^{M} x_jW^h_{1j}$

$Loss function: J_t(w) = \frac{1}{2}\sum(\hat{y}-y)^2$ where $\hat y$ is a vector of $\hat y_1,\hat y_2,..\hat y_k$ *would the loss function be included with a summation?

Would like to know if I've done the beginning part right.

Answer 1

We have $$\hat{y}_{\color{blue}k}=h_1W_{k1}^o + h_2W_{k2}^o$$

If we let $\hat{y} = (\hat{y}_1, \ldots, \hat{y}_K)^T$, $W_1^o=(W_{11}, \ldots, W_{K1})^T$, and $W_2^o=(W_{12}, \ldots, W_{K2})^T$

Then we have $$\hat{y}=h_1W_1^o+h_2W_2^o$$

I believe you are performing a regression, $$J(w) = \frac12 \|\hat{y}-y\|^2=\frac12\sum_{k=1}^K(\hat{y}_k-y_k)^2$$

It is possible to weight individual term as well depending on applications.