So I've drawn a neural network diagram below:

enter image description here

where $x_1, x_2,\ldots,x_m$ are the input layer, $h_1, h_2$ are the hidden layer and $\hat y_1, \hat y_2,\ldots \hat y_k$ are the output layer. In the $W^h_{im}$ notation, it represents the weight, where $i$ is representing to which node it's pointing to in the hidden layer, which is either $h_1$ or $h_2$ and $m$ is representing from which input node.

For example, if $W^h_{11}$, this means it is the weight represented by the line connecting $x_1$ node to $h_1$ node. $W^o_{ki}$, where $k$ is the output node and $i$ is the hidden layer node. Therefore, $W^o_{11}$ is the arrow connecting $h_1$ node with $\hat y_1$ node.

I'm currently working on the back propagation process of updating the gradient descent equation for $W^h_{11}$. Before this, I've only learnt to deal with a diagram with only one output node, but now it's multiple output nodes instead.

So I've attempted the beginning part of the calculus, where I'm not entirely sure if $\hat y$ equation i wrote below is correct and also for the loss function?

For $W^h_{11}$:

$Input: (x,y)$

$\hat{y} = forward(x)$

$\hat{y} = \sum ^K_{k=1} h_1W^o_{k1} + \sum ^K_{k=1} h_2W^o_{k2}$ *am i doing this correctly?

$h_1 = \sigma(\bar{h}_1)$ where $\bar{h}_1 = \sum_{j=1}^{M} x_jW^h_{1j}$

$Loss function: J_t(w) = \frac{1}{2}\sum(\hat{y}-y)^2$ where $\hat y$ is a vector of $\hat y_1,\hat y_2,..\hat y_k$ *would the loss function be included with a summation?

Would like to know if I've done the beginning part right.


We have $$\hat{y}_{\color{blue}k}=h_1W_{k1}^o + h_2W_{k2}^o$$

If we let $\hat{y} = (\hat{y}_1, \ldots, \hat{y}_K)^T$, $W_1^o=(W_{11}, \ldots, W_{K1})^T$, and $W_2^o=(W_{12}, \ldots, W_{K2})^T$

Then we have $$\hat{y}=h_1W_1^o+h_2W_2^o$$

I believe you are performing a regression, $$J(w) = \frac12 \|\hat{y}-y\|^2=\frac12\sum_{k=1}^K(\hat{y}_k-y_k)^2$$

It is possible to weight individual term as well depending on applications.

  • $\begingroup$ if I were to differentiate $\frac {\partial J}{\partial \hat y}$ would it be $(\hat y_k - y_k)$ in this case? $\endgroup$
    – Maxxx
    Mar 31 '19 at 4:37
  • $\begingroup$ If $J=\frac12 \| \hat{y}-y\|^2$, then $\frac{\partial{J}}{\partial{\hat{y}}}= \hat{y}-y$, it is a vector. $\endgroup$ Mar 31 '19 at 4:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.