2
$\begingroup$

enter image description here

I have the question which is mentioned in the above picture. It is trying to find the derivative of f with respect to weight matrix W1. Can any one help me how I can accomplish that?

I solved in this way :

enter image description here enter image description here

$\endgroup$
  • $\begingroup$ Do you mind telling me what book that is from @DukeLover? $\endgroup$ – Bucephalus Jan 21 '18 at 13:32
  • $\begingroup$ @DukeLover you should employ chain rule for derivative. $\endgroup$ – Media Jan 21 '18 at 15:48
  • 1
    $\begingroup$ @Bucephalus Andrew Ng's Stanford classes $\endgroup$ – DukeLover Jan 23 '18 at 5:54
3
$\begingroup$

You are missing one little step here. At the output node the backpropagation algorithm starts with the loss function $L$. You want to find out how much the weight $w_1$ impacts our loss function thus

$\frac{\partial L}{\partial w_1}$.

We can break this down with the chain rule as

$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial \sigma} \frac{\partial \sigma}{\partial w_1}$.

Why do we do this?

We do this to simplify the math. Because the $L$ is a function of $f$ and $f$ is a function of $\sigma$ and $\sigma$ is a function of the weights. But if we ignore the loss function for now.

How do we calculate $\frac{\partial f}{\partial w^2_{i, j}}$?

$\frac{\partial f}{\partial w^2_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial w^2_1}$

$f = h_1^2 w_{1,1}^3 + h_2^2 w_{2,1}^3$

$\frac{\partial f}{\partial h^2_i} = w_{i, 1}^3$

$\frac{\partial h^2_i}{\partial w^2_1} = \sigma(W^2h^1) (1 - \sigma(W^2h^1))$

Backpropagation

Based on this answer we can see that $h_i^2$ is a function of the inputs from the previous layer and the activation at the neuron. Thus, we can substitute this function into our equation and continue taking the derivatives and we can see how the error propagates backwards through the network.

$\frac{\partial f}{\partial w^1_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial h^1_i} \frac{\partial h^1_i}{\partial w^1_{i,j}}$


(Update)

The answer

This may not be correct, the work needs to be checked.

$f = \begin{bmatrix} w_{11}^3 & w_{21}^3 \end{bmatrix} \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \textbf{W}^3\textbf{h}^2$

$\textbf{h}^2 = \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^2 & w_{21}^2 \\ w_{21}^2 & w_{22}^2\end{bmatrix} \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix}) = \sigma(\textbf{W}^2\textbf{h}^1)$

$\textbf{h}^1 = \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^1 & w_{21}^1 \\ w_{21}^1 & w_{22}^1\end{bmatrix} \begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix}) = \sigma(\textbf{W}^1\textbf{X})$

Then using the chain rule

$\frac{\partial f}{\partial \textbf{W}^1} = \frac{\partial f}{\partial \textbf{h}^2} \frac{\partial \textbf{h}^2}{\partial \textbf{h}^1} \frac{\partial \textbf{h}^1}{\partial \textbf{W}^1} = \textbf{W}^3 * \sigma(\textbf{W}^2\textbf{h}^1)(1-\sigma(\textbf{W}^2\textbf{h}^1)) * \sigma(\textbf{W}^1\textbf{X})(1-\sigma(\textbf{W}^1\textbf{X}))$

$\endgroup$
  • $\begingroup$ hi @JahKnows I have updated the question , with the approach i had taken. Can you please check the approach once ? If I am missing on something. $\endgroup$ – DukeLover Jan 23 '18 at 5:52
  • $\begingroup$ I'm a bit confused with what you have written. I will append my answer with the full answer as I have computed it. However, it may not be entirely correct. $\endgroup$ – JahKnows Jan 23 '18 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.