I don't know how the equation below goes from line 2 to 3 after the derivative term is moved inside the brackets. Specifically, how is it calculating the derivative of log(y_hat)? Also, if anyone can point to a good textbook or website to learn this stuff. I've just started on this free course (option to pay for extras) at edx that is so far pretty good because it has easy-to-understand lectures and jupyter notebook assignments that get you coding: https://courses.edx.org/courses/course-v1:Microsoft+DAT256x+1T2019/course/

from Udacity deep learning course


Based on chain rule we have:

$$\frac{\partial \mbox{log}(f(x))}{\partial x}=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=f(x)}\frac{\partial f(x)}{\partial x}=\frac{1}{z}\Bigr|_{z=f(x)}\frac{\partial f(x)}{\partial x}=\frac{1}{f(x)}\frac{\partial f(x)}{\partial x}$$

More close to the notations used in the question:

$$\frac{\partial \mbox{log}(f_{\mathbf{w}}(\mathbf{x}))}{\partial w_j}=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}=\frac{1}{z}\Bigr|_{z=f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}=\frac{1}{f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}$$

Now by setting $f_{\mathbf{w}}(\mathbf{x})=\hat{y}$, or $f_\mathbf{w}(\mathbf{x})=1-\hat{y}$, line 2 to 3 follows. For example for $f_\mathbf{w}(\mathbf{x})=1-\hat{y}$:

$$\frac{\partial \mbox{log}(1-\hat{y})}{\partial w_j} =\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=1-\hat{y}} \frac{\partial (1-\hat{y})}{\partial w_j}=\frac{1}{z}\Bigr|_{z=1-\hat{y}}\frac{\partial (1-\hat{y})}{\partial w_j}=\frac{1}{1-\hat{y}}\frac{\partial (1-\hat{y})}{\partial w_j}$$

Note that input $\mathbf{x}$ to the network is constant w.r.t. changes in $\mathbf{w}$. Also $x|_{x=y}$ means replace $x$ with $y$.

| improve this answer | |
  • $\begingroup$ Of the three given, I think only the last one (the very right one) is applicable in the posted question. Is that correct? $\endgroup$ – mLstudent33 Mar 16 '19 at 15:19
  • $\begingroup$ Actually, I cannot seem to be able to chain at all from left equation to middle to right. That is what I'm supposed to do right? I learned chain rule when studying RNNs and LSTMs but this seems different. Can you link somewhere with all the different chain rules? $\endgroup$ – mLstudent33 Mar 16 '19 at 15:27
  • 1
    $\begingroup$ @flexitarian33 chain rule is only one rule. Only notations differ slightly. Here is a good link for practice tutorial.math.lamar.edu/Classes/CalcI/ChainRule.aspx $\endgroup$ – Esmailian Mar 16 '19 at 15:30
  • 1
    $\begingroup$ I see. I'm going to try reading this other book manning.com/books/grokking-deep-learning because even with the edited answer, I still don't see how that first equality holds. $\endgroup$ – mLstudent33 Mar 16 '19 at 16:04

For those like myself who are not as mathematically astute, that Grokking book is very helpful. It explains how multiplying the raw error (y_hat - y) by the input x has 3 effects:

  1. stopping, ie. when the input is 0, prediction is 0 so there is no change in weight

  2. negative reversal, ie. when the input is negative, the weight change will be negative so that it moves in the correct direction (adjusts so that next y_hat is closer to y)

  3. scaling so when input x is large, the change in weight will be large but this is where alpha comes in to avoid overshooting.

Anyways, as it relates to my original posting, without the chain rule, I can find the partial derivative of y_hat w.r.t. the w_j which is equal to partial derivative of (w_j *x) w.r.t. w_j. Basically it is just the input x.

So Grokking has a variable called weight_delta (the gradient of error function) which is raw error * input, ie. (y_hat - y) * x. The input x happens to be the derivative of (w_1x_1 + w_2x_2 +...+w_nx_n) and since all other w_jx_j are constants, we get x_j left as the derivative. In a one input model, that is just x. As stated earlier we multiply the raw error by input for stopping, negative reversal and scaling.

So we end up with the last line of the equation, ie. (y_hat - y) * x = -(y - y_hat) * x

Udacity concludes the section with this:

If you think about it, this is fascinating. The gradient is actually a scalar times the coordinates of the point! And what is the scalar? Nothing less than a multiple of the difference between the label and the prediction.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.