1
$\begingroup$

I was going through Stanford CS 224 lecture notes on Back propagation.

Page 5 states:

We can see from the max-margin loss that:

∂J /∂s = − ∂J/∂s(c) = −1

I'm not sure I understand why this is the case.

My thought process :

If we're maximizing loss, the derivative of the cost (J) must go to 0.

Since s and s(c) aren't necessarily independent, I'm struggling to understand why / how the above equation is correct, since using the chain rule, the result should depend on ∂s(c) /∂s.

What am I missing here ?

$\endgroup$

1 Answer 1

2
$\begingroup$

Normally, you’d be correct in that we typically try to set the derivative of cost to 0 for gradient descent to find the minimum, but here we are talking about max-margin loss (typically used in SVMs, but here in the context of neural networks). Max margin means that we want to create the biggest margin between our classes. In the notation used on page 4, they are talking about minimizing J, and the introduction of the margin:

minimize J = max(sc − s, 0) 

“However, the above optimization objective is risky in the sense that it does not attempt to create a margin of safety. We would want the "true" labeled data point to score higher than the "false" labeled data point by some positive margin ∆. In other words, we would want error to be calculated if (s − sc < ∆) and not just when (s − sc < 0). Thus, we modify the optimization objective:

minimize J = max(∆ + sc − s, 0)

We can scale this margin such that it is ∆ = 1”

So we end up with the cost function:

J  = 1 + s(c) −s

Then taking the partial derivatives of J wrt s and sc gives ∂J /∂s = − ∂J/∂s(c) = −1

$\endgroup$
4
  • $\begingroup$ Thank you so much for your answer! Are s(c) and s necessarily independent variables ? I'm not sure I understand why that should be the case. Thanks again! $\endgroup$
    – Hormigas
    Feb 8 at 15:55
  • $\begingroup$ Yes, s(c) and s are the outputs of the network to two different inputs, x(c) and c. sc = UTf(Wxc + b) and s = UTf(Wx + b) from mid-page 4. I take the x(c) to mean a corrupted version of input x, so not entirely unrelated, but the two outputs of the network are independent for purposes of computing the cost gradient. $\endgroup$ Feb 8 at 17:23
  • $\begingroup$ "the two outputs of the network are independent for purposes of computing the cost gradient." -> I'm not sure I understand this well. Is this because the two inputs x, and x(c) aren't necessarily related ? And that means that ds and ds(c) aren't related ? which would mean that ds/ds(c) = 0. Is that correct ? $\endgroup$
    – Hormigas
    Feb 9 at 0:47
  • $\begingroup$ The inputs to the network are x, and xc (x corrupted). The outputs are s, and sc. We take the partial derivatives of the cost function J with respect to the outputs in order to compute the gradients to back propagate. Taking the partial derivative of one output wrt another is meaningless. In other words ds/ds(c) is not useful. hth. $\endgroup$ Feb 9 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.