CS 224N Back Propagation and Margin Loss in Neural Networks

Question

I was going through Stanford CS 224 lecture notes on Back propagation.

Page 5 states:

We can see from the max-margin loss that:

∂J /∂s = − ∂J/∂s(c) = −1

I'm not sure I understand why this is the case.

My thought process :

If we're maximizing loss, the derivative of the cost (J) must go to 0.

Since s and s(c) aren't necessarily independent, I'm struggling to understand why / how the above equation is correct, since using the chain rule, the result should depend on ∂s(c) /∂s.

What am I missing here ?

Answer 1

Normally, you’d be correct in that we typically try to set the derivative of cost to 0 for gradient descent to find the minimum, but here we are talking about max-margin loss (typically used in SVMs, but here in the context of neural networks). Max margin means that we want to create the biggest margin between our classes. In the notation used on page 4, they are talking about minimizing J, and the introduction of the margin:

minimize J = max(sc − s, 0) 

“However, the above optimization objective is risky in the sense that it does not attempt to create a margin of safety. We would want the "true" labeled data point to score higher than the "false" labeled data point by some positive margin ∆. In other words, we would want error to be calculated if (s − sc < ∆) and not just when (s − sc < 0). Thus, we modify the optimization objective:

minimize J = max(∆ + sc − s, 0)

We can scale this margin such that it is ∆ = 1”

So we end up with the cost function:

J  = 1 + s(c) −s

Then taking the partial derivatives of J wrt s and sc gives ∂J /∂s = − ∂J/∂s(c) = −1