Normally, you’d be correct in that we typically try to set the derivative of cost to 0 for gradient descent to find the minimum, but here we are talking about max-margin loss (typically used in SVMs, but here in the context of neural networks). Max margin means that we want to create the biggest margin between our classes. In the notation used on page 4, they are talking about minimizing J, and the introduction of the margin:
minimize J = max(sc − s, 0)
“However, the above optimization objective is risky in the sense that it does not attempt to create a margin of safety. We would want the "true" labeled data point to score higher than the "false" labeled data point by some positive margin ∆. In other words, we would want error to be calculated if (s − sc < ∆) and not just when (s − sc < 0). Thus, we modify the optimization objective:
minimize J = max(∆ + sc − s, 0)
We can scale this margin such that it is ∆ = 1”
So we end up with the cost function:
J = 1 + s(c) −s
Then taking the partial derivatives of J wrt s and sc gives
∂J /∂s = − ∂J/∂s(c) = −1
