$$\mathcal{L}(w,b,\xi,\alpha,r) = \frac12w^Tw+C\sum_{i=1}^m \xi_i-\sum_{i=1}^m \alpha_i[y^{(i)}(x^Tw+b)-1+\xi_i]-\sum_{i=1}^mr_i\xi_i$$

Here, the $\alpha_i$'s and $r_i$'s are our Lagrange multipliers (constrained to be $\ge 0$)

To maximize the Lagrangian of soft margin SVM (see the formula above), we set the derivatives with respect to $w$, $\xi$ and $b$ to $0$ respectively.

But what if we set the derivatives w.r.t $r$ to zero first? Wouldn't that result in $\xi$ being all $0$s? Meaning that the optimal solution is reached only when all the relaxing terms $\xi$ are $0$? But that doesn't seem right, does it?


Since you have an inequality constraint you need to meet the necessary Kuhn-Tucker Conditions which for non-negativity constraints are:

  • $\xi \geq 0$
  • $-r \cdot \xi = 0$

It is not enough to set the derivative w.r.t. to $r$ to zero as is the case for equality constraints and Lagrange multipliers.

Therefore, there are two cases to distinguish:

  • Either the condition is tight ($\xi=0$) and you can have $r \neq 0$
  • Or $\xi > 0$ and $r = 0$.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.