A math question about solving the Lagrangian of Support Vector Machine

Question

$$\mathcal{L}(w,b,\xi,\alpha,r) = \frac12w^Tw+C\sum_{i=1}^m \xi_i-\sum_{i=1}^m \alpha_i[y^{(i)}(x^Tw+b)-1+\xi_i]-\sum_{i=1}^mr_i\xi_i$$

Here, the $\alpha_i$'s and $r_i$'s are our Lagrange multipliers (constrained to be $\ge 0$)

To maximize the Lagrangian of soft margin SVM (see the formula above), we set the derivatives with respect to $w$, $\xi$ and $b$ to $0$ respectively.

But what if we set the derivatives w.r.t $r$ to zero first? Wouldn't that result in $\xi$ being all $0$s? Meaning that the optimal solution is reached only when all the relaxing terms $\xi$ are $0$? But that doesn't seem right, does it?

Answer 1

Since you have an inequality constraint you need to meet the necessary Kuhn-Tucker Conditions which for non-negativity constraints are:

• $\xi \geq 0$
• $-r \cdot \xi = 0$

It is not enough to set the derivative w.r.t. to $r$ to zero as is the case for equality constraints and Lagrange multipliers.

Therefore, there are two cases to distinguish:

• Either the condition is tight ($\xi=0$) and you can have $r \neq 0$
• Or $\xi > 0$ and $r = 0$.