I don't understand the difference between the parameter $C$ and $\lambda$ in terms of the SVM. It seems to me that they are both involved in regulating over-fitting of the data.
What difference between $C$ and $\lambda$?
$\begingroup$
$\endgroup$
asked Aug 19, 2017 at 15:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You are right to be confused. What's going on is that the hyperparameters refer to different formulations. On the one hand there is a generic empirical risk minimizer
$$\lambda\lVert \mathbf w \rVert^2 + \frac 1 n \sum_{i=1}^n \max\left(0, 1 - y_i(\mathbf x_i^T \mathbf w + w_0)\right)$$
On the other there is the soft-margin objective
$$\frac{1}{2} \lVert \mathbf w \rVert^2 + C \sum_{i=1}^n \xi_i$$
such that $$\xi_i \geq 0, y_i\left(\mathbf x_i^T \mathbf w + w_0\right) \geq 1-\xi_i, \forall i$$
The first one places SVM in the framework of empirical risk minimization, in which the objective is an expected loss plus the $L_p$-norm of the main parameter. The purpose of introducing slack variables and constraints in the second formulation is to allow a fraction of points (adjustable by $C$) to lie on the wrong side of the margin, and eliminate the non-differentiable $\max$ function. Moreover, it draws attention to the fact that SVMs encode sparsity in the loss function, rather than the prior. This is called the primal form and can be solved using quadratic programming. Wikipedia currently explains this in the Computing the classifier section. You should be able to see that $C=1/{2n\lambda}$. Some texts omit the $1/2$ or the $n$.
