1
$\begingroup$

$$ \lambda||\hat w||^2 +(1/n)\sum max(0,1-y_i(\hat w \hat x_i -b)) $$

we know that $2/||\hat w||$ is the width of the margin.

The second term penalizes a misclassified point for how far away it is from the margin relative to the width of the margin. E.g., suppose there is a misclassified point $x_0$: $$ 1-y_0(\hat w \hat x_0 -b)=3 $$ That means $x_0$ is $3/||\hat w||$ away from $1-y_i(\hat w \hat x_i -b)=0$ and is penalized for $3$.

The first term penalizes for the inverse of the width of the margin squared. I find it hard to reconcile with the second term - they seem to be of different scales. Is there any reason (intuitively) why $||\hat w||^2$ is used instead of just $||\hat w||$?

PS: Perhaps one reason is that $||\hat w||^2 $ is easier computation-wise (quadratic programming)? Or perhaps norm squared assumes sample noise to be Gaussian? I am not sure. Has anyone seen the use of $||\hat w||$ instead of $||\hat w||^2$?

$\endgroup$
1
$\begingroup$

It is indeed for computational tractability. You would not lose the all-important sparsity, since that is provided by the second term. The original formulation of SVM, which you can find on Wikipedia actually uses $\lVert w \rVert$:

Minimize $\|{\vec {w}}\|$ subject to ${\displaystyle y_{i}({\vec {w}}\cdot {\vec {x}}_{i}-b)\geq 1}-\xi_i$

Obviously, minimizing $\lVert w \rVert$ is the same as $\lVert w \rVert^2$. A quantative difference arises when introducing the constraint using the Lagrangian function, but qualitatively it is the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.