3
$\begingroup$

I read on this Wikipedia page the following about soft-margin SVM:

loss

"The parameter $λ$ determines the trade-off between increasing the margin size and ensuring that the $x_i$ lie on the correct side of the margin. Thus, for sufficiently small values of $λ$, the second term in the loss function will become negligible, hence, it will behave similar to the hard-margin SVM, if the input data are linearly classifiable, but will still learn if a classification rule is viable or not."

I can't understand why in the case that λ=0 the algorithm will behave like hard-margin SVM. If λ=0, it seems to me that the algorithm won't have any reason to perform any optimization on the margin. Doesn't it just become a perceptron in that case, since the algorithm only "cares" about classifying all the train data correctly, while not reaching any optimal solution regarding the margin?

I'll appreciate a clarification about the issue, please.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

First, the remark from the wikipedia article deals with small (positive) values of $\lambda$, not $\lambda=0$. Indeed, if $\lambda=0$, then every separating hyperplane achieves the minimum score of 0.

If the data is linearly separable, then taking $\lambda$ small enough that the first term dominates ensures that minimizing the loss will require taking a separating hyperplane to zero out the first term, and subject to that minimizing the second term is equivalent to the original hard SVM.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Thanks! Can you please clarify what you meant by "and subject to that minimizing the second term is equivalent to the original hard SVM."? if λ is small, why will the algorithm aim to minimize the second term if it's pretty negligible in comparison to the first term? You wrote "subject to", and that's what confuses me. $\endgroup$
    – Ben
    Jun 24, 2019 at 8:15
  • $\begingroup$ Indeed, I was a little bit wrong there: I was thinking of the first term as more discrete than it is, and thinking that way lead me to treating the term as either zero or large, so that minimizing the loss would necessarily zero out the first term entirely. Then, among perfect separators (those achieving zero in the first term), we still want to minimize the relatively less impactful second term. $\endgroup$
    – Ben Reiniger ♦
    Jun 24, 2019 at 14:33
  • $\begingroup$ I think the reality is still close enough though. If the hyperplane chosen misclassifies a point, the penalty is at least 1 in the first term, so small enough $\lambda$ will still avoid it. My claim "equivalent to the original hard SVM" isn't quite right though: for any positive $\lambda$, there is still a small amount to be gained by bringing the hard SVM's support vectors very slightly inside the margin. $\endgroup$
    – Ben Reiniger ♦
    Jun 24, 2019 at 14:37
  • $\begingroup$ (...Though I think then we need a more precise formulation than wikipedia's, since normalizing so that support vectors are 1 "unit" away from the hyperplane won't work anymore. See e.g. sklearn's documentation, scikit-learn.org/stable/modules/… ) $\endgroup$
    – Ben Reiniger ♦
    Jun 24, 2019 at 14:38
  • $\begingroup$ Thank you for your clarification! $\endgroup$
    – Ben
    Jun 25, 2019 at 7:33
0
$\begingroup$

The reason is that your data is in a way that the algorithm does not make any mistake on it in the current feature space. it is an easy problem that the algorithm does not need to ignore wrong points due to the easy data which is provided. If it occurs that your data is hard to be classified, the margin then will try to ignore some data points which may lead to narrow margins.

It is worth mentioning that even though it performs the same, it won't be as a simple perceptron, at least in most cases. Consider that SVM somehow considers the geometrical position of data points while a simple perceptron always tries to reduce the cost function. You can take a look at the pictures which are provided here.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the response, however I'm not sure if I understand you correcly or if you understood me. I'm talking about the specific case in which λ=0. It seems that in this case, according to the loss function I attached, SVM doesn't anymore consider the geometrical position or aims to maximize the margin, just classify the data points correctly like perceptron. Am I wrong? $\endgroup$
    – Ben
    Jun 22, 2019 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.