3
$\begingroup$

I have a question about SVM that some of you may help me with…

I know that y(xi), by convention, would be -1 or 1 depending on which class the Xi belongs to.

But I don't fully understand why it's stablished that the hyperplane equation should be:

w·xi + b >= 1 or w·xi + b <= -1

Where do those "1" and "-1" come from?

Shouldn't it be that, for any point, depending on its classification, the hyperplane equation would be like this?

w·xi + b >= yi * margin/2 (yi = -1 or 1)

There is no 1 or -1 anywhere…

Thanks in advance

$\endgroup$
1
$\begingroup$

The -1 and +1 you see in many SVM proofs is due to a scaling factor that is applied to the distance between the margin and the 2 closest points. This is useful for simplifying the math, but most textbooks gloss over this super important detail which makes it complicated.

This is all based on the idea that given a dataset you can scale it, and your decision boundary however you want without affecting its performance. For example, if you have dataset with units in the millions of dollars and decision boundary also in the millions of dollars, you can scale these all to the dollar scale to get the same performance.


The math

We agree that for an arbitrary dataset the SVM algorithm wants to maximize the distance of the boundary $\gamma$. The decision boundary is described by a linear line as described in our first constraint. The second constraint ensures that the functional margin is equal to the geometric margin, and thus we can employ the scale invariability property of the geometric margin. This optimization is described as

$max_{\gamma, w, b} \gamma$

$\text{s.t. } y^{(i)}(w^Tx^{(i)} + b) \geq \gamma, i = 1, ..., m \text{ and } ||w|| = 1$.

We then want to find the maximum possible margin between the positive and negative classes.

We can reformulate this problem such that we get rid of the $||w|| = 1$ constraint to get

$max_{\hat{\gamma}, w, b} \frac{\hat{\gamma}}{||w||}$

$\text{s.t. } y^{(i)}(w^Tx^{(i)} + b) \geq \hat{\gamma}, i = 1, ..., m$

where

$\gamma = \frac{\hat{\gamma}}{||w||}$

Now all we will do is we will force the margin of the separation to be 1 in order to simplify our math. In other words we will want $\hat{\gamma} = 1$. So we now have

$max_{\gamma, w, b} \frac{1}{||w||}$

$\text{s.t. } y^{(i)}(w^Tx^{(i)} + b) \geq 1, i = 1, ..., m$

Finally, maximizing $1\||w||$ is equivalent to minimizing $||w||$, which is equivalent to minimizing $||w||^2/2$ and thus the usual quadratic optimization SVM equation is born

$min_{\gamma, w, b} \frac{1}{2}||w||^2$

$\text{s.t. } y^{(i)}(w^Tx^{(i)} + b) \geq 1, i = 1, ..., m$

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.