1
$\begingroup$

I am following this note on SVM.

The constraint, $y_i(wx_i+b) \geq 1$, basically said all inputs, $x_i$, lie at least 1 unit away from the hyperplane on the correct side.

What does it mean by 1 unit? In 2D space, does it mean moving the hyperplane, $w$, up by 1?

$\endgroup$
1
$\begingroup$

Main formula for SVM is - $y_i(wx_i +b) \geq d$

In the derivation process, it is changed to 1 to make it standardized for all hyper-plane.
If it has to be described, it will be -
"Greater than" "per unit of minimum margin distance"

Let's suppose,
If a hyper-plane has the minimum margin point at 4 Eucledien distance
Another one has it at 4.5 Eucledien distance

So, this $y_i(wx_i +b) \geq$ 1 means,
1 unit of "every 4 units" for first hyper-plane and
1 unit of "every 4.5 units" for the other hyper-plane

What it meant -
This is more for Mathematical convenience.
Another neatness it added, the maximizing equation changes to 1/$w$ from F/$w$.
F is the distance of the point which is nearest to the plane.

Why it will not affect point position
A plane i.e. $(wx_i +b)$ will not change if we rescale $w$ and $b$. So we rescaled it in such a way such that F becomes 1.
This "1" will be different for different Hyperplanes depending on it's $w$.

Added this screen from Support Vector Machines Succinctly.
Please read it if you want a very detailed start to end explanation of SVM with python code


enter image description here


Good references for SVM
Alexandre Kowalczyk
Shuzhanfan
Professor Yaser Abu-Mostafa

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have specifically looked at Shuzhanfan. "If we rescale w and b, we are still maximizing M and the optimization result will not change. Let’s rescale w and b and make F=1". Would rescalling move some points which originally are outside the gutters to inside? $\endgroup$ – JOHN Jun 1 at 7:08
  • $\begingroup$ No, it will not. Points distance would shift relative to the nearest point but will not flip i.e. 4 might become 1.75 then 8 will also become 3.75. Edited the answer and added a very great read (though no one ever kept this in his must-have list). $\endgroup$ – 10xAI Jun 2 at 12:13
2
$\begingroup$

It is a unit of distance, I would usually assume euclidean distance.

In more detail:

The data point $x_i$ is projected onto the vector $w$, which defines the orientation of the discriminating linear hyperplane as it is orthogonal to $w$. Where the discriminating hyperplane is "fixed" along the orientation of $w$ is decided by the bias Term $b$.

So for classifying a positive data point $x_{+}$ we would want $wx_{+} + b$ to a positive number.

For a negative $x_{-}$ datapoint $wx_{-}+b$ should be negative.

We can combine these two disired outcomes into a single constraint by using the labels, so we want $y_{+}(wx_{+}+b) \geq 0$ and $y_{-}(wx_{-}+b) \geq 0$.

By replacing the $0$ with a positive value $d \geq 0$, we can alter the constraint to encode our desire that the points should be $d$ far away from from the discriminating hyperplane, giving the constraint is $y_i(wx_i +b) \geq d$.

What kind of unit of distance depends on the vector space. Usually orthogonal projections in euclidean vector spaces are used, in that case it would be euclidean distance.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Can I set d to number other than 1? $\endgroup$ – JOHN May 31 at 9:51
  • 1
    $\begingroup$ Sure. It's a hyperparameter and can be set according to ML problem you want to solve. This is the simple formulation of SVM. Usually the data is not so easily seperable and then one works with slack variables or rather the hinge loss to allow for some points to lie unavoidably within the margin (closer than the desired distance to the decision boundary). $\endgroup$ – bonfab May 31 at 20:37
  • $\begingroup$ So choosing 1 is just for mathematical convenience? $\endgroup$ – JOHN Jun 1 at 6:58
  • $\begingroup$ Yes it helps to exemplify the purpose of SVM. In the end the distance is a hyperparameter that you can change to fit your problem accordingly. The central idea of SVMs is that a larger margin around the decision boundary helps to generalize better. Imagine you move every datapoint by some small $\epsilon > 0$ then when you have a larger margin the chances are higher that the datapoints are still classified correctly. So it adds some robustness to your model. However it is not possible to pick an arbitrarily large margin, so the best margin distance has to be found according to your data. $\endgroup$ – bonfab Jun 1 at 9:02
0
$\begingroup$

Actaully the 1 doesn't matter. It's just a random parameter. No real meaning. You just assume some positive distance.

Because the hyperplane is scale invariant, we can fix the scale of w,b anyway we want. Let's be clever about it, and choose it such that
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.