0
$\begingroup$

Trying to understand SVM. I am sure that my reasoning can't be correct but I can't see what is wrong.

We know that two points determine a line. I am trying to understand support vector machines using this kind of analysis or understanding. Of course support vector machines involve data and I'm going to be thinking of it as data in the plane. So I'm going to be thinking of it as having two variables that are numeric and 1 output variable which is binary. Typically, the decision boundary in its vector form is given as w dot x +b =0, Where both W and X are vectors. As I understand it this is an equation that initially is not tied down to any particular data points. However, the next step involves considering the support vectors which are both points. Now as I said at the beginning typically two points determine a line. But the two support vectors in and of themselves don't determine the decision boundary per se. This can be seen since there are many lines that can pass through the midpoint of the sector joining the two support vectors. So, the two support vectors don't determine a unique decision boundary. However, I think it is also true that the decision boundary needs to be perpendicular to the line segment that joins the two support vectors. First of all, is this true? If it is true, can we say that the two support vectors are enough information to nail down the decision boundary? If that is true, then it seems also true that the so-called gutters will also be nailed down. Because they have to go through the support vectors, and they have to be parallel to the already nailed down decision boundary. Do you agree with this?

Note that my analysis does not talk about maximizing the margin. I simply want to know if having the two support vectors and the fact that the decision boundary has to be perpendicular to the line segment which joins the two support vectors is enough information to determine the equation of the decision boundary.

Next, I am trying to do the same kind of analysis when thinking about the equations for the gutters. I think this kind of analysis might be referred to as a sufficiency argument, but I am not sure. In any case, for the decision boundary, we said the equation is w dot x +b =0. The equations for the gutters are similar with the difference only on the right-hand side replacing the 0 with either plus or negative one. Now obviously these gutter lines have a different equation than the decision boundary. But of course, they are parallel to the decision boundary. I'm pretty sure that since they are parallel that in terms of their vector equations, they will both contain the same W that is in the equation or the decision boundary. I guess we can conclude that because we know that W, as a vector, is perpendicular to the decision boundary, and the same should be true or the gutter lines as well. So the only thing that can be different is the y-intercept. The only thing I'm not quite clear about is why we can freely set the right-hand sides of these gutter equations two plus and negative 1? We know that they have to pass through the support vectors and it seems like therefore we don't have the freedom to set these values two plus and -1. It seems like in a kind of sufficiency argument that they will be already determined and that we don't have the freedom to set them to these two values. My question isn't about why they are equal to + and - one, but rather why is it possible to set them at all. It seems they must be pre-determined already by the fact that they have to pass through the support vectors.

$\endgroup$

1 Answer 1

1
$\begingroup$

It is true that for two opposing class points there could be an infinitely linear separators between them. Here comes the next requirement, the margin of the separator we want has to be maximized. When the margin is maximal there is an unique separator, the one perpendicular on the line which passes through the middle of the segment between the two points.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.