2
$\begingroup$

In every explanation of SVMs, we're shown how training finds a hyperplane that best separates the data. Presumably then for inference, you just check which side of the plane a point is on.

However, all the "disadvantages of SVMs" posts [1, 2] lament that SVM models are large and slow because they end up storing most of the data as support vectors.

Why would SVMs store any of the data, rather than just the (coefficients of the) separating hyperplane? (And what is a "support vector" in the soft-margin case, when points of both classes are scattered on both sides of the hyperplane, anyway?)

$\endgroup$

2 Answers 2

3
$\begingroup$

The hyperplane is a linear combination of the support vectors. In the soft margin case, there is only a limited amount of slack; every input does not get to be support vector. In the nonlinear case, the separating hypersurface may be embedded in an infinite-dimensional space, making it impossible to store. To borrow from the Wikipedia article, the normal vector $w$ is given by

$$w = \sum_i c_i y_i \phi(x_i)$$

where $\phi$ is the feature embedding function, and $c_i$ is a Lagrangian dual variable that is zero for points on the correct side of the margin. Instead, test points are classified through a kernel function $k(x_i,x_j) = \left< \phi(x_i), \phi(x_j) \right>$ like so:

$$x \to \mathrm{sgn}(\left<w , \phi(x)\right> + b) \equiv \mathrm{sgn} \left( b+\sum_i c_i y_i k(x_i, x)\right)$$

Notice how we avoided explicitly calculating $w$.

$\endgroup$
2
  • $\begingroup$ So if I may restate that: it is possible to just store the coefficients in the linear (non-kernel) case. In the kernel case, the only way to get to kernel space is via the kernel; the kernel requires vectors to operate on; the hypersurface is a linear combination of the support vectors. $\endgroup$
    – Doctor J
    Sep 30, 2017 at 23:42
  • $\begingroup$ Yes, that is right. $\endgroup$
    – Emre
    Oct 1, 2017 at 1:50
0
$\begingroup$

I would be very grateful if I could receive some help regarding generating hyperplane equation. I need to generate an equation for hyperplane, I have two independent variables and one binary dependent variable.

Regarding this following equation for svm , f(x)=sgn( sum_i alpha_i K(sv_i,x) + b )

I have two independent variables (say P and Q) with 130 point values for each variable. I used svm radial basis function for binary classification (0 and 1) and I calculated for radial basis kernelized case,and now I have one column of 51 y (i) alpha (i) or (dual coeffficients), two columns of 51 sv (support vectors)for P and Q, and one single value for b . I received these using scikit SVC.

https://scikit-learn.org/stable/modules/svm.html

So, how can I generate the equation now? Can I multiply those 51 y (i) alpha (i) or (dual coeffficients) with 51 sv (support vectors) for each variable P and Q so that I have two coefficients for P and Q so that finally my equation appears as : f(x)=sgn( mP + nQ +b) where m = sum of the (product of 51 sv of P with 51 dual coefficients) and n = sum of the (product of 51 sv of Q with 51 dual coefficients). i would be grateful for any kind of suggestion. Many thanks in advance.

$\endgroup$
2
  • $\begingroup$ Hi Alejandro. Welcome to DataScience SE. I see this as a separate question, rather than an answer to this question. So, please go ahead and post a new question :) $\endgroup$
    – Dawny33
    Feb 25, 2019 at 6:37
  • $\begingroup$ Thank you so much for your kind suggestion Sir. $\endgroup$ Feb 26, 2019 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.