3
$\begingroup$

The optimisation problem for support vector regression is (see http://alex.smola.org/papers/2003/SmoSch03b.pdf):

minimise: \begin{align*} C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \rVert^{2} \end{align*}

subject to the constraints:

\begin{align*} & y_{i} - <w,x_{i}> - b \leq \epsilon + \xi_{i} \\ & <w,x_{i}> + b - y_{i} \leq \epsilon + \xi^{*}_{i} \\ & \xi_{i}, \xi^{*}_{i} \geq 0 \end{align*}

I do not understand where the $\lVert w \rVert^{2}$ comes from. I understand how the $\lVert w \rVert^{2}$ is derived in the case of support vector classification (by maximizing the margin), but not in the case of regression.

The paper says that the "goal is to find a function $f(x)$ that has at most $\epsilon$ deviation from the actually obtained targets $y_{i}$ for all the training data, and at the same time is as flat as possible. In other words, we do not care about errors as long as they are less than $\epsilon$, but will not accept any deviation larger than this."

But I am not sure what "flat" means.

Does someone know?

$\endgroup$

1 Answer 1

4
$\begingroup$

Flat means parallel to the x axis; having a small slope. The smaller w is, the closer f(x) is to b; recall that $f(x) \equiv \left< w, x \right> + b$. One way to think about this is as a form of regularization; the flatter the function, the simpler or more parsimonious it is. This applies to both classification and regression.

$\endgroup$
2
  • $\begingroup$ thanks for your answer. So in the case of regression, there is no idea of maximising a margin? $\endgroup$
    – SVR
    Mar 15, 2015 at 20:43
  • $\begingroup$ You've used the $\epsilon$-insensitive formulation by Vapnik, whose goal is to find the flattest (simplest) solution that minimizes the $\epsilon$-insensitive loss, $\mathcal L(x) = (|x| - \epsilon)^+$. I suppose you could say that decreasing $w$ would increase the margin--distance to the line--for a given $\epsilon$ (think of a straight line with $\epsilon$ margin in the vertical direction and mentally rotate it so its slope tends to zero), but I don't usually think of it this way. $\endgroup$
    – Emre
    Mar 15, 2015 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.