Cost function for support vector regression

Question

The optimisation problem for support vector regression is (see http://alex.smola.org/papers/2003/SmoSch03b.pdf):

minimise: \begin{align*} C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \rVert^{2} \end{align*}

subject to the constraints:

\begin{align*} & y_{i} - <w,x_{i}> - b \leq \epsilon + \xi_{i} \\ & <w,x_{i}> + b - y_{i} \leq \epsilon + \xi^{*}_{i} \\ & \xi_{i}, \xi^{*}_{i} \geq 0 \end{align*}

I do not understand where the $\lVert w \rVert^{2}$ comes from. I understand how the $\lVert w \rVert^{2}$ is derived in the case of support vector classification (by maximizing the margin), but not in the case of regression.

The paper says that the "goal is to find a function $f(x)$ that has at most $\epsilon$ deviation from the actually obtained targets $y_{i}$ for all the training data, and at the same time is as flat as possible. In other words, we do not care about errors as long as they are less than $\epsilon$, but will not accept any deviation larger than this."

But I am not sure what "flat" means.

Does someone know?

Answer 1

Flat means parallel to the x axis; having a small slope. The smaller w is, the closer f(x) is to b; recall that $f(x) \equiv \left< w, x \right> + b$. One way to think about this is as a form of regularization; the flatter the function, the simpler or more parsimonious it is. This applies to both classification and regression.