# Cost function for support vector regression

The optimisation problem for support vector regression is (see http://alex.smola.org/papers/2003/SmoSch03b.pdf):

minimise: \begin{align*} C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \rVert^{2} \end{align*}

subject to the constraints:

\begin{align*} & y_{i} - <w,x_{i}> - b \leq \epsilon + \xi_{i} \\ & <w,x_{i}> + b - y_{i} \leq \epsilon + \xi^{*}_{i} \\ & \xi_{i}, \xi^{*}_{i} \geq 0 \end{align*}

I do not understand where the $\lVert w \rVert^{2}$ comes from. I understand how the $\lVert w \rVert^{2}$ is derived in the case of support vector classification (by maximizing the margin), but not in the case of regression.

The paper says that the "goal is to find a function $f(x)$ that has at most $\epsilon$ deviation from the actually obtained targets $y_{i}$ for all the training data, and at the same time is as flat as possible. In other words, we do not care about errors as long as they are less than $\epsilon$, but will not accept any deviation larger than this."

But I am not sure what "flat" means.

Does someone know?

Flat means parallel to the x axis; having a small slope. The smaller w is, the closer f(x) is to b; recall that $f(x) \equiv \left< w, x \right> + b$. One way to think about this is as a form of regularization; the flatter the function, the simpler or more parsimonious it is. This applies to both classification and regression.
• You've used the $\epsilon$-insensitive formulation by Vapnik, whose goal is to find the flattest (simplest) solution that minimizes the $\epsilon$-insensitive loss, $\mathcal L(x) = (|x| - \epsilon)^+$. I suppose you could say that decreasing $w$ would increase the margin--distance to the line--for a given $\epsilon$ (think of a straight line with $\epsilon$ margin in the vertical direction and mentally rotate it so its slope tends to zero), but I don't usually think of it this way. – Emre Mar 15 '15 at 21:27