SVM regularization - minimizing margin?

Question

I'm currently studying from Andrew Ng's Stanford handouts here (I'm at part 8). Now from what I gathered from before, all the time our goal was to minimize ||w||^2 so that we can maximize the margin. However, now as he's writing about the regularization, he's saying:

The parameter C controls the relative weighting between the twin goals of making the ||w||^2 large (which we saw earlier makes the margin small) and of ensuring that most examples have functional margin at least 1.

Why would making the margin small be currently a "goal"?

Answer 1

You are minimizing the entire loss equation. If it contains regularization, you force the weights to be small too. Having small weights is favorable characteristic because the algorithm is not focusing strongly on one feature, all happen to be important, so the risk of overfitting to some feature is smaller.

So it is really some tradeoff as always, between the big margin but small focus on dataset distinct examples.

Answer 2

No, making the margin small is not generally a goal. In the context of the linked notes, it is a goal if we know that there aren't outliers. This way, the margins may become smaller because we are imposing that all the points should have a certain functional margin.

Before that section of the notes, the author was assuming that the data was linearly separable and there weren't outliers. In that context it's true that what we want is to minimize only $\Vert w\Vert^2$ in order to find the hyperplane that will correctly separate all the data.

But what happens with the pressence of outliers is that they force the hyperplane to have lower margins than the desired ones. That is what the author refers to when he says:

[...] making the $\Vert w\Vert^2$ large (which we saw earlier makes the margin small) [...]

An intuitive way to see this is by looking at the plot given in that section of the notes :

There we can see that in order to separate all the points, the margins have been severely reduced. If that point wasn't an outlier this is what we would want. We would want to reduce our margin in order to separate all the points. But knowing that point is an outlier, what we want is to ignore the outliers and have bigger margins for the rest of the points.

Outliers lead to bigger $\Vert w\Vert$ because they make features to have more influence than the desired one (in that picture, the influence of the feature of the y-axis grows significantly i.e. the SVM is not learning the overall behaviour of data, it is learning the particular points/ noise on the data).

This is what the cost function (with the term $C$) takes into account. If we make $C$ bigger, then we are penalising more the fact that the points don't have a functional margin of $1$. This will lead to low margins in order to try to separate correctly all the data, so $\Vert w \Vert$ growing is less important.

On the other hand, if we make $C$ smaller, then what we are caring the most is to reduce $\Vert w \Vert$. This way we are giving less importance to the points having a functional margin less than $1\rightarrow$ the pressence of outliers will affect less.