I'm currently studying from Andrew Ng's Stanford handouts here (I'm at part 8). Now from what I gathered from before, all the time our goal was to minimize ||w||^2 so that we can maximize the margin. However, now as he's writing about the regularization, he's saying:

The parameter C controls the relative weighting between the twin goals of making the ||w||^2 large (which we saw earlier makes the margin small) and of ensuring that most examples have functional margin at least 1.

Why would making the margin small be currently a "goal"?

  • $\begingroup$ I might be mistaken. But $\|w\|^2$ is simply the regularization term. $\endgroup$ – Ricardo Cruz Apr 13 '17 at 19:38

You are minimizing the entire loss equation. If it contains regularization, you force the weights to be small too. Having small weights is favorable characteristic because the algorithm is not focusing strongly on one feature, all happen to be important, so the risk of overfitting to some feature is smaller.

So it is really some tradeoff as always, between the big margin but small focus on dataset distinct examples.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.