working through Dr. Ng's coursera course

cost function intro:

basically, you want to minimize the difference squared between your guess and the actual value

cost function for SVM: this or this

he says the cost function is now the norm of the vector representing the hyperparameters: sqrt(theta_1^2 + theta_2^2... )

Why is that the case?


Given a linearly separable data set, the intuition behind the SVM-algorithm is to find a hyperplane that seperates both classes and maximizes the margin (which is twice the distance of the hyperplane to the closest data point).

So from all hyperplanes that seperate both classes, we want to find the one with the biggest margin (to achieve the best generalization).

Let's imagine we have two parallel seperating hyperplanes defined by the vector $\theta$ and with no data points in between, their distance is then given by: $$m= \frac{2}{\|\theta\|}$$ To maximize this distance, the two hyperplanes must pass through data points of the opposing classes. In this case we can define another hyperplane in the middle (with same distance to both hyperplanes and the same vector $\theta$) that now has the margin $m$.

As you can see from the formula: the smaller $\|\theta\|$ the greater the margin $m$.

So maximizing the margin $m$ is equivalent to minimizing $\|\theta\|$. Thus, a separating hyperplane is optimal if it minimizes $\|\theta\|$.

This is just the rough intuition - if you want to go deeper into the maths behind it, I could recommend: http://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.