2
$\begingroup$

Following Andrew Ng's machine learning course, he explains how we can modify logistic regression to obtain SVM algorithm. First he replaces (sort of approximating) cross entropy loss with hinge loss as shown in the image below:

enter image description here

Then he removes the $\frac{1}{m}$ coefficient and divides the whole cost function by the regularization parameter $\lambda$, which leads to parameter $C$ behind the sigma ($C=\frac{1}{\lambda}$). Resulting cost function is shown in image below which is the minimization objective in SVM:

enter image description here

Later he shows that by choosing a very large value for parameter $C$, SVM will be a large margin classifier. Large value for parameter $C$ is the same as choosing a small value for parameter $\lambda$ in logistic regression. So, is logistic regression with a small value for $\lambda$ also a large margin classifier?

$\endgroup$
2
$\begingroup$

Logistic Regression is a large margin loss not classifier. (in literal sense, meaning there is a margin with the loss function not the classification boundary

There are two terms in the softmax loss, true values minus the exp or log of the observations (roughly and depending on the variant of loss) this just implies that distance of observed example from its true decision boundary needs to beat the log sum of the distances from all of the decision boundaries.

Because the softmax function is a probability distribution, the largest the log softmax can be is 0, so the log softmax returns a negative value (i.e. a penalty) that approaches 0 as the probability of the true class under the softmax function approaches 1.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Actually i read your answer but i didn't get the point. How those two changes ( 1)approximating cross entropy with hinge, 2) dividing minimization objective by $\lambda$) convert logistic regression to a large margin classifier? From my point of view, these two changes are not so fundamental to make differences between SVM and logistic regression resulting decision boundaries. $\endgroup$ – Mehran Torki Mar 13 at 8:05
0
$\begingroup$

I understood why logistic regression is not a large margin classifier by reading this medium post.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.