What is the difference between SVM and logistic regression?

While reading the book by Aurelien Geron, I noticed that both logistic regression and SVM predict classes in exactly the same way, so I suspect there must be something that I am missing. In the Logistic regression chapter we can read:

$$σ(t) < 0.5$$ when $$t < 0$$, and $$σ(t) ≥ 0.5$$ when $$t ≥ 0$$, so a Logistic Regression model predicts $$1$$ if $$θ^T · x$$ is positive, and $$0$$ if it is negative.

Similarly, in the SVM chapter:

The linear SVM classifier model predicts the class of a new instance x by simply computing the decision function $$w^T · x + b = w_1 x_1 + ⋯ + w_n x_n + b$$: if the result is positive, the predicted class $$ŷ$$ is the positive class ($$1$$), or else it is the negative class ($$0$$).

I know that one way they could be different is because of the loss function they use: while log loss is used in logistic regression, SVM uses hinge loss to optimize the cost function. However, I would like to get this thing completely clear. How are the two models actually different?

Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:

$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$

Note that I am regarding the "primal", linear form of the SVM here.

In both cases the parameters $$\mathbf{w}$$ and $$b$$ are estimated by minimizing a certain function, and, as you correctly noted, the core difference between the models boils down to the use of different optimization objectives. For logistic regression:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i \log(1+e^{-z_i}),$$

where $$z_i = y_if_{\mathbf{w},b}(\mathbf{x}_i)$$.

For SVM:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i (1-z_i)_+ + \frac{1}{2C}\Vert \mathbf{w} \Vert^2$$

Note that the regularization term $$\Vert \mathbf{w} \Vert^2$$ may just as well be added to the logistic regression objective - this will result in regularized logistic regression.

You do not have to limit yourself to $$\ell_2$$-norm as the regularization term. Replace it with $$\Vert \mathbf{w} \Vert_1$$ in the SVM objective, and you will get $$\ell_1$$-SVM. Add both $$\ell_1$$ and $$\ell_2$$ regularizers to get the "elastic net regularization". In fact, feel free to pick your favourite loss, add your favourite regularizer, and voila - help yourself to a freshly baked machine learning algorithm.

This is not a coincidence. Any machine learning modeling problem can be phrased as the task of finding a probabilistic model $$M$$ which describes a given dataset $$D$$ sufficiently well. One general method for solving such a task is the technique of maximum a-posteriori (MAP) estimation, which suggests you should always choose the most probable model given the data:

$$M^* = \mathrm{argmax}_M P(M|D).$$

Using the Bayes rule and remembering that $$P(D)$$ is constant when the data is fixed:

\begin{align*} \mathrm{argmax}_M P(M|D) &= \mathrm{argmax}_M \frac{P(D|M)P(M)}{P(D)} \\ &= \mathrm{argmax}_M P(D|M)P(M) \\ &= \mathrm{argmax}_M \log P(D|M)P(M) \\ &= \mathrm{argmax}_M \log P(D|M) + \log P(M) \\ &= \mathrm{argmin}_M (-\log P(D|M)) + (-\log P(M)) \end{align*}

Observe how the loss turns out to be just another name for the (minus) log-likelihood of the data (under the chosen model) and the regularization penalty is the log-prior of the model. For example, the familiar $$\ell_2$$-penalty is just the minus logarithm of the Gaussian prior on the parameters:

$$-\log\left((2\pi)^{-m/2}e^{-\frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2}\right) = \mathrm{const} + \frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2$$

Hence, another way to describe the difference between SVM and logistic regression (or any other model), is that these two postulate different probabilitic models for the data. In logistic regression the data likelihood is given via the Bernoulli distribution (with $$p$$=sigmoid), while the model prior is uniform (or simply ignored). In SVM the data likelihood is modeled via some $$\mathrm{exp}(-\mathrm{hinge})$$ distribution (not sure it even has a name, but I hope you get the idea that undoing the minus-logarithm would always bring you back to $$P(D|M)$$, up to a constant), and the model prior is the Gaussian.

In practice, the two models have different properties, of course. For example, SVM has sparse dual representations, which makes it possible to kernelize it efficiently. Logistic regression, on the other hand, is usually well-calibrated (which is not the case with SVM). Hence, you choose the model based on your needs (or, if you are unsure, on whatever cross-validation tells you).

• Thanks for your crisp explaination. But one thing which is still not clear is, Both the Logistic regression and SVM models make predictions based on similar equations, although things under the hood are different. So, for an unseen data point, won't logistic regression and SVM predict it of the same class? – Akash Dubey Nov 25 '18 at 7:51
• The equation is the same, but the parameters (weights) are different, because you use different objectives to estimate them. Consider two one-dimensional linear classifiers f1(x) = sign(2x - 1) and f2(x) = sign(-x + 3). See whether they predict the same class for x = 100. – KT. Nov 25 '18 at 10:31
• Oh Yes. I get. One more thing, can we say that SVM is powerful then Logistic regression or the opposite? or It just depends on the cases and our data ? – Akash Dubey Nov 25 '18 at 10:33
• The difference boils down to different postulated distributions, and it would be unfair to say that one distribution is "more powerful" than the other. However, when people speak about SVM they most often imply a kernelized version of it, which is effectively a nonlinear classifier. You can, of course, also kernelize LR, but this would not be sparse, hence it is rarely, if ever, done. The kernelized, nonlinear SVM is indeed more powerful than a linear LR in terms of its representation power (e.g. its VC dimension). – KT. Nov 25 '18 at 10:40

It might be similar in the sense of finding some line to classify data, but the main differences are in their approach! Logistic regression try to maximize a probability and support vector machine try to maximize margins of the vectors. Also, SVM has some parameters to tune and LR has not! To know more about the difference you can see here and here.