What is the difference between SVM and logistic regression?

Question

While reading the book by Aurelien Geron, I noticed that both logistic regression and SVM predict classes in exactly the same way, so I suspect there must be something that I am missing. In the Logistic regression chapter we can read:

$σ(t) < 0.5$ when $t < 0$, and $σ(t) ≥ 0.5$ when $t ≥ 0$, so a Logistic Regression model predicts $1$ if $θ^T · x$ is positive, and $0$ if it is negative.

Similarly, in the SVM chapter:

The linear SVM classifier model predicts the class of a new instance x by simply computing the decision function $w^T · x + b = w_1 x_1 + ⋯ + w_n x_n + b$: if the result is positive, the predicted class $ŷ$ is the positive class ($1$), or else it is the negative class ($0$).

I know that one way they could be different is because of the loss function they use: while log loss is used in logistic regression, SVM uses hinge loss to optimize the cost function. However, I would like to get this thing completely clear. How are the two models actually different?

Answer 1

Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:

$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$

Note that I am regarding the "primal", linear form of the SVM here.

In both cases the parameters $\mathbf{w}$ and $b$ are estimated by minimizing a certain function, and, as you correctly noted, the core difference between the models boils down to the use of different optimization objectives. For logistic regression:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i \log(1+e^{-z_i}),$$

where $z_i = y_if_{\mathbf{w},b}(\mathbf{x}_i)$.

For SVM:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i (1-z_i)_+ + \frac{1}{2C}\Vert \mathbf{w} \Vert^2$$

Note that the regularization term $\Vert \mathbf{w} \Vert^2$ may just as well be added to the logistic regression objective - this will result in regularized logistic regression.

You do not have to limit yourself to $\ell_2$-norm as the regularization term. Replace it with $\Vert \mathbf{w} \Vert_1$ in the SVM objective, and you will get $\ell_1$-SVM. Add both $\ell_1$ and $\ell_2$ regularizers to get the "elastic net regularization". In fact, feel free to pick your favourite loss, add your favourite regularizer, and voila - help yourself to a freshly baked machine learning algorithm.

This is not a coincidence. Any machine learning modeling problem can be phrased as the task of finding a probabilistic model $M$ which describes a given dataset $D$ sufficiently well. One general method for solving such a task is the technique of maximum a-posteriori (MAP) estimation, which suggests you should always choose the most probable model given the data:

$$M^* = \mathrm{argmax}_M P(M|D).$$

Using the Bayes rule and remembering that $P(D)$ is constant when the data is fixed:

\begin{align*} \mathrm{argmax}_M P(M|D) &= \mathrm{argmax}_M \frac{P(D|M)P(M)}{P(D)} \\ &= \mathrm{argmax}_M P(D|M)P(M) \\ &= \mathrm{argmax}_M \log P(D|M)P(M) \\ &= \mathrm{argmax}_M \log P(D|M) + \log P(M) \\ &= \mathrm{argmin}_M (-\log P(D|M)) + (-\log P(M)) \end{align*}

Observe how the loss turns out to be just another name for the (minus) log-likelihood of the data (under the chosen model) and the regularization penalty is the log-prior of the model. For example, the familiar $\ell_2$-penalty is just the minus logarithm of the Gaussian prior on the parameters:

$$ -\log\left((2\pi)^{-m/2}e^{-\frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2}\right) = \mathrm{const} + \frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2$$

Hence, another way to describe the difference between SVM and logistic regression (or any other model), is that these two postulate different probabilitic models for the data. In logistic regression the data likelihood is given via the Bernoulli distribution (with $p$=sigmoid), while the model prior is uniform (or simply ignored). In SVM the data likelihood is modeled via some $\mathrm{exp}(-\mathrm{hinge})$ distribution (not sure it even has a name, but I hope you get the idea that undoing the minus-logarithm would always bring you back to $P(D|M)$, up to a constant), and the model prior is the Gaussian.

In practice, the two models have different properties, of course. For example, SVM has sparse dual representations, which makes it possible to kernelize it efficiently. Logistic regression, on the other hand, is usually well-calibrated (which is not the case with SVM). Hence, you choose the model based on your needs (or, if you are unsure, on whatever cross-validation tells you).

Answer 2

It might be similar in the sense of finding some line to classify data, but the main differences are in their approach! Logistic regression try to maximize a probability and support vector machine try to maximize margins of the vectors. Also, SVM has some parameters to tune and LR has not! To know more about the difference you can see here and here.