Are support vector machines and logistic regression equivalent if data is linearly separable?

Question

I understand that SVMs separate data drawing an hyperplane with the biggest margin, but doesn't logistic regression do the same thing if data is linearly separable?

Answer 1

Very closely, but not exactly. This post by Georgios Drakos goes through the question mathematically and visually. Here is an image from the post:

which compares soft-margin SVM (SVM) and Logistic Regression (LR).

Answer 2

If the data are completely separable, during training, validation, and testing, then yes the two algorithms will perform equivalently. They both will find the optimal decision boundary to separate the data.

They're not the only algorithms that will draw such a boundary. A linear discriminant will also perform equivalently, as will many other algorithms. There isn't something special about a SVC and LR, they and others are all trying to find a way to divide the data in the way that makes the most sense (i.e. minimal error).

Completely separable data rarely are found in the wild though. It's true that if the data are separable then you could potentially just define a decision boundary yourself (i.e. if x>5, y=1 else y=0) and achieve perfect performance without the hassle of optimization. As soon as the data aren't completely separable, then you'll find differences in how the algorithms separate the data. You're correct by saying SVC attempts to find the biggest margin between data, and when it's not perfect, they'll use a loss function (usually hinge loss) to help guide the hyperplane to minimize the number of misclassifications. LR works off of probabilities, which is a little different but usually performs similarly.

Answer 3

SVM is a kernel-based method, with at its core, the classification being binary, and obeying Mercer's condition of the dot product. Also, there are various kernels of the SVM, like 'linear', 'poly' and 'radial basis function' kernel.

Generally, none of the datasets in the real world are linearly separable. Even a basis change or dimensionality reduction cannot bring it to linear space sometimes.

In Neural network at the last layer, applying logistic regression is more popular than doing a SVM.