12
$\begingroup$

I am reading about SVM and I've faced to the point that non-kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

Apparently, SVM chooses the maximum margin classifier and logistic regression is the one that minimizes the cross-entropy loss. Are there situations where SVM performs better than logistic regression or vice-versa?

$\endgroup$

2 Answers 2

11
$\begingroup$

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.

logistic regression decision boundaries

What SVM does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. SVM is geometrically motivated rather than statistically.

enter image description here

None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an SVM for diminishing the risk of error on test data.

Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.

Yes, as stated SVM is based on geometrical properties of the data whilst logistic regression is based on statistical approaches.

In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?

Ostensibly, their results are not very different, but they are. SVMs are better for generalization 1, 2.

$\endgroup$
4
  • $\begingroup$ How do the decision boundaries compare when the data is not linearly separable? $\endgroup$
    – kennysong
    Jul 30, 2020 at 5:58
  • $\begingroup$ @kennysong That simply means use another approach :) which can separate them. You can use nonlinear SVM using kernel methods, or neural networks. If you have limitations, and you are supposed to utilise linear approaches, you may want to employ soft margin SVM, which can tolerate nonseparable data. $\endgroup$ Jul 30, 2020 at 10:06
  • $\begingroup$ Ah yep, I was specifically wondering about logistic regression vs soft-margin linear SVM decision boundaries. Is there a notable difference? Maybe if the data is linearly separable + one outlier, SVM would be less sensitive to the outlier. $\endgroup$
    – kennysong
    Jul 31, 2020 at 6:04
  • $\begingroup$ Yes, they are different. There are multiple regression approaches which deal with outliers differently. Soft SVM is basically dealing differently than those depending on the approach. The answer to your question cannot be summarised in solely one sentence. You may want to ask it. $\endgroup$ Aug 1, 2020 at 16:31
2
$\begingroup$

Logistic regression isn’t trying to find a class boundary per se as linear SVMs do. LR attempts to model the logit-transformed y scores using predictors. To use a silly analogy , LR tries to put the function ‘through the points’ while SVMs attempt to put support vectors ‘between the points’

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.