12
$\begingroup$

I am reading about SVM and I've faced to the point that non-kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

Apparently, SVM chooses the maximum margin classifier and logistic regression is the one that minimizes the cross-entropy loss. Are there situations where SVM performs better than logistic regression or vice-versa?

$\endgroup$
11
$\begingroup$

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.

logistic regression decision boundaries

What SVM does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. SVM is geometrically motivated rather than statistically.

enter image description here

None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an SVM for diminishing the risk of error on test data.

Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.

Yes, as stated SVM is based on geometrical properties of the data whilst logistic regression is based on statistical approaches.

In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?

Ostensibly, their results are not very different, but they are. SVMs are better for generalization 1, 2.

| improve this answer | |
$\endgroup$
  • $\begingroup$ How do the decision boundaries compare when the data is not linearly separable? $\endgroup$ – kennysong Jul 30 at 5:58
  • $\begingroup$ @kennysong That simply means use another approach :) which can separate them. You can use nonlinear SVM using kernel methods, or neural networks. If you have limitations, and you are supposed to utilise linear approaches, you may want to employ soft margin SVM, which can tolerate nonseparable data. $\endgroup$ – Media Jul 30 at 10:06
  • $\begingroup$ Ah yep, I was specifically wondering about logistic regression vs soft-margin linear SVM decision boundaries. Is there a notable difference? Maybe if the data is linearly separable + one outlier, SVM would be less sensitive to the outlier. $\endgroup$ – kennysong Jul 31 at 6:04
  • $\begingroup$ Yes, they are different. There are multiple regression approaches which deal with outliers differently. Soft SVM is basically dealing differently than those depending on the approach. The answer to your question cannot be summarised in solely one sentence. You may want to ask it. $\endgroup$ – Media Aug 1 at 16:31
2
$\begingroup$

Logistic regression isn’t trying to find a class boundary per se as linear SVMs do. LR attempts to model the logit-transformed y scores using predictors. To use a silly analogy , LR tries to put the function ‘through the points’ while SVMs attempt to put support vectors ‘between the points’

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.