Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am reading about SVM and I've faced to the point that non-kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?
Apparently, SVM chooses the maximum margin classifier and logistic regression is the one that minimizes the cross-entropy loss. Are there situations where SVM performs better than logistic regression or vice-versa?
If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.
What SVM does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. SVM is geometrically motivated rather than statistically.
None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?
They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an SVM for diminishing the risk of error on test data.
Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.
Yes, as stated SVM is based on geometrical properties of the data whilst logistic regression is based on statistical approaches.
In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?
Ostensibly, their results are not very different, but they are. SVMs are better for generalization 1, 2.
Logistic regression isn’t trying to find a class boundary per se as linear SVMs do. LR attempts to model the logit-transformed y scores using predictors. To use a silly analogy , LR tries to put the function ‘through the points’ while SVMs attempt to put support vectors ‘between the points’
