If you use logistic regression and the
cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.
SVM does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there.
SVM is geometrically motivated rather than statistically.
None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?
They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an
SVM for diminishing the risk of error on test data.
Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.
Yes, as stated
SVM is based on geometrical properties of the data whilst
logistic regression is based on statistical approaches.
In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?
Ostensibly, their results are not very different, but they are.
SVMs are better for generalization 1, 2.