If you use logistic regression and the cross-entropy
cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.

What SVM
does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. SVM
is geometrically motivated rather than statistically.
None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?
They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an SVM
for diminishing the risk of error on test data.
Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.
Yes, as stated SVM
is based on geometrical properties of the data whilst logistic regression
is based on statistical approaches.
In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?
Ostensibly, their results are not very different, but they are. SVM
s are better for generalization 1, 2.