If I understand the math behind the classic SVM for non-separable data correctly, the addition of a non-support vector (non-SV) should theoretically not alter the solution. My reasoning is that since its slack variable is zero (because it is correctly classified) and its lagrange multiplier is also zero (because it is a non-SV), it does not negatively effect both the value for wTw (w = weight vector) and the sum of the slack variables in the primal. Next, since in SVM the objective function results in a convex problem, the solution should stay the same. My first question is: is my interpretation of the mathematics behind SVMs correct?

Now, I sometimes do observe changes in the solution (i.e., different values for the bias term b and the Lagrange multipliers alpha) when I look at e.g. this applet (https://cs.stanford.edu/people/karpathy/svmjs/demo/) (see example below). The only explanation I could come up with is that this is due to the use of a numerical solver that stops if it is 'close enough' to the global solution. My second question is: is this a reasonable explanation?

As an illustration:


enter image description here

AFTER (more vertical line and shifted):

enter image description here



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.