If I understand the math behind the classic SVM for non-separable data correctly, the addition of a non-support vector (non-SV) should theoretically not alter the solution. My reasoning is that since its slack variable is zero (because it is correctly classified) and its lagrange multiplier is also zero (because it is a non-SV), it does not negatively effect both the value for wTw (w = weight vector) and the sum of the slack variables in the primal. Next, since in SVM the objective function results in a convex problem, the solution should stay the same. My first question is: is my interpretation of the mathematics behind SVMs correct?
Now, I sometimes do observe changes in the solution (i.e., different values for the bias term b and the Lagrange multipliers alpha) when I look at e.g. this applet (https://cs.stanford.edu/people/karpathy/svmjs/demo/) (see example below). The only explanation I could come up with is that this is due to the use of a numerical solver that stops if it is 'close enough' to the global solution. My second question is: is this a reasonable explanation?
As an illustration:
BEFORE:
AFTER (more vertical line and shifted):
