In every explanation of SVMs, we're shown how training finds a hyperplane that best separates the data. Presumably then for inference, you just check which side of the plane a point is on.
However, all the "disadvantages of SVMs" posts [1, 2] lament that SVM models are large and slow because they end up storing most of the data as support vectors.
Why would SVMs store any of the data, rather than just the (coefficients of the) separating hyperplane? (And what is a "support vector" in the soft-margin case, when points of both classes are scattered on both sides of the hyperplane, anyway?)