Polynomial Kernel Parameters in SVMs

Question

In SVMs the polynomial kernel is defined as:

(scale * crossprod(x, y) + offset)^degree

How do the scale and offset parameters affect the model and what range should they be in? (intuitively please)

Are the scale and offset for numeric stability only (that's what it looks like to me), or do they influence prediction accuracy as well?

Can good values for scale and offset be calculated/estimated when the data is known or is a grid search required? The caret package always sets the offset to 1, but it does a grid search for scale. (Why) is an offset of 1 a good value?

Thanks

PS.: Wikipedia didn't really help my understanding:

For degree-d polynomials, the polynomial kernel is defined as

where x and y are vectors in the input space, i.e. vectors of features computed from training or test samples, is a constant trading off the influence of higher-order versus lower-order terms in the polynomial. When , the kernel is called homogeneous.(A further generalized polykernel divides by a user-specified scalar parameter .)

Neither did ?polydot's explanation in R's help system:

scale: The scaling parameter of the polynomial and tangent kernel is a convenient way of normalizing patterns (<-!?) without the need to modify the data itself

offset: The offset used in a polynomial or hyperbolic tangent kernel (<- lol thanks)

Answer 1

The offset parameter is sometimes called "bias" in classification tasks, and its intuitive understanding doesn't have to do with what kind of kernel is used. It is basically used to compensate for feature vectors that are not centered around 0.

I will try to intuitively explain what the bias does with a toy example. Let's say you have some feature vector x who's parameters are always negative. The set of weights w you use in your SVM (let's say linear, just for clarity) will perhaps transform the features into the range [0, 1] - so they will always be negative. But, those elements that belong to class 1 fall in the range [0, 0.5], and the ones that are from class 2 fall into the range [0.5, 1].

To classify into a class, the SVM uses 0 as the threshold breakpoint - if greater than 0, it is an element of class 1, if less than 0, it is an element of class -1. But in this case, all the elements will be classified into class 2. However, with a bias of 0.5 (in this linear case), they will be classified correctly.

This geometric interpretation doesn't quite work for more complex kernels, but the idea is the same: the bias term attempts to compensate for features that are not centered around zero. In practice, if the features are centered around zero the bias term isn't always needed. To get around the bias issue, you can either:

1. center your features and forget about the bias term (doesn't always work)

2. augment your feature data to include a 1 at the beginning. i.e.:

   [feat1 feat2 feat3] --> [feat1 feat2 feat3 1]
   

   for all of your feature vectors. Then the bias term will just be learned as another SVM weight parameter.

3. Learn your weights, then calculate the bias based on the optimal prediction rule:

That is, given all the support vectors $S$, the learned weights $\alpha_i$, the training class $y^{(i)}$, training features $x^{(i)}$, and features-to-use-for-prediction $x$, the predicted class will be $y(x)$. The optimal $w_0$ or bias will then be calculated as the average distance between the correct and calculated predicted class:

Hope that helps. I think the easiest thing is to center your features ;)