How do I interpret the length-scale parameter of the RBF kernel?

Question

According to the Scikit-Learn documentation for the RBF kernel: The length scale of the kernel. If a float, an isotropic kernel is used. If an array, an anisotropic kernel is used where each dimension of l defines the length-scale of the respective feature dimension.

I am currently working on a problem where I am setting the length-scale of each individual feature (which I assume is synonymous with dimension here). My understanding is that a smaller length scale implies a more complex function.

My question is, can I use this parameter to explain how well a certain feature will help a model generalize to new data?

For example, if I have a data set which, after optimizing the length-scale value looks like this: [Feature_1: length-scale = 20] [Feature_2: length-scale = 1] [Feature_3: length-scale = 5]

Does this mean that, if I had to pick one feature which would help a model generalize to new data, it would be Feature_1? Is Feature_2 potentially causing my model to overfit? Are these fair assumptions to make?

Note: I am using support vector regression with this kernel.

Answer 1

Even though I am more familiar with the use of RBF kernel with Gaussian Processes, I think your intuition is correct since, generally speaking, a larger lengthscale means that the learnt function varies less in that direction, which is another way of saying that that feature is irrelevant for the learnt function.

So if you have to choose which feature is the most important you could say that the one with the lowest lengthscale parameter is the most relevant.

All this only applies if the characteristic scale of the features (the standard deviation for example) is already comparable (i.e. you have to standardize the features before optimizing the lengthscales if you want to retain this intuition).

That said, making assumptions about what is making you overfit is difficult from the lengthscale viewpoint. I think we could say that shorter lengthscales across many independent features are going to increase model complexity, and thus the risk of overfitting. As always there's an optimal complexity for improving out-sample performance.

Just to complete the answer, I like to track the complexity of a kernel model by evaluating the kernel matrix k(x1, x2) and computing its effective rank as in https://infoscience.epfl.ch/record/110188/files/RoyV07.pdf . A lower rank means a less complex model (because all the points look more similar in your implicit kernel space), whereas a higher effective rank allows learning very complex functions but you lose statistical power since in your kernel space all your points are "outliers", in the sense that they are apart from each other.

I hope that helps.