SVM, which range to use when normalizing

Question

I am using the SVM classifier from Scikit Learn. I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized data are now between 0 and 1.

Using GridSearchCV, I always end up picking the linear kernel for the SVC. Would it be better if I were to normalize having my values between -1 and 1?

I feel like the answer is yes, thinking about situations like this one:

But I wanted to know if there is an official answer. Thanks a lot

Answer 1

Typically one decides between a range based normalization technique and a mean and standard deviation based normalization technique, which is known as standardization.

Standardization is often preferred to some types of normalization as normalization can put too much weight on anomalous data at the extrema of the cohort. This is typically not a problem, but certain discretization techniques can then hide the interesting structure in the data in a very narrow region. In your case, the structure might be being missed by your grid search, which is why you always arrive at a linear kernel. I suggest first standardizing your data and retrying your grid search. Does this arrive at a non-linear kernel?

If not, then also just try choosing some different kernel types and see if they outperform the linear kernels after you have used a grid search to optimize the other parameters.

The 3D structure in your example that is not apparent when projected into 2D should not be particularly applicable to this case since dimensionality and linearity are very different concepts.

Hope this helps!

Answer 2

Nope. There no definite answer. You cannot arrive at a conclusive decision point unless you experiment and analyze for yourself on the data with various kernel tricks. So feel Free to experiment as much as you can to learn to your best abilities