Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
When we have linearly inseparable datasets and we are using machine learning algorithms such as SVMs, we use kernels to implicitly map datapoints into a feature space that makes them linearly separable.
But how do we know if a kernel has indeed, implicitly, been successful in making the datapoints linearly separable in the new feature space? What is the guarantee?
You cannot guarantee this. Some data is not separable by any kernel because of duplicates.
By trying too hard, you will cause overfitting. Essentially, you force the implicit mapping to be so complex it contains a copy of your training data (which is exactly what happens if you choose a too small bandwidth with RBF).
If you want a good generalization performance, you will have to tolerate some errors, and use e.g. soft-margin and such techniques.
Perfect separation is not something to aim for. Such a guarantee is just a guarantee of being able to overfit! Use cross-validation to reduce the risk of overfitting and find the right balance between being optimal on training data and actual performance.
