4
$\begingroup$

I'm having troubles trying to understand why SVM works well with high dimensional data, the case when p >> n.

I read the following: SVM is automatically regularized. You don't have to pick a regularization parameter because picking the widest separation margin is a way to automatically regularize.

However I don't understand why this implies that a SVM works well on this type of data. Also I read about that the optimization problem to maximize the margin doesn't depend on the dimensions, so in what depend?

$\endgroup$
5
  • $\begingroup$ The curse of dimensionality affects SVMs as well I assure you of this. On the other hand SVMs can use kernel mappings, which can map high-dimensional features to more informative spaces which can be easily separated linearly $\endgroup$
    – Nikos M.
    Oct 28 at 14:31
  • $\begingroup$ @NikosM. Indeed, but it works better than most of classifiers. Only having a kernel function does not address the issues. $\endgroup$
    – Norhther
    Oct 28 at 20:00
  • $\begingroup$ I doubt SVMs work better unconditionally and under all cases since both experience and no free lunch theorems inform us otherwise $\endgroup$
    – Nikos M.
    Oct 28 at 22:08
  • 2
    $\begingroup$ related stats.stackexchange.com/questions/484289/… $\endgroup$
    – Nikos M.
    Oct 29 at 15:33
  • 2
    $\begingroup$ also related stats.stackexchange.com/questions/64053/… $\endgroup$
    – Nikos M.
    Oct 29 at 15:39
1
$\begingroup$

The ever-present danger with high-dimensional data is overfitting. When there are a lot of features (p) and relatively few examples (n), it is easy for models to find spurious relationships between features and target.

There are two generic solutions to this problem: dimensionality reduction and regularization. Dimensionality reduction reduces the number of features prior to training. Regularization penalizes the model for adding complexity. For example, L1 or L2 regularization are commonly used in linear models to penalize the size of coefficients. This encourages models to "ignore" certain features by reducing their coefficient to zero.

So to your question directly: the reason that SVMs work well with high-dimensional data is that they are automatically regularized, and regularization is a way to prevent overfitting with high-dimensional data.

$\endgroup$
8
  • 3
    $\begingroup$ By that logic LASSO/RIDGE/ELASTIC NET should also work very well on high-dimensional data since they are automatically regularized. I am not saying they are not, but the OP's question is specifically about SVM. $\endgroup$ Oct 29 at 5:44
  • $\begingroup$ Regularization in svm is going to add just a subset of the training points to be used as support vectors (correct me otherwise) However, the dimension of those is going to be always p. So still don't get how this regularization is going to penalize the model in a dimensional sense. $\endgroup$
    – Norhther
    Oct 29 at 10:39
  • 1
    $\begingroup$ @Norhther, I think what the answerer wants to say is that maximum margin of separation (a feature of SVM algorithm) can lead to better generalisation, thus bypasses the overfitting trap. This has nothing to do with limiting dimensions to regularize. Also in high dimensions linear separation is often more feasible. However this should not be a sweeping statement as my other comments on top point out $\endgroup$
    – Nikos M.
    Oct 29 at 16:45
  • $\begingroup$ @user2974951 I understood OP's question to be "I know that SVMs are automatically regularized, but why does that make them suitable for high-dimensional data?" My answer is that regularization is one of the remedies for high dimensionality. $\endgroup$
    – zachdj
    Oct 29 at 17:18
  • $\begingroup$ OP didn't ask in what way are SVMs regularized or why the max-margin criterion leads to regularization. So IMO the question is only superficially about SVMs. It really asks "why do automatically regularized models work well with high-dimensional data?", using SVM as an example $\endgroup$
    – zachdj
    Oct 29 at 17:18
0
$\begingroup$

Not sure where your statement comes from, but you need to consider the ideas behind the SVM and how it works to answer your question.

Here is the summary:

  1. SVM approach is to actually map data to higher dimension space than the dataset has - to achieve better separability. You can refer to kernel trick article. SVM's advantage is that it works faster, and only samples near the boundary affect the separating hyperplane.

  2. Success of any model depends on proper parametrization, and in case of SVM - proper kernel choice. Here is an example of under- and overfitting of SVM from ISL book. You can read about regularization there too. enter image description here

  3. As noted in comments - there is no free lunch, and same SVM setup can work for some problems and fail for the others. The only way is to perform several experiments and observe quality for yourself on specific datasets you've got.

$\endgroup$
2
  • $\begingroup$ I'm sorry, but your answer is not telling me anything that I didn't know, and its not adressing the question. $\endgroup$
    – Norhther
    Nov 3 at 23:53
  • $\begingroup$ @Norhther Probably your question can be better formulated or put into some specific context. $\endgroup$
    – mikalai
    Nov 4 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.