Take the 2-minute tour ×
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It's 100% free, no registration required.

This question is in response to a comment I saw on another question.

The comment was regarding the Machine Learning course syllabus on Coursera, and along the lines of "SVMs are not used so much nowadays".

I have only just finished the relevant lectures myself, and my understanding of SVMs is that they are a robust and efficient learning algorithm for classification, and that when using a kernel, they have a "niche" covering number of features perhaps 10 to 1000 and number of training samples perhaps 100 to 10,000. The limit on training samples is because the core algorithm revolves around optimising results generated from a square matrix with dimensions based on number of training samples, not number of original features.

So does the comment I saw refer some real change since the course was made, and if so, what is that change: A new algorithm that covers SVM's "sweet spot" just as well, better CPUs meaning SVM's computational advantages are not worth as much? Or is it perhaps opinion or personal experience of the commenter?

I tried a search for e.g. "are support vector machines out of fashion" and found nothing to imply they were being dropped in favour of anything else.

And Wikipedia has this: http://en.wikipedia.org/wiki/Support_vector_machine#Issues . . . the main sticking point appears to be difficulty of interpreting the model. Which makes SVM fine for a black-box predicting engine, but not so good for generating insights. I don't see that as a major issue, just another minor thing to take into account when picking the right tool for the job (along with nature of the training data and learning task etc).

share|improve this question
    
See also stats.stackexchange.com/questions/tagged/svm –  StasK Jul 10 at 11:55

1 Answer 1

up vote 14 down vote accepted

SVM is a powerful classifier. It has some nice advantages (which I guess were responsible for its popularity)... These are:

  • Efficiency: Only the support vectors play a role in determining the classification boundary. All other points from the training set needn't be stored in memory.
  • The so-called power of kernels: With appropriate kernels you can transform feature space into a higher dimension so that it becomes linearly separable. The notion of kernels work with arbitrary objects on which you can define some notion of similarity with the help of inner products... and hence SVMs can classify arbitrary objects such as trees, graphs etc.

There are some significant disadvantages as well.

  • Parameter sensitivity: The performance is highly sensitive to the choice of the regularization parameter C, which allows some variance in the model.
  • Extra parameter for the Gaussian kernel: The radius of the Gaussian kernel can have a significant impact on classifier accuracy. Typically a grid search has to be conducted to find optimal parameters. LibSVM has a support for grid search.

SVMs generally belong to the class of "Sparse Kernel Machines". The sparse vectors in the case of SVM are the support vectors which are chosen from the maximum margin criterion. Other sparse vector machines such as the Relevance Vector Machine (RVM) perform better than SVM. The following figure shows a comparative performance of the two. In the figure, the x-axis shows one dimensional data from two classes y={0,1}. The mixture model is defined as P(x|y=0)=Unif(0,1) and P(x|y=1)=Unif(.5,1.5) (Unif denotes uniform distribution). 1000 points were sampled from this mixture and an SVM and an RVM were used to estimate the posterior. The problem of SVM is that the predicted values are far off from the true log odds.

RVM vs. SVM

A very effective classifier, which is very popular nowadays, is the Random Forest. The main advantages are:

  • Only one parameter to tune (i.e. the number of trees in the forest)
  • Not utterly parameter sensitive
  • Can easily be extended to multiple classes
  • Is based on probabilistic principles (maximizing mutual information gain with the help of decision trees)
share|improve this answer
2  
Thank you for this detailed answer. I have one question about it: The graph you included has no axis labels at the moment. I can see that RVM is tracking some theoretical result better than SVM in the graph, but I cannot figure out the context. –  Neil Slater Jul 9 at 13:10
    
Sorry for missing out on the figure description... i've now added a couple of lines... –  Debasis Jul 9 at 17:44
1  
I don't mean to be overly critical, but SVMs are NOT efficient. They have a cubic complexity in most cases, which is why there is a lot of phasing out happening. –  indico Jul 9 at 19:05
    
yes, standard convergence methods takes O(n^3)... but i think i've seen somewhere (may be from the home page of T. Joachims) that it's been reduced to O(n^2) –  Debasis Jul 9 at 20:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.