1
$\begingroup$

I am using SVM with linear kernel for one of my multiclass text classification problem and getting accuracy of 0.78. I also tried Random Forest but the best accuracy I could get was 0.72. This set me thinking what if I could simulate Random Forest but with a SVM classifier and not decision tree. I am not sure whether it will result in improved accuracy or not. I wanted to have something in favor before I implemented this and a quick google search didn't help either.

Any thoughts on this?

Improve this question
$\endgroup$
3
  • $\begingroup$ Why do the accuracies of those two models set you thinking of a simulating? $\endgroup$
    – kkk
    Commented Jun 10, 2017 at 19:52
  • $\begingroup$ @JekaterinaKokatjuhha because when I used Decision Tree the best I could get was 0.67, and RF gave 0.72. So, I thought if randomization can help DT, then why not SVM. $\endgroup$
    – ac-lap
    Commented Jun 11, 2017 at 3:32
  • $\begingroup$ Ensembles of different types of algorithms or the same type of algorithm with different training in some respect are seen commonly in Kaggle comps, for example. So to answer the essential question here, yes, ensembles of models of all types often lead to better results to the extent the underlying models correct each others bias. However, there seems to be an unstated question of 'is there an off the shelf "Random Vectors" algorithm, or similar?' - less certain about that one. $\endgroup$
    – Robert de Graaf
    Commented Jun 13, 2017 at 3:26

1 Answer 1

Reset to default
2
$\begingroup$

The notion of ensembles of models leading to better outcomes is widespread in the Data Science and Machine Learning communities. For example, the widely read text 'The Elements of Statistical Learning' by Hastie, Tibshirani and Friedman, devotes space to the general notion of model averaging (section 8.7-8.8) with further discussion of boosting in relation to trees (e.g. chapter 10). Hence, it is common to see examples of ensembled models in the wild, using any conceivable algorithm as the base, even to the point of effectively seeing ensembles of ensembles.

Specifically with respect to Support Vector Machines, there are at least a few attempts at improving SVM performance by using ensembles. A recent example is EnsemblesSVM (Claesen,De Smet, Suykens, De Moor - see homes.esat.kuleuven.be/~claesenm/ensemblesvm/). The authors promise that this algorithm will lead to better training times by ensembling lightly trained SVMs, and distribute a free implementation via the website above.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.