10
$\begingroup$

R's caret package works with 180 models. The author warns that some of the package may be intractably slow or less accurate than top-choice models.

The author is not wrong about this. I've tried to train Boruta and evtree models and had to give up after they ran > 5 hours on a cluster.

The author links to a set of machine learning benchmarks, but those only cover performance of a small number of algorithms, comparing different implementations.

Is there some other resource I can turn to, for guidance on which of the 180 models are worth trying, and which will be very inaccurate or unreasonably slow?

| improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Totally depends on your data. What are trying to do, how much data you've got and what does it look like? $\endgroup$ – stmax Mar 16 '16 at 22:38
  • $\begingroup$ @stmax This is true. It definitely does depend in part on the specific data. But it's also somewhat generalizable, which is why they do ML benchmarking. I'm really just looking for some general benchmarks. At any one time I've got 4 - 5 different projects I'm working on and I'm asking this more for general / future reference than for a specific analysis. I typically deal with 40,000 - 2,000,000 rows and usually about 100 predictors. Most commonly multiclass dependent variables. $\endgroup$ – y0gapants Mar 16 '16 at 22:46
  • 1
    $\begingroup$ read this research where they compare 179 different models on 121 data sets. It talks about the accuracy of the models across the data sets, but not so much about the speed. $\endgroup$ – phiver Mar 17 '16 at 6:57
  • 1
    $\begingroup$ @phiver That's highly useful. I might publish one like that on speed if no one has done so. $\endgroup$ – Hack-R Apr 3 '16 at 15:46
13
$\begingroup$

Benchmarking mlr (default) learners on OpenML

Philipp Probst's Ml benchmarking The entire openml database of ML results.

Test from RStudio suggests SVM.

Mlmastery suggests LDA and Trial and Error.

Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fern ́andez-Delgado et al.

Paper concludes parallel random forest (parRF_t) is best followed by random forest, LibSVM with Gaussian kernel (svm), extreme learning machine with Gaussian kernel, C5.0 decision tree and multi-layer perceptron (avNNet).

The best boosting and bagging ensembles use LibSVM as base classifiers (in Weka), being slightly better than the single LibSVM classifier, and adaboost R (ensemble of decision trees trained using Adaboost.M1). The probabilistic neural network in Matlab, tuning the Gaussian kernel spread (pnn m), and the direct kernel perceptron in C (dkp C), a very simple and fast neural network proposed by us (Fern ́andez-Delgado et al.,2014), are also very near to the top-20.

Wainer, Jacques (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets Based on Fernandez-Delgado et al. (2014). "We have shown that random forests, RBF SVM, and gradient boosting machines are classification algorithm that most likely will result in the highest accuracy"

Rich Caruana & Alexandru Niculescu-Mizil () An Empirical Comparison of Supervised Learning Algorithms (classification) concludes with Platt-Calibrated Boosted Trees as best followed RF BagT Cal.SVM NN.

Many other studies include comparisons of models used. Some papers prefer SVM others SVM with radial-basis or polynomial kernel for classification. (maybe same thing)

From my own regressions on generated data I recommend earth(MARS) Cubist SVMlinear.

Manisha Thesis first runs tests on UCI Machine Learning Repository then soil fertility which is the focus of the thesis. Best models on UCI were :"elm-kernel is the ELM neural network but with Gaussian kernel", "svr is the support vector machine for regression, with Gaussian kernel using the Lib-SVM library with the C++ interface", extraTrees and cubist. The thesis includes great descriptions of each model and links to more papers."extraTrees achieved the best RMSE for 7 of 10 soil problems". Paper is definitely worth a read.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Most algorithms need careful tuning of hyperparameters, even OLS (choosing interactions, non-linearities etc.) "Blind" comparison will favour easy to tune algorithms like random forests. $\endgroup$ – Michael M Dec 9 '17 at 10:57
  • $\begingroup$ Or algorithms that detect interactions and non-linearity. Especially in my artificial toy tests. Cubist moba mars like models. SVM with advanced kernels too. $\endgroup$ – ran8 May 19 '18 at 10:10
  • 1
    $\begingroup$ There are good algorithms to detect interactions and linearities that need fine tuning to perform better. RF usually don't have much room for improvement, but others algo may improve much more with fine tuning. As an exemple, you can take xgboost that rank poorly in your benchmark despite being used extensively to win kaggle competitions. Another exemple of what poorly set parameters will do : your penalised regressions perform worse than your linear model ... $\endgroup$ – lcrmorin Feb 24 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.