To which category does this algorithm belongs?

Question

I have came across the Catboost package. Among the classes in categories in Sklearn, Catboost seems to belong to Ensemble methods. What are then the advantages of Catboost over AdaBoost, Bagging etc.?

Answer 1

Ensemble Methods as defined in Wikipedia:

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.

All those methods you mentioned are tree-based ensemble models:

• Bagging (Breiman, 1996): Fit many large trees to bootstrap-resampled versions of the training data, and classify by majority vote.
• Random Forests (Breiman 1999): Fancier version of bagging (only a subset of features are selected at random, unlike bagging where all features are considered for splitting a node).
• Boosting (Freund & Shapire, 1996): Fit many large or small trees to reweighted versions of the training data. Classify by weighted majority vote. There is a nice article explaining gradient boosting trees.

In general (in terms of prediction capability, boosting to be the best):

Boosting > Random Forests > Bagging > Single Tree

You might be wondering where AdaBoost fits?

Adaptive Boosting (or in short AdaBoost, was the first really successful boosting algorithm) works on improving the base learner esp. where it fails on predictions. Please note that the base learner can be any machine learning algorithms upon which the boosting is applied to obtain a strong learner. When Decision stumps are used as base learner, AdaBoost is comparable to the above-mentioned boosting trees. You might be asking, again, what are their differences, see below (taken from this book):

Modern Boosting Trees

Due to the success of gradient boosting trees, there are types of boosting algorithms, namely: Gradient Boosting, XGBoost, and Catboost. They are conceptually very similar, yet they differ in e.g. sampling methods, regularizations, handling categorical features, performance etc. Strongly recommend checking this article out if interested to learn more.

Personal Note: About 1.5 year ago I was a fan of XGBoost (for many reasons), till I experimented Catboost. Now I really like Catboost. First of all, it easily handles a mixture of numerical and categorical features EVEN without coding the categorical ones. And the default hyperparameters give comparable results to fine-tuned hyperparameters in XGBoost, thus less hassle. At present Catboost community is smaller than let's say XGboost, kind of makes it less attractive, but it is growing. Last note: I am not affiliated to any of these methods/implementations.

Hope, it is now better! ;-)