36

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more ...


11

The X axis is the number of instances in the training set, so this plot is a data ablation study: it shows what happens for different amount of training data. The Y axis is an error score, so lower value means better performance. In the leftmost part of the graph, the fact that the error is zero on the training set until around 6000 instances points to ...


9

Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables. In general, tree based models are not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares. You can easily add a variable twice ...


8

It means that it can be run on a distributed system (i.e. on multiple networked computers). From XGBoost's documentation: The same code runs on major distributed environment(Hadoop, SGE, MPI) and can solve problems beyond billions of examples. The most recent version integrates naturally with DataFlow frameworks(e.g. Flink and Spark).


8

Yes, gradient boosted trees can make predictions outside the training labels' range. Here's a quick example: from sklearn.datasets import make_classification from sklearn.ensemble import GradientBoostingRegressor X, y = make_classification(random_state=42) gbm = GradientBoostingRegressor(max_depth=1, n_estimators=10, ...


7

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds. From the docs: class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, ...


6

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here). However, LightGBM may still return other warnings - e.g. No further splits with positive gain. This can be suppressed as follows (source: here ): lgb_train = lgb.Dataset(X_train, y_train, params={'verbose': -1}, free_raw_data=False) lgb_eval = lgb.Dataset(...


6

This is an open feature request (at time of writing): https://github.com/dmlc/xgboost/issues/2175 https://github.com/dmlc/xgboost/issues/3439 There, a very wasteful but working solution is mentioned: predict using ntree_limit for each number of trees of interest. I've put together a quick demonstration Colab notebook here. It also has been asked several ...


5

To suppress (most) output from LightGBM, the following parameter can be set. Suppress warnings: 'verbose': -1 must be specified in params={}. Suppress output of training iterations: verbose_eval=False must be specified in the train{} parameter. Minimal example: params = { 'objective': 'regression', 'learning_rate' : 0.9, ...


5

Your first interpretation is correct. One base learner will be added per boosting iteration/round and that is probably what people are referring to when talking about iterations. From wiki: One natural regularization parameter is the number of gradient boosting iterations M (i.e. the number of trees in the model when the base learner is a decision ...


5

Depends. The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric. Which one? Depends on which underlying problem you are solving, if what you are doing is to determine which algorithm is better, your conclusion will only be applicable to your specific dataset Accuracy: Is ...


4

The decision boundary in (4) from your example is already different from a decision tree because a decision tree would not have the orange piece in the top right corner. After step (1), a decision tree would only operate on the bottom orange part since the top blue part is already perfectly separated. The top blue part would be left unchanged. The boosted ...


4

I am going to talk about some ways you could do it later but first I want to talk about whether you should! If the relation that you describe exists XGB will be able to learn and detect it! There is no real benefit in "hard-coding" a rule into the algorithm, it won't speed up the training, it won't benefit accuracy, etc. Simply put the benefit of ML ...


4

At first glance, your conclusion appears correct, but there are some important caveats to keep in mind. First, what are the sizes of your training and validation sets? If your validation set is too small, then the observed difference may not be statistically significant. Second, you should verify that your validation set is a representative sample. (i.e. ...


4

Just to add some general thoughts to the other answers. Gradient boosting is fairly robust to overfitting through increasing the number of trees. Increasing the number of trees is expected to increase the performance if the learning rate is small. It is therefore generally considered best to set the number of trees through early stopping instead of treating ...


4

why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance) Clearly it wouldn't make sense to bag a bunch of shallow trees/weak learners. The average of many bad predictions will still be pretty bad. For many problems decision stumps (a tree with a single split node) will produce ...


4

I don't believe this is possible, CatBoost does target encoding per split, so you end up with different values of encoding at different trees. Before each split is selected in the tree (see Choosing the tree structure), categorical features are transformed to numerical. This is done using various statistics on combinations of categorical features and ...


4

It is pretty clear that your model is overfitting as your validation error is way higher than your training error. This also means that more data allows your model to overfit less. If you are to have 20k examples I'm betting that your validation error will be slightly lower and your training error will be slightly higher. However, I also see a plateau in ...


3

GB method works by minimizing a loss function and by splitting each node in a fashion that produces high pure leaves. there is no population formula being estimated and therefore you can estimate all types of relations between the target and the features. However I wouldn't put in the model correlated variables as: For gradient boosted trees, there's ...


3

I would say AUC is the best overall metric for classification but does not have to be the only metric, accuracy is useful too. For reference you can check this Quora regarding accuracy vs. AUC: They both measure different things, so they are complementary. Accuracy: Measures, for a given threshold, the percentage of points correctly classified, ...


3

Question 1: Bagging (Random Forest) is just an improvement on Decision Tree; Decision Tree has lot of nice properties, but it suffers from overfitting (high variance), by taking samples and constructing many trees we are reducing variance, with minimal effect on bias. Boosting is a different approach, we start with a simple model that has low variance and ...


3

By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the GridSearchCV act the way you want it to. Just pass a single split for the cv parameter, as @jncranton suggests; you can even go further and make that single split use all the data for the training ...


3

Your chart seems to show that light GBM models are very inconsistent in terms of F1 score. The other two types of model tend to have lower validation accuracy than training accuracy, suggesting overfitting is occurring to some extent (but this is ubiquitous in machine learning so it’s not a deal breaker by any means). The best median validation performance ...


3

This is actually a feature of tree-based models in general, not just gradient boosting trees. Not exactly a reference, but this Medium article explains why ordinal encoding is often more efficient. On the topic of safety, I think the author should have said that the use of ordinal encoding is more safe compared to linear methods, but still not perfectly safe....


3

Training of XGBoost is based on a boosting model, which is a general ensemble method creating a strong model from a number of weak models. This process is performed by building a model from the training dataset, then, creating a second model that attempts to correct the errors from the first model. Models are added until the training set is predicted ...


3

This is likely behavior when several of your original features are discrete. Each tree, when splitting, considers a split for each unique value of each feature (in the current node). For discrete features this is often significantly smaller than the number of rows, while for continuous features it is often very nearly the same as the number of rows. When ...


2

Boosting is a type of Ensemble Learning, but it is not the only one. Apart from stacking, bagging is also another type of Ensemble Learning. Ensemble Learning is the combination of individual models together trying to obtain better predictive performance that could be obtained from any of the constituent learning algorithms alone. Boosting involves ...


2

Ensemble learning combines predictions from multiple learners. Boosting methods are one way to form an ensemble. Stacking is another. The important difference between boosting and stacking (and other ensemble methods) is that boosting applies a number of weak learners sequentially and then produces a final result via a weighted majority vote. The learners ...


2

Performing such benchmark is not that easy. Meaning one can not just pick a few data set and run these models as there is a data dependency. In such cases, one need to simulate data through various process - the simulation helps to design various data in various condition. for example perhaps model one is doing a better job at binning so, the data with ...


Only top voted, non community-wiki answers of a minimum length are eligible