The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [boosting]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3
votes
2answers
53 views

Bagging vs Boosting, Bias vs Variance, Depth of trees

I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees First ...
1
vote
1answer
12 views

Error rate of AdaBoost weak learner always bigger than 0.5?

As far as i understand, weak learners of AdaBoost should never yield a error rate > 0.5 After training one, i only receive error rates above 0.5. How is that even possible? The AdaBoost Tree still ...
0
votes
1answer
19 views

Ensemble Techniques - Boosting

I understand boosting is a sequential learning technique and it use the prediction from previous model as a dataset for new model ,after adding weight to the misclassified data points. The point ...
0
votes
1answer
23 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
3
votes
2answers
126 views

XGBoost validation for number of trees

I have a simple Question: I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2 2.) With 500 ...
0
votes
1answer
19 views

How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

I am using XGBoost for a machine learning model that learns from tabular data. XGBoost uses boosting method on decision trees. When I look at the decision-making logic of decision trees, I notice the ...
1
vote
1answer
23 views

How to control the amount of positives in classification?

I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "...
0
votes
0answers
8 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
1
vote
0answers
25 views

Use LightGBM or FFM - imbalanced dataset

I have a highly imabalanced dataset but one that is not sparse. In train there are 1328 positives out of 104000. In validation ...
0
votes
1answer
38 views

When should I prefer XGBoost over others? [closed]

Given all the other bagging and boosting algorithms, when should I prefer XGBoost over others?
1
vote
0answers
12 views

different outcome of feature importance and coefficient from same data

I built a regression model to predict profit based on client, sales person, product category, client industry and client region. After trying several models with tuning hyperparameters, I found that ...
1
vote
1answer
37 views

Bagging with Neural Networks Best practices

I am trying to build a majority vote system for 3 Neural Networks, and I came across the concept of Bagging method. Actually, I want to use neural networks as weak learners (I know it's debatable, but ...
0
votes
1answer
22 views

How does one decide when to use boosting over bagging algorithm?

What kind of problem, circumstances and data makes it more suitable to apply boosting instead of bagging methods?
4
votes
1answer
60 views

sklearn's cross_validate does not work with catboost

I would like to use cross validation with catboost. Since I do not just want to use catboost but also sampling I am using a ...
4
votes
1answer
111 views

How to extract trees in XGBoost?

I want to extract each tree so that I can feed it with any data, and see the output. ...
2
votes
1answer
47 views

Gradient Boosted Decision Trees How to Find Prediction of Each Tree?

I'm doing a project. I have a classification problem that I should solve using gradient boosted decision trees. What I want to do is create a matrix that gives prediction of each decision tree for ...
1
vote
0answers
12 views

Gradient Boosting Partial Dependency Plot

I have been trying to generate a partial development plot using gradient boosting. The Plot looks like as below. My question is why the plot shows two or three steps rather than several broken ...
1
vote
0answers
35 views

How to save a lightGBM model that updates predictions after each fold?

Hi When I use gradient boosting on Kaggle and large data sets, I run a code like this: ...
1
vote
2answers
65 views

Lowering learning rate makes my accuracy on the validation set go down

I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase ...
0
votes
1answer
15 views

If There is a case where decision trees are getting overfitted so by using gradient boost method do we solve that problem?

I have came across a case where my decision trees are getting overfitting so by using methods like gradient boost can I solve that problem.
1
vote
1answer
42 views

Can we use boosting algorithms like Adaboost and gradient boosting with only one classifier

I have been working on ensemble learning and I came across this doubt that unlike other ensemble learning algorithms like voting classifier a can we only use one classifier with boosting.
0
votes
1answer
19 views

How can the Adaboost technique be called an ensemble learning technique?

I have read that in ensemble learning we use the outputs of various classifiers to make the predictive modeling better but in Adaboost we just use one classifier and we make it a strong learner but ...
1
vote
0answers
37 views

Gradient boosting, where did the constant go?

In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e. $F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$ The constant is fitted first (i.e. if ...
2
votes
2answers
40 views

difference between model-based boosting and gradient boosting

What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf ...
0
votes
0answers
24 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
0
votes
2answers
944 views

How to make LightGBM to suppress output?

I'm trying for a while to figure out how to "shut up" LightGBM. Especially, I want to suppress the output of LightGBM during training (the feedback on the boosting steps). My model is: ...
0
votes
0answers
67 views

Bad input shape; XGB

I'm trying out a simple code to test the xgboost library in python. My input matrix has 17 features and 16,718 observations X = (16718,17) My output has 3 ...
2
votes
1answer
148 views

Is the number of iterations in gradient tree boosting just the number of trees?

I have been searching for a while and I just can't find any indication. When people talk about iterations in algorithms like XGBoost or LightGBM, or Catboost, do they mean how many decision trees i.e. ...
0
votes
0answers
8 views

Use “big data” (via batchwise streaming) in boosting or neural nets

I often work with boosting (e.g. lightgbm) and neural nets (e.g. Keras) but I usually work with data that is "small enough" to be loaded in the RAM memory as a whole (except in the case of images ...
1
vote
0answers
16 views

Is there a term for measuring error on a second prediction based on the first's?

I have created a dataset which contains six values per row which may be the target value. Two rows for example: ...
3
votes
1answer
58 views

What does “exaggeration” mean in the context of Boosting?

I am learning boosting, the machine learning ensemble meta-algorithm. The professor is grouping 3 weak classifiers into an ensemble and said that before this time point it is easy to understand. Take ...
0
votes
0answers
12 views

Help in knowing if I am doing Boosting right on satellite data

I'm having trouble with a Machine Learning project, and knowing if I am performing methods properly or not. We are given data from a real satellite, and we have to predict when it will pass through ...
1
vote
1answer
42 views

How can I make ROC and compute AUC?

I created a boosting tree and got the probability for each tuple in my testing set. But I'm confused on how to combine each probability. Can someone tell me how to combine the probabilities?
0
votes
0answers
39 views

Combine AdaBoost and Support Vector Regression?

I have read several papers about using SVM instead of decision tree in AdaBoost, but I haven't seen any papers about using support vector regression (SVR) in AdaBoost. However, if using support vector ...
1
vote
2answers
152 views

which metric is better for boosting methods

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one. ...
3
votes
3answers
282 views

why do we need row sampling in random forests?

In random forests, where our estimators are decision trees, we do column (feature) sampling without replacement within an estimator, and with replacement in between estimators. This is perfectly fine ...
0
votes
1answer
48 views

Updating weights in Adaboost

I'm studying the Adaboost algorithm. This algorithm updates the weight after training. This is the table when they explain about weight on ...
0
votes
2answers
33 views

How to ensure same encoding pattern?

I created a XGBRegressor model with certain encoded 'object' dtypes in the data. Now if I want to run the model with new set of data which is freshly encoded it's giving wrong predictions. How to ...
2
votes
1answer
116 views

Gradient Boosting RandomSearchCV or GridSearchCV

In your algorithms, when you use Gradient Boosting, do you prefer RandomSearchCV or GridSearchCV in order to optimize your hyperparameters ? Thanks for sharing your experience.
1
vote
1answer
93 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
4
votes
1answer
217 views

Does Gradient Boosting detect non-linear relationships?

I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn. My questions are: 1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2,...
1
vote
2answers
42 views

Adaboost - Show that adjusting weights brings error of current iteration to 0.5

I'm trying to solve the following problem but I've gotten sort of stuck. So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$ and $\alpha_t = \frac{...
1
vote
1answer
312 views

What are the limitations while using XGboost algorithm? [closed]

Will XGBoost pose any problem while dealing with categorical variables with more than 2 levels. For example, occupation variable can have values like doctor, engineer, lawyer, data scientist, farmer e....
0
votes
1answer
28 views

Can parallel computing be utilized for boosting?

Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources ...
1
vote
0answers
17 views

SmoteBoost: Should SMOTE be ran individually for each iteration/tree in the boosting?

As per the paper on SmoteBoost, SMOTE is ran for each iteration of the boosting, generating N samples, which are further added to the original training data and the weight distribution of the ...
0
votes
0answers
652 views

Why Catboost is giving error with MultiClass

I am trying Catboost package with iris dataset with following code: ...
0
votes
1answer
52 views

Methods for augmenting binary datasets

I have a small (~100 samples) dataset with roughly 20 features which are mostly binary, and a few are numeric (~5). I wanted to use methods for augmenting the training set and see if I can get better ...
9
votes
1answer
126 views

What is meant by Distributed for a gradient boosting library?

I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library. What is meant by distributed? Have a nice day
2
votes
1answer
49 views

To which category does this algorithm belongs?

I have came across the Catboost package. Among the classes in categories in Sklearn, Catboost seems to belong to Ensemble methods. What are then the advantages of Catboost over AdaBoost, Bagging etc.?...
5
votes
1answer
5k views

Adaboost vs Gradient Boosting

How AdaBoost is different than Gradient Boosting algorithm since both of them works on Boosting technique? I could not figure out actual difference between these both algorithms from theory point of ...