Questions tagged [boosting]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
14 views

weighted quantile sketch in xgboost

I am unable to understand what is weighted quantile sketch in xgboost. Can anyone help me give an intuitive understanding of this?
4
votes
1answer
50 views

What is Pruning & Truncation in Decision Trees?

Pruning & Truncation As per my understanding Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to ...
1
vote
1answer
18 views

best way to regularize gradient boosting regressor?

i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets). https://scikit-learn.org/stable/modules/generated/sklearn.ensemble....
0
votes
0answers
11 views

Improve the results of imbalanced multi-classification multi-lables data

I have 10k rows of multi-classification (x1..x27,y), size of dataframe is: 28*10k and its ...
4
votes
2answers
84 views

Bagging vs Boosting, Bias vs Variance, Depth of trees

I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees First ...
1
vote
1answer
22 views

Error rate of AdaBoost weak learner always bigger than 0.5?

As far as i understand, weak learners of AdaBoost should never yield a error rate > 0.5 After training one, i only receive error rates above 0.5. How is that even possible? The AdaBoost Tree still ...
0
votes
1answer
21 views

Ensemble Techniques - Boosting

I understand boosting is a sequential learning technique and it use the prediction from previous model as a dataset for new model ,after adding weight to the misclassified data points. The point ...
0
votes
1answer
26 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
3
votes
2answers
166 views

XGBoost validation for number of trees

I have a simple Question: I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2 2.) With 500 ...
0
votes
1answer
19 views

How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

I am using XGBoost for a machine learning model that learns from tabular data. XGBoost uses boosting method on decision trees. When I look at the decision-making logic of decision trees, I notice the ...
1
vote
1answer
26 views

How to control the amount of positives in classification?

I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "...
0
votes
0answers
10 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
1
vote
0answers
83 views

Use LightGBM or FFM - imbalanced dataset

I have a highly imabalanced dataset but one that is not sparse. In train there are 1328 positives out of 104000. In validation ...
1
vote
0answers
13 views

different outcome of feature importance and coefficient from same data

I built a regression model to predict profit based on client, sales person, product category, client industry and client region. After trying several models with tuning hyperparameters, I found that ...
1
vote
1answer
50 views

Bagging with Neural Networks Best practices

I am trying to build a majority vote system for 3 Neural Networks, and I came across the concept of Bagging method. Actually, I want to use neural networks as weak learners (I know it's debatable, but ...
0
votes
1answer
27 views

How does one decide when to use boosting over bagging algorithm?

What kind of problem, circumstances and data makes it more suitable to apply boosting instead of bagging methods?
4
votes
1answer
119 views

sklearn's cross_validate does not work with catboost

I would like to use cross validation with catboost. Since I do not just want to use catboost but also sampling I am using a ...
4
votes
1answer
366 views

How to extract trees in XGBoost?

I want to extract each tree so that I can feed it with any data, and see the output. ...
2
votes
1answer
54 views

Gradient Boosted Decision Trees How to Find Prediction of Each Tree?

I'm doing a project. I have a classification problem that I should solve using gradient boosted decision trees. What I want to do is create a matrix that gives prediction of each decision tree for ...
1
vote
0answers
12 views

Gradient Boosting Partial Dependency Plot

I have been trying to generate a partial development plot using gradient boosting. The Plot looks like as below. My question is why the plot shows two or three steps rather than several broken ...
1
vote
0answers
87 views

How to save a lightGBM model that updates predictions after each fold?

Hi When I use gradient boosting on Kaggle and large data sets, I run a code like this: ...
2
votes
2answers
122 views

Lowering learning rate makes my accuracy on the validation set go down

I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase ...
0
votes
1answer
15 views

If There is a case where decision trees are getting overfitted so by using gradient boost method do we solve that problem?

I have came across a case where my decision trees are getting overfitting so by using methods like gradient boost can I solve that problem.
1
vote
1answer
54 views

Can we use boosting algorithms like Adaboost and gradient boosting with only one classifier

I have been working on ensemble learning and I came across this doubt that unlike other ensemble learning algorithms like voting classifier a can we only use one classifier with boosting.
0
votes
1answer
22 views

How can the Adaboost technique be called an ensemble learning technique?

I have read that in ensemble learning we use the outputs of various classifiers to make the predictive modeling better but in Adaboost we just use one classifier and we make it a strong learner but ...
1
vote
0answers
42 views

Gradient boosting, where did the constant go?

In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e. $F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$ The constant is fitted first (i.e. if ...
2
votes
2answers
54 views

difference between model-based boosting and gradient boosting

What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf ...
0
votes
0answers
24 views

Using bagging and random forests together

I was looking at a kernel implementation (for text classification) and the following piece of code got me a little bit confused (I removed part of the features - in order to keep it light - as most of ...
0
votes
2answers
2k views

How to make LightGBM to suppress output?

I'm trying for a while to figure out how to "shut up" LightGBM. Especially, I want to suppress the output of LightGBM during training (the feedback on the boosting steps). My model is: ...
0
votes
0answers
117 views

Bad input shape; XGB

I'm trying out a simple code to test the xgboost library in python. My input matrix has 17 features and 16,718 observations X = (16718,17) My output has 3 ...
2
votes
1answer
325 views

Is the number of iterations in gradient tree boosting just the number of trees?

I have been searching for a while and I just can't find any indication. When people talk about iterations in algorithms like XGBoost or LightGBM, or Catboost, do they mean how many decision trees i.e. ...
0
votes
0answers
8 views

Use “big data” (via batchwise streaming) in boosting or neural nets

I often work with boosting (e.g. lightgbm) and neural nets (e.g. Keras) but I usually work with data that is "small enough" to be loaded in the RAM memory as a whole (except in the case of images ...
1
vote
0answers
17 views

Is there a term for measuring error on a second prediction based on the first's?

I have created a dataset which contains six values per row which may be the target value. Two rows for example: ...
3
votes
1answer
64 views

What does “exaggeration” mean in the context of Boosting?

I am learning boosting, the machine learning ensemble meta-algorithm. The professor is grouping 3 weak classifiers into an ensemble and said that before this time point it is easy to understand. Take ...
0
votes
0answers
12 views

Help in knowing if I am doing Boosting right on satellite data

I'm having trouble with a Machine Learning project, and knowing if I am performing methods properly or not. We are given data from a real satellite, and we have to predict when it will pass through ...
1
vote
1answer
43 views

How can I make ROC and compute AUC?

I created a boosting tree and got the probability for each tuple in my testing set. But I'm confused on how to combine each probability. Can someone tell me how to combine the probabilities?
0
votes
0answers
51 views

Combine AdaBoost and Support Vector Regression?

I have read several papers about using SVM instead of decision tree in AdaBoost, but I haven't seen any papers about using support vector regression (SVR) in AdaBoost. However, if using support vector ...
1
vote
2answers
164 views

which metric is better for boosting methods

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one. ...
3
votes
3answers
425 views

why do we need row sampling in random forests?

In random forests, where our estimators are decision trees, we do column (feature) sampling without replacement within an estimator, and with replacement in between estimators. This is perfectly fine ...
0
votes
1answer
62 views

Updating weights in Adaboost

I'm studying the Adaboost algorithm. This algorithm updates the weight after training. This is the table when they explain about weight on ...
0
votes
2answers
37 views

How to ensure same encoding pattern?

I created a XGBRegressor model with certain encoded 'object' dtypes in the data. Now if I want to run the model with new set of data which is freshly encoded it's giving wrong predictions. How to ...
2
votes
1answer
146 views

Gradient Boosting RandomSearchCV or GridSearchCV

In your algorithms, when you use Gradient Boosting, do you prefer RandomSearchCV or GridSearchCV in order to optimize your hyperparameters ? Thanks for sharing your experience.
1
vote
1answer
102 views

Is there any formal explanation for the sensitivity of AdaBoost to outliers?

AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
4
votes
1answer
276 views

Does Gradient Boosting detect non-linear relationships?

I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn. My questions are: 1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2,...
1
vote
2answers
50 views

Adaboost - Show that adjusting weights brings error of current iteration to 0.5

I'm trying to solve the following problem but I've gotten sort of stuck. So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$ and $\alpha_t = \frac{...
1
vote
1answer
417 views

What are the limitations while using XGboost algorithm? [closed]

Will XGBoost pose any problem while dealing with categorical variables with more than 2 levels. For example, occupation variable can have values like doctor, engineer, lawyer, data scientist, farmer e....
0
votes
1answer
34 views

Can parallel computing be utilized for boosting?

Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources ...
1
vote
0answers
19 views

SmoteBoost: Should SMOTE be ran individually for each iteration/tree in the boosting?

As per the paper on SmoteBoost, SMOTE is ran for each iteration of the boosting, generating N samples, which are further added to the original training data and the weight distribution of the ...
0
votes
0answers
749 views

Why Catboost is giving error with MultiClass

I am trying Catboost package with iris dataset with following code: ...
0
votes
1answer
55 views

Methods for augmenting binary datasets

I have a small (~100 samples) dataset with roughly 20 features which are mostly binary, and a few are numeric (~5). I wanted to use methods for augmenting the training set and see if I can get better ...