Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [boosting]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

Question from a paper: I do not understand why it is stated that SGD employs the bootstrapping to calculate gradient?

In this paper, they state that: As SGD employs the bootstrapping (i.e., random sampling with replacement) [67] for gradient calculation, we can obtain the unbiased estimation of standard gradients ...
user avatar
  • 121
2 votes
3 answers
55 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
user avatar
  • 163
1 vote
1 answer
20 views

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

I have asked this question here, but seems no one was interested in it: https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the ...
user avatar
0 votes
0 answers
237 views

How does XGBoost perform in Parallel

So what I know about boosting technique, Like we train the data and update the weights of falsely predicted values or try to minimize the loss in the next model. So basically, it's the sequential ...
user avatar
  • 163
0 votes
2 answers
39 views

How is a single classifier fitted on AdaBoost?

The AdaBoost algorithm is: My trouble is how the classifier $G_m(x)$ is trained, What does mean a classifier to be trained using weights $w_i$? Is it to fit classifier through $\{w_i,y_i\}_{i=1}^{N}$?...
user avatar
0 votes
0 answers
12 views

XGB Regression: Is there a way to handle somewhat bimodal Y variable?

I am using XGBRegression to predict on continuous percentage data with 80% of the values around 100, 10% around 0 and 10% data distributed in the middle. Models are struggling with predictions around ...
user avatar
  • 1
0 votes
0 answers
27 views

Gradient boosting algorithms and filling categorical variables

I have house prices dataset Link on Kaggle and I am having some dilemma. Some categorical variables having explicit majority. If we look at MSZoning and SaleType columns, there is "RL" type ...
user avatar
1 vote
1 answer
43 views

Boosting algorithms only built with decision trees? why?

My understanding of boosting is just training models sequentially and learning from its previous mistakes. Can boosting algorithms be built with bunch of logistic regression? or logistic regression + ...
user avatar
  • 343
0 votes
0 answers
12 views

Train and test data fixed during boosting?

I have question about boosting algorithm. I know that boosting is a sequential process and it gives high weight to misclassification of previous model. Then, its' train and test data are fixed through ...
user avatar
  • 1
2 votes
1 answer
112 views

Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”

The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
user avatar
0 votes
0 answers
53 views

LightGBM boosting and bagging parameters

When training a gradient boosted decision tree model, I can use the LightGBM package to efficiently train my model. It's possible to define the hyperparameter search space with eg. ...
user avatar
1 vote
0 answers
26 views

Contradictory learning curves in cross validation

I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$. I'm trying to improve the fitting procedure using 10 fold cross ...
user avatar
0 votes
1 answer
208 views

Is feature importance in XGBoost or in any other tree based method reliable?

This question is quite long, if you know how feature importance to tree based methods works i suggest you to skip to text below the image. Feature importance (FI) in tree based methods is given by ...
user avatar
0 votes
0 answers
21 views

Uncertainty prediction in Gradient Boosted Tree based Quantile Regression

For an application, I am using a Gradient boosting Tree based quantile regression model (LightGBM, Catboot) to predict the 5th percentile of the target variable. The model predicts point estimates, ...
user avatar
  • 1
0 votes
0 answers
15 views

xgboost performance

XGBoostRegressor is not performing better than AdaBoostRegressor for the same set of parameters for some reason. Since my ...
user avatar
  • 101
2 votes
1 answer
41 views

What if root of a such tree is pruned in xgboost?

Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
user avatar
0 votes
1 answer
28 views

GradientBoostingRegressor Text Classifier

I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input ...
user avatar
  • 31
1 vote
0 answers
15 views

building a boosting model for repeated measurments

I am working on an e-commerce data where the goal is to predict how will the user rate a movie from 1 to 5. We have a bunch of data from users but also from products. Some users have previously rated ...
user avatar
5 votes
1 answer
201 views

Can Boosting and Bagging be applied to heterogeneous algorithms?

Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in ...
user avatar
3 votes
3 answers
68 views

Understanding Weighted learning in Ensemble Classifiers

I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
user avatar
1 vote
0 answers
15 views

Looking for CART/ML model that works with relative data [duplicate]

I am a beginner at AI and ML. I have been given a dataset, where I have noticed the columns are relative to one another. So is there any CART or ML model that can work with relative data ? For example ...
user avatar
  • 11
0 votes
1 answer
53 views

Why does Catboost outperform other boosting algorithms?

I have noticed while working with multiple datasets that catboost with its default parameters tends to outperform lightgbm or xgboost with its default parameters even on a tabular dataset with no ...
user avatar
2 votes
1 answer
48 views

How to handle highly Imabalanced classification?

I have been dealing with a classification problem. Real issue is the imbalance here I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification ...
user avatar
2 votes
1 answer
116 views

Forecasting using Boosting methods on Non-stationary Time Series data

Theoretical Noob question - Can we use boosting methods to effectively forecast the future after being trained on a non-stationary time series? Or do you train/fit on the residual of the training set ...
user avatar
  • 167
1 vote
0 answers
163 views

XGBoost: Typical gamma and min_child_weight range

What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm? Is the range of min_child_weight correlated with the number of feature or samples in the training ...
user avatar
1 vote
1 answer
25 views

Output of evaluation metric for XGBoost - is it cumulative?

On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
user avatar
0 votes
0 answers
21 views

How to interpret gradient descent in boosting ensembles?

I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) ...
user avatar
  • 867
1 vote
1 answer
144 views

Tree-based algorithms and ordinal features

For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, ...
user avatar
1 vote
1 answer
200 views

Counting the number of trainable parameters in a gradient boosted tree

I recently ran the gradient boosted tree regressor using scikit-learn via: GradientBoostingRegressor() This model depends on the following hyperparameters: Estimators ($N_1$) Min Samples Leaf ($N_2$...
user avatar
2 votes
1 answer
61 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
user avatar
  • 161
5 votes
1 answer
45 views

Is the way to combine weak learners in AdaBoost for regression arbitrary?

I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression. In paper Improving Regressors using Boosting Techniques, the final ...
user avatar
  • 175
4 votes
1 answer
54 views

What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding: hessian ...
user avatar
  • 203
3 votes
0 answers
54 views

What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?

I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the ...
user avatar
  • 161
1 vote
0 answers
35 views

Solving the dual problem of boosting using column generation

In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem. So lets say we have want to solve the following primal linear problem ...
user avatar
  • 121
1 vote
2 answers
384 views

Does gradient boosting algorithm error always decrease faster and lower on training data?

I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which ...
user avatar
  • 203
2 votes
1 answer
245 views

splitting mechanism with one hot encoded variables (tree based/boosting)

I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
user avatar
  • 466
1 vote
0 answers
52 views

AdaBoost.R2 learning rate from scikit learn

AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar. The implementation of regression for ...
user avatar
0 votes
1 answer
150 views

Which other algorithms fit residuals like XGBoost? [closed]

XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which ...
user avatar
  • 114
0 votes
0 answers
371 views

Getting unexpected keyword error in CatBoostRegressor while using GridSearchCV

I am trying to use GridSearchCV on a CatBoostRegressor algorithm, but get some "unexpected keyword" errors on 3 different params (classes_count, auto_class_weights, and bayesian_matrix_reg) ...
user avatar
  • 1,769
4 votes
1 answer
253 views

On gradient boosting and types of encodings

I am having a look at this material and I have found the following statement: For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly more ...
user avatar
1 vote
1 answer
30 views

Explanation on some steps of AdaBoost.R2

I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really ...
user avatar
2 votes
0 answers
21 views

Understanding additive function approximation or Understanding matching pursuit

I am trying to read Greedy function approximation: A gradient boosting machine. On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach ...
user avatar
9 votes
4 answers
2k views

Can Boosted Trees predict below the minimum value of the training label?

I am using gradient Gradient Boosted Trees (with Catboost) for a Regression task. Can GBtrees predict a label that is below the minimum (or above the max) that was seen in the training ? For instance ...
user avatar
  • 109
1 vote
1 answer
65 views

is it possible get a overfit underfit comparation between models, with this chart? (homework) [closed]

I am trying to interpret this chart. I am not sure how to interpret this, because, I think that the fact of the for examples LGBM Validation error, is wide and similar to train boxplot, there arent ...
user avatar
  • 317
10 votes
2 answers
842 views

What is a good interpretation of this 'learning curve' plot?

I read about the validation_curve and how interpret it to know if there are over-fitting or underfitting, but how can interpret the plot when the data is the error like this: The X-axis is "Nº ...
user avatar
  • 317
2 votes
2 answers
33 views

Can i use other regression types that arent based in decision trees to use it like a weak learners in gradient boosting?

I was thinking if i can use polynomial regression like a weak learners in gradient boosting but i read that decision trees are used for that and i cannot find anything that show me the possibility of ...
user avatar
1 vote
0 answers
32 views

Tuning parameters for gradient boosting/xgboost

In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ...
user avatar
  • 111
2 votes
1 answer
47 views

Random Forest Stacking Experiment for Imbalanced Data-set Problem

In order to solve a Imbalanced Dataset Problem, I experimented with Random Forest in the given manner (Somewhat inspired by Deep-Learning) Trained a Random Forest which will take in the input data ...
user avatar
6 votes
1 answer
746 views

Boosting with highly correlated features

I have a conceptual question. My understanding is, that Random Forest can be applied even when features are (highly) correlated. This is because with bagging, the influence of few highly correlated ...
user avatar
  • 6,872
2 votes
1 answer
1k views

Decreasing n_estimators is increasing accuracy in AdaBoost?

I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset. (X,Y are the predictor columns and the color is the label) As you can see there are exactly 16 points in either ...
user avatar