Questions tagged [boosting]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Is AdaBoost an online classificer only?

I have been using CControl library for classify data. As I understand AdaBoost, it's an online non-linear classifier algorithm and not an algorithm, such as SVM, that gives you weights back were you ...
euraad's user avatar
  • 115
0 votes
1 answer
34 views

How is the weight of each new weak learner is calculated in Xgboost?

In Xgboost we have multiple sequential weak learner. Let say I have weak learner WL1 and we fitted it on our data and we calulated the error. Now we have another weak learner WL2. And as I have read ...
XGB's user avatar
  • 15
0 votes
1 answer
134 views

Gradient Boosting - Why pseudo-residuals?

I have some questions I don't really understand regarding the Gradient Boosting algorithm with Decision Trees: Does the initial value matter as $\hat{y}$ or could you pick any, f.e between 0 and 1? ...
Caj's user avatar
  • 11
0 votes
0 answers
16 views

why do we get negative predictions for boosting model even if the target variables are strictly positive value?

why do we get negative predictions for boosting model even if the target variables are strictly positive value? I read another thread but I don't understand the explanation.
user392987's user avatar
1 vote
0 answers
57 views

Understanding lgbm histogram building

...
figs_and_nuts's user avatar
0 votes
0 answers
46 views

How to detect rare events in Time series?

I am working on a time series dataset in which each time step can be classified under 4 classes: ~EOI : P(~EOI) = .85 ...
Fr_nkenstien's user avatar
0 votes
1 answer
128 views

How does LGBM make a prediction?

We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards. In my current understanding, it works as follows: Multiple "weak learners" are ...
Julian's user avatar
  • 121
0 votes
0 answers
10 views

Scikit-learn's SAMME AdaBoost error fraction implementation

I am taking a look at scikit-learn's discrete SAMME implementation and came across the following logic for computing the weighted error fraction. ...
CarterKF's user avatar
0 votes
0 answers
39 views

Retraining gradient boosting classifier on its hits and misses

I have trained a gradient boosting model on historical data to predict whether person registers a business or not (binary classification problem). Right now the model is on the stage of online A/B-...
Fydorov Maxim's user avatar
0 votes
1 answer
92 views

DART algorithm implementation. Converting mathematical notation to pseudocode

I am learning how DART algorithm (https://arxiv.org/abs/1505.01866) works and I want to implement it in C# I have the algorithm's description in mathematical notation and I don't understand most of it....
omike's user avatar
  • 1
0 votes
1 answer
362 views

Should highly correlated features be removed, even if they have different type of information?

A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and ...
morqueatsz's user avatar
0 votes
1 answer
141 views

Training XGBoost on time series features of varying sample length

I have some time series data that contain features that that go back anywhere from 5 to 50 years. I've considered imputation (e.g. taking the mean), but I'm not sure it's feasible to impute such large ...
ron burgundy's user avatar
0 votes
1 answer
42 views

Gradient tree boosting additive training

In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as $obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
Hadar's user avatar
  • 147
0 votes
1 answer
110 views

CatBoost solves the problem of bias in pointwise gradient estimates

I've been reading the following papers: https://arxiv.org/abs/1810.11363, https://arxiv.org/abs/1706.09516 and https://www.researchgate.net/publication/318030603_Fighting_biases_with_dynamic_boosting. ...
Patricia Brezeanu's user avatar
1 vote
0 answers
18 views

Inject external prior distribution to my dataset

Input: External Information - distribution between the feature_i & binary_target Internal Dataset - tabular data. ...
orbgr's user avatar
  • 21
0 votes
0 answers
48 views

Does lightGBM handle multicollinearity? [duplicate]

I have a dataset after feature selection of around 6500 features and 10,000 data rows. I am using LightGBM model. I want to know if I should check the feature set for multicollinearity. If two or more ...
As13's user avatar
  • 77
3 votes
3 answers
165 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
Chris_007's user avatar
  • 193
1 vote
1 answer
29 views

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

I have asked this question here, but seems no one was interested in it: https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the ...
user6703592's user avatar
0 votes
1 answer
2k views

How does XGBoost perform in Parallel

So what I know about boosting technique, Like we train the data and update the weights of falsely predicted values or try to minimize the loss in the next model. So basically, it's the sequential ...
Chris_007's user avatar
  • 193
0 votes
2 answers
95 views

How is a single classifier fitted on AdaBoost?

The AdaBoost algorithm is: My trouble is how the classifier $G_m(x)$ is trained, What does mean a classifier to be trained using weights $w_i$? Is it to fit classifier through $\{w_i,y_i\}_{i=1}^{N}$?...
Davi Américo's user avatar
1 vote
1 answer
59 views

Boosting algorithms only built with decision trees? why?

My understanding of boosting is just training models sequentially and learning from its previous mistakes. Can boosting algorithms be built with bunch of logistic regression? or logistic regression + ...
haneulkim's user avatar
  • 469
2 votes
1 answer
177 views

Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”

The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
appeldaniel's user avatar
1 vote
0 answers
29 views

Contradictory learning curves in cross validation

I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$. I'm trying to improve the fitting procedure using 10 fold cross ...
user357269's user avatar
0 votes
1 answer
752 views

Is feature importance in XGBoost or in any other tree based method reliable?

This question is quite long, if you know how feature importance to tree based methods works i suggest you to skip to text below the image. Feature importance (FI) in tree based methods is given by ...
Davi Américo's user avatar
2 votes
1 answer
133 views

What if root of a such tree is pruned in xgboost?

Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
Davi Américo's user avatar
0 votes
1 answer
69 views

GradientBoostingRegressor Text Classifier

I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input ...
Jim's user avatar
  • 31
1 vote
0 answers
15 views

building a boosting model for repeated measurments

I am working on an e-commerce data where the goal is to predict how will the user rate a movie from 1 to 5. We have a bunch of data from users but also from products. Some users have previously rated ...
user702846's user avatar
5 votes
1 answer
582 views

Can Boosting and Bagging be applied to heterogeneous algorithms?

Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in ...
Ahmad Bilal's user avatar
4 votes
3 answers
133 views

Understanding Weighted learning in Ensemble Classifiers

I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
AnonymousMe's user avatar
1 vote
0 answers
15 views

Looking for CART/ML model that works with relative data [duplicate]

I am a beginner at AI and ML. I have been given a dataset, where I have noticed the columns are relative to one another. So is there any CART or ML model that can work with relative data ? For example ...
BannerG's user avatar
  • 11
0 votes
1 answer
250 views

Why does Catboost outperform other boosting algorithms?

I have noticed while working with multiple datasets that catboost with its default parameters tends to outperform lightgbm or xgboost with its default parameters even on a tabular dataset with no ...
Aastha Jha's user avatar
2 votes
1 answer
67 views

How to handle highly Imabalanced classification?

I have been dealing with a classification problem. Real issue is the imbalance here I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification ...
billie_joe's user avatar
2 votes
1 answer
427 views

Forecasting using Boosting methods on Non-stationary Time Series data

Theoretical Noob question - Can we use boosting methods to effectively forecast the future after being trained on a non-stationary time series? Or do you train/fit on the residual of the training set ...
a_jelly_fish's user avatar
1 vote
0 answers
2k views

XGBoost: Typical gamma and min_child_weight range

What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm? Is the range of min_child_weight correlated with the number of feature or samples in the training ...
thereandhere1's user avatar
1 vote
1 answer
67 views

Output of evaluation metric for XGBoost - is it cumulative?

On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
krissy_fong's user avatar
0 votes
0 answers
33 views

How to interpret gradient descent in boosting ensembles?

I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) ...
Fredrik's user avatar
  • 1,007
1 vote
1 answer
648 views

Tree-based algorithms and ordinal features

For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, ...
thereandhere1's user avatar
2 votes
1 answer
926 views

Counting the number of trainable parameters in a gradient boosted tree

I recently ran the gradient boosted tree regressor using scikit-learn via: GradientBoostingRegressor() This model depends on the following hyperparameters: Estimators ($N_1$) Min Samples Leaf ($N_2$...
ABIM's user avatar
  • 123
2 votes
1 answer
118 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
PPR's user avatar
  • 171
5 votes
1 answer
91 views

Is the way to combine weak learners in AdaBoost for regression arbitrary?

I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression. In paper Improving Regressors using Boosting Techniques, the final ...
Akira's user avatar
  • 175
4 votes
1 answer
140 views

What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding: hessian ...
Xaume's user avatar
  • 192
3 votes
0 answers
82 views

What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?

I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the ...
PPR's user avatar
  • 171
1 vote
0 answers
40 views

Solving the dual problem of boosting using column generation

In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem. So lets say we have want to solve the following primal linear problem ...
Code Pope's user avatar
  • 121
1 vote
2 answers
873 views

Does gradient boosting algorithm error always decrease faster and lower on training data?

I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which ...
Xaume's user avatar
  • 192
2 votes
1 answer
390 views

splitting mechanism with one hot encoded variables (tree based/boosting)

I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
Maths12's user avatar
  • 526
1 vote
0 answers
97 views

AdaBoost.R2 learning rate from scikit learn

AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar. The implementation of regression for ...
Lucien Ledune's user avatar
0 votes
1 answer
373 views

Which other algorithms fit residuals like XGBoost? [closed]

XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which ...
Bobby's user avatar
  • 114
0 votes
0 answers
706 views

Getting unexpected keyword error in CatBoostRegressor while using GridSearchCV

I am trying to use GridSearchCV on a CatBoostRegressor algorithm, but get some "unexpected keyword" errors on 3 different params (classes_count, auto_class_weights, and bayesian_matrix_reg) ...
Donald S's user avatar
  • 1,939
5 votes
1 answer
620 views

On gradient boosting and types of encodings

I am having a look at this material and I have found the following statement: For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly more ...
carlo_sguera's user avatar
1 vote
1 answer
108 views

Explanation on some steps of AdaBoost.R2

I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really ...
Lucien Ledune's user avatar