Questions tagged [boosting]
The boosting tag has no usage guidance.
123
questions
0
votes
0
answers
19
views
Theoretical model performance
this is a theoretical question that I would like to learn more form. Let's say we have some task and we have four datasets A, B, C and D. Using this data I want to obtain the best neural network for ...
- 121
0
votes
0
answers
13
views
i am facing problem in combining the forecasts generated by multiple stacked GRU models
first i have used CEEMDAN, to decompose multi-variate data into different IMFs, after that for each IMF stacked GRU model is trained, which gives results for each IMFs,
for the final predictions i ...
0
votes
0
answers
14
views
Is AdaBoost an online classificer only?
I have been using CControl library for classify data.
As I understand AdaBoost, it's an online non-linear classifier algorithm and not an algorithm, such as SVM, that gives you weights back were you ...
- 115
0
votes
1
answer
77
views
How is the weight of each new weak learner is calculated in Xgboost?
In Xgboost we have multiple sequential weak learner.
Let say I have weak learner WL1 and we fitted it on our data and we calulated the error. Now we have another weak learner WL2. And as I have read ...
- 15
0
votes
1
answer
362
views
Gradient Boosting - Why pseudo-residuals?
I have some questions I don't really understand regarding the Gradient Boosting algorithm with Decision Trees:
Does the initial value matter as $\hat{y}$ or could you pick any, f.e between 0 and 1?
...
- 11
0
votes
0
answers
22
views
why do we get negative predictions for boosting model even if the target variables are strictly positive value?
why do we get negative predictions for boosting model even if the target variables are strictly positive value?
I read another thread but I don't understand the explanation.
1
vote
0
answers
108
views
0
votes
0
answers
54
views
How to detect rare events in Time series?
I am working on a time series dataset in which each time step can be classified under 4 classes:
~EOI : P(~EOI) = .85
...
- 125
0
votes
1
answer
160
views
How does LGBM make a prediction?
We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards.
In my current understanding, it works as follows:
Multiple "weak learners" are ...
- 121
1
vote
1
answer
122
views
DART algorithm implementation. Converting mathematical notation to pseudocode
I am learning how DART algorithm (https://arxiv.org/abs/1505.01866) works and I want to implement it in C#
I have the algorithm's description in mathematical notation and I don't understand most of it....
- 11
0
votes
1
answer
744
views
Should highly correlated features be removed, even if they have different type of information?
A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and ...
- 25
1
vote
1
answer
248
views
Training XGBoost on time series features of varying sample length
I have some time series data that contain features that that go back anywhere from 5 to 50 years. I've considered imputation (e.g. taking the mean), but I'm not sure it's feasible to impute such large ...
- 41
0
votes
1
answer
46
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 167
0
votes
1
answer
134
views
CatBoost solves the problem of bias in pointwise gradient estimates
I've been reading the following papers: https://arxiv.org/abs/1810.11363, https://arxiv.org/abs/1706.09516 and https://www.researchgate.net/publication/318030603_Fighting_biases_with_dynamic_boosting.
...
1
vote
0
answers
18
views
Inject external prior distribution to my dataset
Input:
External Information - distribution between the feature_i & binary_target
Internal Dataset - tabular data. ...
- 21
-1
votes
0
answers
51
views
Does lightGBM handle multicollinearity? [duplicate]
I have a dataset after feature selection of around 6500 features and 10,000 data rows. I am using LightGBM model. I want to know if I should check the feature set for multicollinearity. If two or more ...
- 75
3
votes
3
answers
189
views
Example for Boosting
Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
- 193
1
vote
1
answer
33
views
If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree
I have asked this question here, but seems no one was interested in it:
https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the
...
- 127
0
votes
1
answer
2k
views
How does XGBoost perform in Parallel
So what I know about boosting technique, Like we train the data and update the weights of falsely predicted values or try to minimize the loss in the next model. So basically, it's the sequential ...
- 193
0
votes
2
answers
104
views
How is a single classifier fitted on AdaBoost?
The AdaBoost algorithm is:
My trouble is how the classifier $G_m(x)$ is trained, What does mean a classifier to be trained using weights $w_i$? Is it to fit classifier through $\{w_i,y_i\}_{i=1}^{N}$?...
- 123
1
vote
1
answer
87
views
Boosting algorithms only built with decision trees? why?
My understanding of boosting is just training models sequentially and learning from its previous mistakes.
Can boosting algorithms be built with bunch of logistic regression? or logistic regression + ...
- 469
2
votes
1
answer
191
views
Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”
The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
- 41
1
vote
0
answers
29
views
Contradictory learning curves in cross validation
I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$.
I'm trying to improve the fitting procedure using 10 fold cross ...
- 111
0
votes
1
answer
875
views
Is feature importance in XGBoost or in any other tree based method reliable?
This question is quite long, if you know how feature importance to tree based methods works i suggest you to skip to text below the image.
Feature importance (FI) in tree based methods is given by ...
- 123
2
votes
1
answer
144
views
What if root of a such tree is pruned in xgboost?
Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
- 123
0
votes
1
answer
75
views
GradientBoostingRegressor Text Classifier
I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input ...
- 31
1
vote
0
answers
15
views
building a boosting model for repeated measurments
I am working on an e-commerce data where the goal is to predict how will the user rate a movie from 1 to 5. We have a bunch of data from users but also from products. Some users have previously rated ...
- 343
5
votes
1
answer
647
views
Can Boosting and Bagging be applied to heterogeneous algorithms?
Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in ...
- 177
4
votes
3
answers
214
views
Understanding Weighted learning in Ensemble Classifiers
I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
- 225
1
vote
0
answers
15
views
Looking for CART/ML model that works with relative data [duplicate]
I am a beginner at AI and ML. I have been given a dataset, where I have noticed the columns are relative to one another. So is there any CART or ML model that can work with relative data ?
For example ...
- 11
0
votes
1
answer
411
views
Why does Catboost outperform other boosting algorithms?
I have noticed while working with multiple datasets that catboost with its default parameters tends to outperform lightgbm or xgboost with its default parameters even on a tabular dataset with no ...
- 43
2
votes
1
answer
67
views
How to handle highly Imabalanced classification?
I have been dealing with a classification problem. Real issue is the imbalance here
I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification ...
- 21
2
votes
1
answer
516
views
Forecasting using Boosting methods on Non-stationary Time Series data
Theoretical Noob question -
Can we use boosting methods to effectively forecast the future after being trained on a non-stationary time series? Or do you train/fit on the residual of the training set ...
- 197
1
vote
0
answers
2k
views
XGBoost: Typical gamma and min_child_weight range
What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm?
Is the range of min_child_weight correlated with the number of feature or samples in the training ...
- 745
1
vote
1
answer
74
views
Output of evaluation metric for XGBoost - is it cumulative?
On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
- 65
0
votes
0
answers
35
views
How to interpret gradient descent in boosting ensembles?
I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) ...
- 1,027
1
vote
1
answer
740
views
Tree-based algorithms and ordinal features
For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, ...
- 745
2
votes
1
answer
1k
views
Counting the number of trainable parameters in a gradient boosted tree
I recently ran the gradient boosted tree regressor using scikit-learn via:
GradientBoostingRegressor()
This model depends on the following hyperparameters:
Estimators ($N_1$)
Min Samples Leaf ($N_2$...
- 123
2
votes
1
answer
126
views
How to improve model performace when model shows a systemic pattern in residues
I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
- 171
5
votes
1
answer
99
views
Is the way to combine weak learners in AdaBoost for regression arbitrary?
I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression.
In paper Improving Regressors using Boosting Techniques, the final ...
- 175
4
votes
1
answer
183
views
What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?
I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding:
hessian ...
- 202
3
votes
0
answers
84
views
What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?
I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the ...
- 171
1
vote
0
answers
41
views
Solving the dual problem of boosting using column generation
In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem.
So lets say we have want to solve the following primal linear problem ...
- 121
1
vote
2
answers
1k
views
Does gradient boosting algorithm error always decrease faster and lower on training data?
I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which ...
- 202
2
votes
1
answer
416
views
splitting mechanism with one hot encoded variables (tree based/boosting)
I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
- 526
1
vote
0
answers
97
views
AdaBoost.R2 learning rate from scikit learn
AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar.
The implementation of regression for ...
0
votes
1
answer
418
views
Which other algorithms fit residuals like XGBoost? [closed]
XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which ...
- 124
0
votes
0
answers
790
views
Getting unexpected keyword error in CatBoostRegressor while using GridSearchCV
I am trying to use GridSearchCV on a CatBoostRegressor algorithm, but get some "unexpected keyword" errors on 3 different params (classes_count, auto_class_weights, and bayesian_matrix_reg) ...
- 1,979
6
votes
1
answer
732
views
On gradient boosting and types of encodings
I am having a look at this material and I have found the following statement:
For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly
more ...
- 161
1
vote
1
answer
115
views
Explanation on some steps of AdaBoost.R2
I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really ...