Questions tagged [boosting]
The boosting tag has no usage guidance.
122
questions
0
votes
0
answers
9
views
why do we get negative predictions for boosting model even if the target variables are strictly positive value?
why do we get negative predictions for boosting model even if the target variables are strictly positive value?
I read another thread but I don't understand the explanation.
1
vote
0
answers
33
views
0
votes
0
answers
42
views
How to detect rare events in Time series?
I am working on a time series dataset in which each time step can be classified under 4 classes:
~EOI : P(~EOI) = .85
...
- 125
0
votes
1
answer
84
views
How does LGBM make a prediction?
We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards.
In my current understanding, it works as follows:
Multiple "weak learners" are ...
- 121
0
votes
0
answers
10
views
Scikit-learn's SAMME AdaBoost error fraction implementation
I am taking a look at scikit-learn's discrete SAMME implementation and came across the following logic for computing the weighted error fraction.
...
- 11
0
votes
0
answers
35
views
Retraining gradient boosting classifier on its hits and misses
I have trained a gradient boosting model on historical data to predict whether person registers a business or not (binary classification problem). Right now the model is on the stage of online A/B-...
0
votes
1
answer
58
views
DART algorithm implementation. Converting mathematical notation to pseudocode
I am learning how DART algorithm (https://arxiv.org/abs/1505.01866) works and I want to implement it in C#
I have the algorithm's description in mathematical notation and I don't understand most of it....
0
votes
1
answer
141
views
Should highly correlated features be removed, even if they have different type of information?
A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and ...
- 25
0
votes
0
answers
41
views
If possible in a regression model, is splitting up the target variable to more target variable (using multiple models) have any drawbacks?
As in the title, my main goal is to have one or more boosting regression model for a target variable(s). Let's call the main target d . I can split it up like <...
- 25
0
votes
0
answers
44
views
In Catboost, how do I calculate the gradient and delta of the data entering the leaf node?
I am studying the following paper : https://arxiv.org/pdf/1706.09516.pdf
On page 7, the first paragraph reads:
At the candidate splits evaluation step, the leaf value ∆(i) for example i is obtained ...
- 1
0
votes
1
answer
85
views
Training XGBoost on time series features of varying sample length
I have some time series data that contain features that that go back anywhere from 5 to 50 years. I've considered imputation (e.g. taking the mean), but I'm not sure it's feasible to impute such large ...
- 11
0
votes
1
answer
38
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 137
0
votes
1
answer
87
views
CatBoost solves the problem of bias in pointwise gradient estimates
I've been reading the following papers: https://arxiv.org/abs/1810.11363, https://arxiv.org/abs/1706.09516 and https://www.researchgate.net/publication/318030603_Fighting_biases_with_dynamic_boosting.
...
1
vote
0
answers
18
views
Inject external prior distribution to my dataset
Input:
External Information - distribution between the feature_i & binary_target
Internal Dataset - tabular data. ...
- 21
0
votes
0
answers
44
views
Does lightGBM handle multicollinearity? [duplicate]
I have a dataset after feature selection of around 6500 features and 10,000 data rows. I am using LightGBM model. I want to know if I should check the feature set for multicollinearity. If two or more ...
- 47
3
votes
3
answers
137
views
Example for Boosting
Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
- 193
1
vote
1
answer
26
views
If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree
I have asked this question here, but seems no one was interested in it:
https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the
...
- 127
0
votes
1
answer
1k
views
How does XGBoost perform in Parallel
So what I know about boosting technique, Like we train the data and update the weights of falsely predicted values or try to minimize the loss in the next model. So basically, it's the sequential ...
- 193
0
votes
2
answers
87
views
How is a single classifier fitted on AdaBoost?
The AdaBoost algorithm is:
My trouble is how the classifier $G_m(x)$ is trained, What does mean a classifier to be trained using weights $w_i$? Is it to fit classifier through $\{w_i,y_i\}_{i=1}^{N}$?...
- 123
1
vote
1
answer
54
views
Boosting algorithms only built with decision trees? why?
My understanding of boosting is just training models sequentially and learning from its previous mistakes.
Can boosting algorithms be built with bunch of logistic regression? or logistic regression + ...
- 457
2
votes
1
answer
167
views
Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”
The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after ...
- 41
1
vote
0
answers
29
views
Contradictory learning curves in cross validation
I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$.
I'm trying to improve the fitting procedure using 10 fold cross ...
- 111
0
votes
1
answer
605
views
Is feature importance in XGBoost or in any other tree based method reliable?
This question is quite long, if you know how feature importance to tree based methods works i suggest you to skip to text below the image.
Feature importance (FI) in tree based methods is given by ...
- 123
2
votes
1
answer
123
views
What if root of a such tree is pruned in xgboost?
Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
- 123
0
votes
1
answer
58
views
GradientBoostingRegressor Text Classifier
I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input ...
- 31
1
vote
0
answers
15
views
building a boosting model for repeated measurments
I am working on an e-commerce data where the goal is to predict how will the user rate a movie from 1 to 5. We have a bunch of data from users but also from products. Some users have previously rated ...
- 323
5
votes
1
answer
533
views
Can Boosting and Bagging be applied to heterogeneous algorithms?
Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in ...
- 177
4
votes
3
answers
103
views
Understanding Weighted learning in Ensemble Classifiers
I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was ...
- 225
1
vote
0
answers
15
views
Looking for CART/ML model that works with relative data [duplicate]
I am a beginner at AI and ML. I have been given a dataset, where I have noticed the columns are relative to one another. So is there any CART or ML model that can work with relative data ?
For example ...
- 11
0
votes
1
answer
125
views
Why does Catboost outperform other boosting algorithms?
I have noticed while working with multiple datasets that catboost with its default parameters tends to outperform lightgbm or xgboost with its default parameters even on a tabular dataset with no ...
- 43
2
votes
1
answer
63
views
How to handle highly Imabalanced classification?
I have been dealing with a classification problem. Real issue is the imbalance here
I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification ...
- 21
2
votes
1
answer
363
views
Forecasting using Boosting methods on Non-stationary Time Series data
Theoretical Noob question -
Can we use boosting methods to effectively forecast the future after being trained on a non-stationary time series? Or do you train/fit on the residual of the training set ...
- 197
1
vote
0
answers
1k
views
XGBoost: Typical gamma and min_child_weight range
What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm?
Is the range of min_child_weight correlated with the number of feature or samples in the training ...
- 735
1
vote
1
answer
63
views
Output of evaluation metric for XGBoost - is it cumulative?
On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been ...
- 55
0
votes
0
answers
30
views
How to interpret gradient descent in boosting ensembles?
I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) ...
- 987
1
vote
1
answer
534
views
Tree-based algorithms and ordinal features
For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, ...
- 735
2
votes
1
answer
759
views
Counting the number of trainable parameters in a gradient boosted tree
I recently ran the gradient boosted tree regressor using scikit-learn via:
GradientBoostingRegressor()
This model depends on the following hyperparameters:
Estimators ($N_1$)
Min Samples Leaf ($N_2$...
- 123
2
votes
1
answer
100
views
How to improve model performace when model shows a systemic pattern in residues
I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
- 171
5
votes
1
answer
84
views
Is the way to combine weak learners in AdaBoost for regression arbitrary?
I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression.
In paper Improving Regressors using Boosting Techniques, the final ...
- 175
4
votes
1
answer
103
views
What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?
I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding:
hessian ...
- 182
3
votes
0
answers
76
views
What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?
I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the ...
- 171
1
vote
0
answers
40
views
Solving the dual problem of boosting using column generation
In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem.
So lets say we have want to solve the following primal linear problem ...
- 121
1
vote
2
answers
683
views
Does gradient boosting algorithm error always decrease faster and lower on training data?
I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which ...
- 182
2
votes
1
answer
356
views
splitting mechanism with one hot encoded variables (tree based/boosting)
I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature ...
- 506
1
vote
0
answers
86
views
AdaBoost.R2 learning rate from scikit learn
AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar.
The implementation of regression for ...
0
votes
1
answer
343
views
Which other algorithms fit residuals like XGBoost? [closed]
XGBoost and standard gradient boosting train learners to fit the residuals rather than the observations themselves. I understand that this aspect of the algorithm matches the boosting mechanism which ...
- 114
0
votes
0
answers
647
views
Getting unexpected keyword error in CatBoostRegressor while using GridSearchCV
I am trying to use GridSearchCV on a CatBoostRegressor algorithm, but get some "unexpected keyword" errors on 3 different params (classes_count, auto_class_weights, and bayesian_matrix_reg) ...
- 1,899
4
votes
1
answer
527
views
On gradient boosting and types of encodings
I am having a look at this material and I have found the following statement:
For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly
more ...
- 141
1
vote
1
answer
106
views
Explanation on some steps of AdaBoost.R2
I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really ...
2
votes
0
answers
28
views
Understanding additive function approximation or Understanding matching pursuit
I am trying to read Greedy function approximation: A gradient boosting machine.
On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach ...
- 821