Skip to main content

Questions tagged [gradient-boosting-decision-trees]

Filter by
Sorted by
Tagged with
1 vote
0 answers
22 views

Why do I get missing value error in XGBoost when it is supposed to be supported?

I read that in the latest versions of XGBoost, the model can handle missing values. I am using the model on some data that contains for example, the BMI, bloodpressure, age, binary data (yes/no) and ...
NancyChez's user avatar
0 votes
0 answers
13 views

Latest Tree-based models

What are the latest Tree-based models that are used in machine learning? Tell the new models except the old ones such as the Decision tree, Random Forest, Gradient Boosting, LightGBM, XGBoost, and ...
Madhes Monnish's user avatar
1 vote
1 answer
35 views

XGB predict_proba estimates don't match sum of leaves

When using an XGB model in the context of binary classification, I observed that the test estimates given by predict_proba were close but not equal to the results I ...
Juan Felipe Salamanca Lozano's user avatar
2 votes
0 answers
57 views

Transfer learning for tabular data

I wonder if transfer learning can be used in tabular data similarly to how it's used in neural networks for image recognition. My idea would be to train a "general" model and then "...
Dudelstein's user avatar
0 votes
0 answers
22 views

Question on theory from original GBM article

I am reading the original gradient boosting machine article and, maybe because my statistics are a bit rusty, have a few questions on one section. In section ...
CarterKF's user avatar
0 votes
0 answers
11 views

How are the cross validation and training processes interlinked here?

Please consider the code given below. ...
Masroor's user avatar
  • 101
2 votes
0 answers
58 views

Tuning the learning rate parameter for GBDT models

I've always been taught that decreasing the learning rate parameter in gbdt models such as XGBoost, LightGBM and Catboost will improve the out-of-sample performance, assuming the number of iterations ...
Casper's user avatar
  • 21
0 votes
0 answers
28 views

Gradient boosting algorithm implemented in LightGBM

I'm currently reading the documentation of LightGBM and I'm wondering which gradient boosting algorithm is exactly implemented there if choose boosting parameter as "gbdt" or "dart?&...
Julian's user avatar
  • 1
0 votes
0 answers
44 views

Gradient function in LogisitcLoss class

I am going through a code for XGBoost from scratch and I am referring to this repository here The log-loss function is given by On differentiating the above function with respect to y_pred (referring ...
Mehul Jain's user avatar
0 votes
0 answers
26 views

Gradient boosting to forecast just one-step ahead

I'm training a gradient boost algorithm (trying both XGBoost and LightGBM) for cash flow forecasting. I was able to do it well separating my training and test sets using the default separation (80/20) ...
vibebizarrinha's user avatar
0 votes
0 answers
101 views

Improving the performance of gradient boosting classifier

I am training a gradient boosting classifier on an imbalanced data but the model is not performing very well. These are the things I have done to improve the model's performance. Balanced the data ...
Toluwalope Owolabi's user avatar
0 votes
1 answer
50 views

How is the weight of each new weak learner is calculated in Xgboost?

In Xgboost we have multiple sequential weak learner. Let say I have weak learner WL1 and we fitted it on our data and we calulated the error. Now we have another weak learner WL2. And as I have read ...
XGB's user avatar
  • 15
0 votes
1 answer
240 views

Gradient Boosting - Why pseudo-residuals?

I have some questions I don't really understand regarding the Gradient Boosting algorithm with Decision Trees: Does the initial value matter as $\hat{y}$ or could you pick any, f.e between 0 and 1? ...
Caj's user avatar
  • 11
2 votes
1 answer
174 views

Model performance impact on social discrimination?

I am currently working on a project where the data concerns people and the dataset contain personal data with sensitive attributes. (typically: age, sex, handicap, race). Now it seems there are mainly ...
Lucas Morin's user avatar
  • 2,289
0 votes
0 answers
181 views

Missing values handling in LightGBM

I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific. In a regression setting with no categorical feature, I have ...
Augustin's user avatar
  • 101
0 votes
0 answers
9 views

Histogram creation in lightgbm in the train API and the scikit-learn API. Is it always benefitial to use the train API?

In the LightGBM for python we have a scikit-learn API in which (either for regression or for classification) there is fit method whose documentation is fit(X, y, sample_weight=None, init_score=None, ...
figs_and_nuts's user avatar
1 vote
0 answers
73 views

Understanding lgbm histogram building

...
figs_and_nuts's user avatar
0 votes
1 answer
24 views

How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV

I'm a programmer who tries to find he's way into ML world. so the Question might be basic. i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
Mostafa Bouzari's user avatar
0 votes
0 answers
73 views

XGBoost Architecture Diagram required

Good Day! My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
P_Z's user avatar
  • 101
0 votes
1 answer
142 views

How does LGBM make a prediction?

We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards. In my current understanding, it works as follows: Multiple "weak learners" are ...
Julian's user avatar
  • 121
0 votes
1 answer
295 views

randomness in lightgbm model training

What are the parameters that add randomness to the training of a lightgbm model? (for a large dataset) I have tried setting all parameters as default and letting bin_construct_sample_cnt be greater ...
kimo's user avatar
  • 1
0 votes
1 answer
598 views

How to reduce the false positives to improve the models performance?

I am currently building a binary classification model to predict order return rates. I used the GradientBoostingClassifier for training the model and also performed hyperparameter tuning using ...
Kedharnath Kb's user avatar
1 vote
1 answer
114 views

DART algorithm implementation. Converting mathematical notation to pseudocode

I am learning how DART algorithm (https://arxiv.org/abs/1505.01866) works and I want to implement it in C# I have the algorithm's description in mathematical notation and I don't understand most of it....
omike's user avatar
  • 11
1 vote
1 answer
404 views

Why is monotonic constraint disabled when using MAE as an objective to LGBM?

I tried to use monotonic constraints in LGBM, but if I use mean absolute error as an objective, it gives a warning that monotonic constraints cannot be done in l1. What is the reason? Thanks!
morqueatsz's user avatar
1 vote
1 answer
766 views

Why is gradient boosting better than random forest for unbalanced data?

I've searched everywhere and still couldn't figure this one out. This post mentioned that Gradient Boosting is better than Random Forest for unbalanced data. Why is that? Is Random Forest worse ...
Aldla E Aoepql's user avatar
0 votes
1 answer
238 views

Impact of many zeros in LightGBM Regressor training set [duplicate]

I have a LightGBM Regressor model with 15 features. 5 of these features have 98.7% NA for the training set. All five of the features are NA for each row. I impute the missing values with zero before I ...
ashton's user avatar
  • 1
1 vote
1 answer
24 views

How does machine learning algorithms process text?

I'm still new in machine learning and I've been trying to expand my knowledge about it. For my first project, I want to classify if a tweet is suicidal or not using the gradient boost algorithm. I do ...
Emman's user avatar
  • 11
0 votes
2 answers
42 views

If we train a binary classifier (lets say tree based) to predict ordinal data do they learn to interpolate?

Let's assume we have data about students in grade 10. We have test scores ranging from 0-100, however we are only provided two labels ; High score = if the score> 80% and low score if the score <...
Somen T's user avatar
1 vote
0 answers
31 views

ML Model that doesnt average/penalize extreme values

I have a 20k dataset, and a couple hundred of those lines are extreme values and 10 of them or so are even extremer values. But they are correct and have a unique tag, so when that tag comes up I am ...
Jroc561's user avatar
  • 11
0 votes
1 answer
45 views

Gradient tree boosting additive training

In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as $obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
Hadar's user avatar
  • 157
3 votes
3 answers
614 views

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing). I have ...
RajendraW's user avatar
0 votes
1 answer
1k views

How does Catboost regressor deal with categorical features at predict time?

I understand that Catboost regressor uses target-based encoding to convert categorical features to numerical features when training. But how does Catboost deal with categorical features at predict ...
cap99's user avatar
  • 3
0 votes
0 answers
72 views

Is it possible to embed a neural network layer into decision tree/random forest?

I want to do a classification task. I designed a customed layer for it. I also want to try decision tree/random forest, but as far as I know there is no way to embed my layer into a decsion tree/...
user900476's user avatar
1 vote
0 answers
108 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
lexan55's user avatar
  • 36
0 votes
1 answer
41 views

GridSearch multiplying the number of trees in XGboost?

I'm having an issue: after running an XGboost in a HalvingGridSearchCV, I receive a certain number of estimators (50 for example), but the number of trees is constantly being multiplied by 3. I don't ...
Cosapocha's user avatar
1 vote
0 answers
126 views

Catboost not working properly when I remove non important variables (source of randomness?)

I was wondering if anyone has encountered the same. The thing is, when I run a catboost boosting model, delete non important variables (feature importance by prediction importance = 0, in fact these ...
Tom's user avatar
  • 65
1 vote
1 answer
42 views

How is the probability prediction of a binary classifier predicted

I have a trained BDT and with sklearn predict_proba(X), I can get a probability between 0 and 1 for a predicted feature. I am now wondering, how this probability is calculated? Any ideas?
gamma's user avatar
  • 129
2 votes
1 answer
285 views

Random LightGBM Forest

I'm not completly sure about the bias/variance of boosted decision trees (LightGBM especially), thus I wonder if we generally would expect a performance boost by creating an ensemble of multiple ...
CutePoison's user avatar
0 votes
1 answer
449 views

XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values

I thought the consensus was that XGBoost was largely scale-invariant and scaling of features isn't really necessary but something's going wrong and I don't understand what. I have a range of features ...
Rob's user avatar
  • 103
1 vote
0 answers
100 views

references on how to use shap values without the shap package

I am familiar with the shap python package and how to use it, I also have a pretty good idea about shap values in general, but it is still new to me. What I'm requesting are references (ideally python ...
Phillip Maire's user avatar
1 vote
0 answers
19 views

How to compare new model to current production model?

Given new data, I trained the same model architecture and same hyperparameters (for example a random forest) as the current production model. How do I know that the new model that I trained is better ...
Stanley Gan's user avatar
0 votes
1 answer
340 views

Low accuracy on the test set

I have a dataset with 16 features and 32 class labels, which shows the following behavior: Neural network classification: high accuracy on train 100%, but low accuracy on the test set 3% (almost like ...
Albert's user avatar
  • 163
1 vote
1 answer
241 views

Understanding feature_parallel distributed learning algorithm in LightGBMClassifier

I want to understand feature_parallel algorithm in LightGBMClassifier. It describes how it is done traditionally and how LightGBM...
figs_and_nuts's user avatar
3 votes
3 answers
179 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
Chris_007's user avatar
  • 193
1 vote
1 answer
113 views

Reasons for a model predicting probability of being class 1 at x value

All, This is a general question. I have a binary classification which predicts if someone is rich or not. I had a question from someone asking that if the probability someone is rich is 0.6 and ...
Maths12's user avatar
  • 526
0 votes
1 answer
466 views

Catboost not able to handle a very simple dataset?

This is a post from a newbie and so might be a really poor question based on lack of knowledge. Thank you kindly! I'm using Catboost, which seems excellent, to fit a trivial dataset. The results are ...
user5406764's user avatar
0 votes
1 answer
122 views

Feature importance by random forest and boosting tree when two features are heavy correlated [closed]

I have asked this question here but seems no one is interested in it. Here is my understanding, pls correct me if there is any misunderstanding: Tree models is used ...
user6703592's user avatar
1 vote
1 answer
31 views

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

I have asked this question here, but seems no one was interested in it: https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the ...
user6703592's user avatar
0 votes
1 answer
494 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
Nikita Budilovskiy's user avatar
1 vote
0 answers
42 views

How are regression trees fitted in gradient boosting for classification?

What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds). For the rest of the trees, we try to fit ...
Nikhil Mishra's user avatar