Questions tagged [gradient-boosting-decision-trees]

Filter by
Sorted by
Tagged with
0 votes
0 answers
44 views

Missing values handling in LightGBM

I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific. In a regression setting with no categorical feature, I have ...
Augustin's user avatar
  • 101
0 votes
0 answers
6 views

Histogram creation in lightgbm in the train API and the scikit-learn API. Is it always benefitial to use the train API?

In the LightGBM for python we have a scikit-learn API in which (either for regression or for classification) there is fit method whose documentation is fit(X, y, sample_weight=None, init_score=None, ...
figs_and_nuts's user avatar
1 vote
0 answers
34 views

Understanding lgbm histogram building

...
figs_and_nuts's user avatar
0 votes
1 answer
21 views

How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV

I'm a programmer who tries to find he's way into ML world. so the Question might be basic. i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
Mostafa Bouzari's user avatar
0 votes
0 answers
55 views

XGBoost Architecture Diagram required

Good Day! My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
P_Z's user avatar
  • 1
0 votes
1 answer
88 views

How does LGBM make a prediction?

We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards. In my current understanding, it works as follows: Multiple "weak learners" are ...
Julian's user avatar
  • 121
0 votes
0 answers
23 views

Feature scales and feature importance

Tree-based algorithms do not require feature scaling before fitting, and I am working on gradient boosted tree models (and random forest) without scaling features. I'm curious if feature scaling ...
Matthew Son's user avatar
0 votes
1 answer
124 views

randomness in lightgbm model training

What are the parameters that add randomness to the training of a lightgbm model? (for a large dataset) I have tried setting all parameters as default and letting bin_construct_sample_cnt be greater ...
kimo's user avatar
  • 1
0 votes
1 answer
62 views

How to reduce the false positives to improve the models performance?

I am currently building a binary classification model to predict order return rates. I used the GradientBoostingClassifier for training the model and also performed hyperparameter tuning using ...
Kedharnath Kb's user avatar
0 votes
1 answer
62 views

DART algorithm implementation. Converting mathematical notation to pseudocode

I am learning how DART algorithm (https://arxiv.org/abs/1505.01866) works and I want to implement it in C# I have the algorithm's description in mathematical notation and I don't understand most of it....
michaelfromsomeplace's user avatar
0 votes
0 answers
111 views

Lightgbm model tuning produces unexpectedly different hyperparameters for similar datasets

I am trying to tune a lightgbm model for each half of a dataset that I have split by a particular feature (stock ticker in this case). Both halves have the same number of features, somewhat similar ...
Generalenthu's user avatar
1 vote
1 answer
240 views

Why is monotonic constraint disabled when using MAE as an objective to LGBM?

I tried to use monotonic constraints in LGBM, but if I use mean absolute error as an objective, it gives a warning that monotonic constraints cannot be done in l1. What is the reason? Thanks!
morqueatsz's user avatar
1 vote
1 answer
374 views

Why is gradient boosting better than random forest for unbalanced data?

I've searched everywhere and still couldn't figure this one out. This post mentioned that Gradient Boosting is better than Random Forest for unbalanced data. Why is that? Is Random Forest worse ...
Aldla E Aoepql's user avatar
0 votes
1 answer
147 views

Impact of many zeros in LightGBM Regressor training set [duplicate]

I have a LightGBM Regressor model with 15 features. 5 of these features have 98.7% NA for the training set. All five of the features are NA for each row. I impute the missing values with zero before I ...
ashton's user avatar
  • 1
1 vote
1 answer
23 views

How does machine learning algorithms process text?

I'm still new in machine learning and I've been trying to expand my knowledge about it. For my first project, I want to classify if a tweet is suicidal or not using the gradient boost algorithm. I do ...
Emman's user avatar
  • 11
0 votes
2 answers
40 views

If we train a binary classifier (lets say tree based) to predict ordinal data do they learn to interpolate?

Let's assume we have data about students in grade 10. We have test scores ranging from 0-100, however we are only provided two labels ; High score = if the score> 80% and low score if the score <...
Somen T's user avatar
1 vote
0 answers
28 views

ML Model that doesnt average/penalize extreme values

I have a 20k dataset, and a couple hundred of those lines are extreme values and 10 of them or so are even extremer values. But they are correct and have a unique tag, so when that tag comes up I am ...
Jroc561's user avatar
  • 11
0 votes
1 answer
38 views

Gradient tree boosting additive training

In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as $obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
Hadar's user avatar
  • 137
3 votes
3 answers
482 views

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing). I have ...
RajendraW's user avatar
0 votes
1 answer
795 views

How does Catboost regressor deal with categorical features at predict time?

I understand that Catboost regressor uses target-based encoding to convert categorical features to numerical features when training. But how does Catboost deal with categorical features at predict ...
cap99's user avatar
  • 3
0 votes
0 answers
58 views

Is it possible to embed a neural network layer into decision tree/random forest?

I want to do a classification task. I designed a customed layer for it. I also want to try decision tree/random forest, but as far as I know there is no way to embed my layer into a decsion tree/...
user900476's user avatar
1 vote
0 answers
91 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
lexan55's user avatar
  • 36
0 votes
1 answer
33 views

GridSearch multiplying the number of trees in XGboost?

I'm having an issue: after running an XGboost in a HalvingGridSearchCV, I receive a certain number of estimators (50 for example), but the number of trees is constantly being multiplied by 3. I don't ...
Cosapocha's user avatar
1 vote
0 answers
83 views

Catboost not working properly when I remove non important variables (source of randomness?)

I was wondering if anyone has encountered the same. The thing is, when I run a catboost boosting model, delete non important variables (feature importance by prediction importance = 0, in fact these ...
Tom's user avatar
  • 63
1 vote
1 answer
42 views

How is the probability prediction of a binary classifier predicted

I have a trained BDT and with sklearn predict_proba(X), I can get a probability between 0 and 1 for a predicted feature. I am now wondering, how this probability is calculated? Any ideas?
gamma's user avatar
  • 129
1 vote
1 answer
245 views

Random LightGBM Forest

I'm not completly sure about the bias/variance of boosted decision trees (LightGBM especially), thus I wonder if we generally would expect a performance boost by creating an ensemble of multiple ...
CutePoison's user avatar
0 votes
1 answer
263 views

XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values

I thought the consensus was that XGBoost was largely scale-invariant and scaling of features isn't really necessary but something's going wrong and I don't understand what. I have a range of features ...
Rob's user avatar
  • 103
1 vote
0 answers
73 views

references on how to use shap values without the shap package

I am familiar with the shap python package and how to use it, I also have a pretty good idea about shap values in general, but it is still new to me. What I'm requesting are references (ideally python ...
Phillip Maire's user avatar
1 vote
0 answers
18 views

How to compare new model to current production model?

Given new data, I trained the same model architecture and same hyperparameters (for example a random forest) as the current production model. How do I know that the new model that I trained is better ...
Stanley Gan's user avatar
0 votes
1 answer
313 views

Low accuracy on the test set

I have a dataset with 16 features and 32 class labels, which shows the following behavior: Neural network classification: high accuracy on train 100%, but low accuracy on the test set 3% (almost like ...
Albert's user avatar
  • 163
1 vote
1 answer
184 views

Understanding feature_parallel distributed learning algorithm in LightGBMClassifier

I want to understand feature_parallel algorithm in LightGBMClassifier. It describes how it is done traditionally and how LightGBM...
figs_and_nuts's user avatar
3 votes
3 answers
138 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
Chris_007's user avatar
  • 193
1 vote
1 answer
86 views

Reasons for a model predicting probability of being class 1 at x value

All, This is a general question. I have a binary classification which predicts if someone is rich or not. I had a question from someone asking that if the probability someone is rich is 0.6 and ...
Maths12's user avatar
  • 506
0 votes
1 answer
325 views

Catboost not able to handle a very simple dataset?

This is a post from a newbie and so might be a really poor question based on lack of knowledge. Thank you kindly! I'm using Catboost, which seems excellent, to fit a trivial dataset. The results are ...
user5406764's user avatar
0 votes
1 answer
98 views

Feature importance by random forest and boosting tree when two features are heavy correlated [closed]

I have asked this question here but seems no one is interested in it. Here is my understanding, pls correct me if there is any misunderstanding: Tree models is used ...
user6703592's user avatar
1 vote
1 answer
26 views

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

I have asked this question here, but seems no one was interested in it: https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the ...
user6703592's user avatar
0 votes
1 answer
267 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
Nikita Budilovskiy's user avatar
1 vote
0 answers
37 views

How are regression trees fitted in gradient boosting for classification?

What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds). For the rest of the trees, we try to fit ...
Nikhil Mishra's user avatar
5 votes
1 answer
564 views

Multi-target regression tree with additional constraint

I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$): $$ (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \...
Peter's user avatar
  • 7,366
1 vote
0 answers
14 views

Why is the average prediction moving away from average response for a reg:gamma model

I'm predicting a response that I would typically model under a gamma distribution, with relatively simple paramters, I'm just using the default other than these: learning_rate = 0.01 max_depth = 6 ...
Mattice Verhoeven's user avatar
1 vote
1 answer
2k views

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
ro23's user avatar
  • 35
2 votes
1 answer
123 views

What if root of a such tree is pruned in xgboost?

Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
Davi Américo's user avatar
4 votes
1 answer
4k views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
thereandhere1's user avatar
1 vote
1 answer
463 views

Tree complexity and gamma parameter in xgboost

According to xgboost paper, regularization is given by: $$\Omega(f) = \gamma T + \lambda || w||^2$$ where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree). The parameter ...
zzzbob's user avatar
  • 45
0 votes
1 answer
558 views

Chossing between gradient boosting algorithms

I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,...
nprime496's user avatar
1 vote
1 answer
890 views

What is "Missing" in output of plot_tree API of XGBoost

What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0. I tried searching on ...
Pramod yadav's user avatar
6 votes
1 answer
3k views

What is Pruning & Truncation in Decision Trees?

Pruning & Truncation As per my understanding Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set ...
Pluviophile's user avatar
  • 3,608