Questions tagged [gradient-boosting-decision-trees]
The gradient-boosting-decision-trees tag has no usage guidance.
47
questions
0
votes
0
answers
44
views
Missing values handling in LightGBM
I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific.
In a regression setting with no categorical feature, I have ...
- 101
0
votes
0
answers
6
views
Histogram creation in lightgbm in the train API and the scikit-learn API. Is it always benefitial to use the train API?
In the LightGBM for python we have a scikit-learn API in which (either for regression or for classification) there is fit method whose documentation is
fit(X, y, sample_weight=None, init_score=None, ...
- 821
1
vote
0
answers
34
views
0
votes
1
answer
21
views
How to pass a Dataframe as train dataframe and another dataframe as Validation to GridSearchCV
I'm a programmer who tries to find he's way into ML world. so the Question might be basic.
i have data from years 2010-2019. Now i'm trying to test different parameters on gradient boosting regression ...
- 101
0
votes
0
answers
55
views
XGBoost Architecture Diagram required
Good Day!
My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
- 1
0
votes
1
answer
88
views
How does LGBM make a prediction?
We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards.
In my current understanding, it works as follows:
Multiple "weak learners" are ...
- 121
0
votes
0
answers
23
views
Feature scales and feature importance
Tree-based algorithms do not require feature scaling before fitting, and
I am working on gradient boosted tree models (and random forest) without scaling features.
I'm curious if feature scaling ...
- 101
0
votes
1
answer
124
views
randomness in lightgbm model training
What are the parameters that add randomness to the training of a lightgbm model? (for a large dataset) I have tried setting all parameters as default and letting bin_construct_sample_cnt be greater ...
- 1
0
votes
1
answer
62
views
How to reduce the false positives to improve the models performance?
I am currently building a binary classification model to predict order return rates. I used the GradientBoostingClassifier for training the model and also performed hyperparameter tuning using ...
0
votes
1
answer
62
views
DART algorithm implementation. Converting mathematical notation to pseudocode
I am learning how DART algorithm (https://arxiv.org/abs/1505.01866) works and I want to implement it in C#
I have the algorithm's description in mathematical notation and I don't understand most of it....
0
votes
0
answers
111
views
Lightgbm model tuning produces unexpectedly different hyperparameters for similar datasets
I am trying to tune a lightgbm model for each half of a dataset that I have split by a particular feature (stock ticker in this case). Both halves have the same number of features, somewhat similar ...
1
vote
1
answer
240
views
Why is monotonic constraint disabled when using MAE as an objective to LGBM?
I tried to use monotonic constraints in LGBM, but if I use mean absolute error as an objective, it gives a warning that monotonic constraints cannot be done in l1.
What is the reason? Thanks!
- 25
1
vote
1
answer
374
views
Why is gradient boosting better than random forest for unbalanced data?
I've searched everywhere and still couldn't figure this one out.
This post mentioned that Gradient Boosting is better than Random Forest for unbalanced data. Why is that? Is Random Forest worse ...
0
votes
1
answer
147
views
Impact of many zeros in LightGBM Regressor training set [duplicate]
I have a LightGBM Regressor model with 15 features. 5 of these features have 98.7% NA for the training set. All five of the features are NA for each row. I impute the missing values with zero before I ...
- 1
1
vote
1
answer
23
views
How does machine learning algorithms process text?
I'm still new in machine learning and I've been trying to expand my knowledge about it. For my first project, I want to classify if a tweet is suicidal or not using the gradient boost algorithm.
I do ...
- 11
0
votes
2
answers
40
views
If we train a binary classifier (lets say tree based) to predict ordinal data do they learn to interpolate?
Let's assume we have data about students in grade 10. We have test scores ranging from 0-100, however we are only provided two labels ; High score = if the score> 80% and low score if the score <...
- 3
1
vote
0
answers
28
views
ML Model that doesnt average/penalize extreme values
I have a 20k dataset, and a couple hundred of those lines are extreme values and 10 of them or so are even extremer values. But they are correct and have a unique tag, so when that tag comes up I am ...
- 11
0
votes
1
answer
38
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 137
3
votes
3
answers
482
views
Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?
I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing).
I have ...
- 33
0
votes
1
answer
795
views
How does Catboost regressor deal with categorical features at predict time?
I understand that Catboost regressor uses target-based encoding to convert categorical features to numerical features when training. But how does Catboost deal with categorical features at predict ...
- 3
0
votes
0
answers
58
views
Is it possible to embed a neural network layer into decision tree/random forest?
I want to do a classification task. I designed a customed layer for it. I also want to try decision tree/random forest, but as far as I know there is no way to embed my layer into a decsion tree/...
- 165
1
vote
0
answers
91
views
Error from XGBoost missing data handling
I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
- 36
0
votes
1
answer
33
views
GridSearch multiplying the number of trees in XGboost?
I'm having an issue: after running an XGboost in a HalvingGridSearchCV, I receive a certain number of estimators (50 for example), but the number of trees is constantly being multiplied by 3. I don't ...
- 21
1
vote
0
answers
83
views
Catboost not working properly when I remove non important variables (source of randomness?)
I was wondering if anyone has encountered the same. The thing is, when I run a catboost boosting model, delete non important variables (feature importance by prediction importance = 0, in fact these ...
- 63
1
vote
1
answer
42
views
How is the probability prediction of a binary classifier predicted
I have a trained BDT and with sklearn predict_proba(X), I can get a probability between 0 and 1 for a predicted feature. I am now wondering, how this probability is calculated?
Any ideas?
- 129
1
vote
1
answer
245
views
Random LightGBM Forest
I'm not completly sure about the bias/variance of boosted decision trees (LightGBM especially), thus I wonder if we generally would expect a performance boost by creating an ensemble of multiple ...
- 460
0
votes
1
answer
263
views
XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values
I thought the consensus was that XGBoost was largely scale-invariant and scaling of features isn't really necessary but something's going wrong and I don't understand what.
I have a range of features ...
- 103
1
vote
0
answers
73
views
references on how to use shap values without the shap package
I am familiar with the shap python package and how to use it, I also have a pretty good idea about shap values in general, but it is still new to me. What I'm requesting are references (ideally python ...
- 111
1
vote
0
answers
18
views
How to compare new model to current production model?
Given new data, I trained the same model architecture and same hyperparameters (for example a random forest) as the current production model. How do I know that the new model that I trained is better ...
- 111
0
votes
1
answer
313
views
Low accuracy on the test set
I have a dataset with 16 features and 32 class labels, which shows the following behavior:
Neural network classification: high accuracy on train 100%, but low accuracy on the test set 3% (almost like ...
- 163
1
vote
1
answer
184
views
Understanding feature_parallel distributed learning algorithm in LightGBMClassifier
I want to understand feature_parallel algorithm in LightGBMClassifier.
It describes how it is done traditionally and how LightGBM...
- 821
3
votes
3
answers
138
views
Example for Boosting
Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
- 193
1
vote
1
answer
86
views
Reasons for a model predicting probability of being class 1 at x value
All,
This is a general question. I have a binary classification which predicts if someone is rich or not. I had a question from someone asking that if the probability someone is rich is 0.6 and ...
- 506
0
votes
1
answer
325
views
Catboost not able to handle a very simple dataset?
This is a post from a newbie and so might be a really poor question based on lack of knowledge. Thank you kindly!
I'm using Catboost, which seems excellent, to fit a trivial dataset. The results are ...
- 127
0
votes
1
answer
98
views
Feature importance by random forest and boosting tree when two features are heavy correlated [closed]
I have asked this question here but seems no one is interested in it.
Here is my understanding, pls correct me if there is any misunderstanding:
Tree models is used ...
- 127
1
vote
1
answer
26
views
If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree
I have asked this question here, but seems no one was interested in it:
https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the
...
- 127
0
votes
1
answer
267
views
Steps of multiclass classification problem
So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
1
vote
0
answers
37
views
How are regression trees fitted in gradient boosting for classification?
What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds).
For the rest of the trees, we try to fit ...
- 111
5
votes
1
answer
564
views
Multi-target regression tree with additional constraint
I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$):
$$ (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \...
- 7,366
1
vote
0
answers
14
views
Why is the average prediction moving away from average response for a reg:gamma model
I'm predicting a response that I would typically model under a gamma distribution, with relatively simple paramters, I'm just using the default other than these:
learning_rate = 0.01
max_depth = 6
...
1
vote
1
answer
2k
views
Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?
I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
- 35
2
votes
1
answer
123
views
What if root of a such tree is pruned in xgboost?
Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
- 123
4
votes
1
answer
4k
views
XGBoost - Imputing Vs keeping NaN
What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
- 735
1
vote
1
answer
463
views
Tree complexity and gamma parameter in xgboost
According to xgboost paper, regularization is given by:
$$\Omega(f) = \gamma T + \lambda || w||^2$$
where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree).
The parameter ...
- 45
0
votes
1
answer
558
views
Chossing between gradient boosting algorithms
I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,...
- 65
1
vote
1
answer
890
views
What is "Missing" in output of plot_tree API of XGBoost
What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0.
I tried searching on ...
- 11
6
votes
1
answer
3k
views
What is Pruning & Truncation in Decision Trees?
Pruning & Truncation
As per my understanding
Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set ...
- 3,608