Questions tagged [gradient-boosting-decision-trees]
The gradient-boosting-decision-trees tag has no usage guidance.
46
questions
0
votes
0
answers
17
views
Lightgbm model tuning produces unexpectedly different hyperparameters for similar datasets
I am trying to tune a lightgbm model for each half of a dataset that I have split by a particular feature (stock ticker in this case). Both halves have the same number of features, somewhat similar ...
1
vote
1
answer
40
views
Why is monotonic constraint disabled when using MAE as an objective to LGBM?
I tried to use monotonic constraints in LGBM, but if I use mean absolute error as an objective, it gives a warning that monotonic constraints cannot be done in l1.
What is the reason? Thanks!
- 25
0
votes
0
answers
55
views
XGBoost Gradient Boosting and Gradient Descent confusion
I am trying to understand how the XGBoost determines the next tree. Some sources state that the model uses gradient descent to find the optimal option:
This answer from a question on this s.e. also ...
- 1
0
votes
0
answers
16
views
compare decision tree vs extended gradient boosting mathematically?
If we want to compare decision tree vs extended gradient boosting vs xgboost mathematically, what are their differences?
- 1,754
1
vote
1
answer
86
views
Why is gradient boosting better than random forest for unbalanced data?
I've searched everywhere and still couldn't figure this one out.
This post mentioned that Gradient Boosting is better than Random Forest for unbalanced data. Why is that? Is Random Forest worse ...
0
votes
0
answers
37
views
is time series data normalization for xgboost required?
According to the developer - xgboost does not require feature normalization https://github.com/dmlc/xgboost/issues/357
no you do not have to normalize the ...
- 101
0
votes
1
answer
38
views
Impact of many zeros in LightGBM Regressor training set
I have a LightGBM Regressor model with 15 features. 5 of these features have 98.7% NA for the training set. All five of the features are NA for each row. I impute the missing values with zero before I ...
- 1
1
vote
1
answer
20
views
How does machine learning algorithms process text?
I'm still new in machine learning and I've been trying to expand my knowledge about it. For my first project, I want to classify if a tweet is suicidal or not using the gradient boost algorithm.
I do ...
- 11
0
votes
2
answers
32
views
If we train a binary classifier (lets say tree based) to predict ordinal data do they learn to interpolate?
Let's assume we have data about students in grade 10. We have test scores ranging from 0-100, however we are only provided two labels ; High score = if the score> 80% and low score if the score <...
- 3
1
vote
0
answers
25
views
ML Model that doesnt average/penalize extreme values
I have a 20k dataset, and a couple hundred of those lines are extreme values and 10 of them or so are even extremer values. But they are correct and have a unique tag, so when that tag comes up I am ...
- 11
0
votes
0
answers
52
views
Gradient tree boosting regularization term parameters
In the XGBoost documentation, they initially define the regularized objective as
\begin{equation}
\begin{split}
L & = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i) + \sum\limits_{i=1}^K \Omega(f_k)\\...
- 137
0
votes
1
answer
22
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 137
3
votes
3
answers
366
views
Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?
I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing).
I have ...
- 33
0
votes
0
answers
19
views
How to Predict Binary Classification problem having less dominant Features
I have following dataset.
Total 31 columns including Target.
Target column has value of either 1 or 0.
This is balanced dataset.
All 30 Feature columns also have value of either 1 or 0.
All these 30 ...
- 101
0
votes
0
answers
18
views
Prediction intervals is the correct way for upper bound prediction?
I was tasked with a relatively straightforward problem at work:
Given an already preprocessed training dataframe X and its corresponding
target vector y, find the estimated upper bound in performance ...
- 1
0
votes
1
answer
324
views
How does Catboost regressor deal with categorical features at predict time?
I understand that Catboost regressor uses target-based encoding to convert categorical features to numerical features when training. But how does Catboost deal with categorical features at predict ...
- 3
0
votes
0
answers
39
views
Is it possible to embed a neural network layer into decision tree/random forest?
I want to do a classification task. I designed a customed layer for it. I also want to try decision tree/random forest, but as far as I know there is no way to embed my layer into a decsion tree/...
- 165
1
vote
0
answers
59
views
Error from XGBoost missing data handling
I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
- 36
0
votes
1
answer
16
views
GridSearch multiplying the number of trees in XGboost?
I'm having an issue: after running an XGboost in a HalvingGridSearchCV, I receive a certain number of estimators (50 for example), but the number of trees is constantly being multiplied by 3. I don't ...
- 11
0
votes
0
answers
129
views
splitting point in LightGBM?
I am not able to understand how the first root node is selected in LightGBM and how the splitting at nodes happens further. I read blogs and related documents and I understand that in this histogram-...
- 47
1
vote
0
answers
27
views
Catboost not working properly when I remove non important variables (source of randomness?)
I was wondering if anyone has encountered the same. The thing is, when I run a catboost boosting model, delete non important variables (feature importance by prediction importance = 0, in fact these ...
- 53
0
votes
0
answers
81
views
Analysis of prediction shift problem in gradient boosting
I was going through the Catboost paper section 4.1 where they talk about the 'Analysis of prediction shift' using an example consisting of 2 features which are bernoulli random variables. I am unable ...
1
vote
1
answer
36
views
How is the probability prediction of a binary classifier predicted
I have a trained BDT and with sklearn predict_proba(X), I can get a probability between 0 and 1 for a predicted feature. I am now wondering, how this probability is calculated?
Any ideas?
- 129
0
votes
0
answers
153
views
Decision tree from boosted tree regressor Google bigquery ML
If I set the num_parallel tree to 1 and max_iteration to 1 in boosted_tree_regressor of Google Big Query ML will it work as Decision tree regressor ?
Also can such decision tree give negative ...
- 379
1
vote
1
answer
154
views
Random LightGBM Forest
I'm not completly sure about the bias/variance of boosted decision trees (LightGBM especially), thus I wonder if we generally would expect a performance boost by creating an ensemble of multiple ...
- 440
0
votes
1
answer
144
views
XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values
I thought the consensus was that XGBoost was largely scale-invariant and scaling of features isn't really necessary but something's going wrong and I don't understand what.
I have a range of features ...
- 103
1
vote
0
answers
27
views
references on how to use shap values without the shap package
I am familiar with the shap python package and how to use it, I also have a pretty good idea about shap values in general, but it is still new to me. What I'm requesting are references (ideally python ...
1
vote
0
answers
14
views
How to compare new model to current production model?
Given new data, I trained the same model architecture and same hyperparameters (for example a random forest) as the current production model. How do I know that the new model that I trained is better ...
- 111
0
votes
1
answer
239
views
Low accuracy on the test set
I have a dataset with 16 features and 32 class labels, which shows the following behavior:
Neural network classification: high accuracy on train 100%, but low accuracy on the test set 3% (almost like ...
- 163
1
vote
1
answer
113
views
Understanding feature_parallel distributed learning algorithm in LightGBMClassifier
I want to understand feature_parallel algorithm in LightGBMClassifier.
It describes how it is done traditionally and how LightGBM...
- 775
3
votes
3
answers
92
views
Example for Boosting
Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
- 193
1
vote
1
answer
39
views
Reasons for a model predicting probability of being class 1 at x value
All,
This is a general question. I have a binary classification which predicts if someone is rich or not. I had a question from someone asking that if the probability someone is rich is 0.6 and ...
- 486
0
votes
1
answer
182
views
Catboost not able to handle a very simple dataset?
This is a post from a newbie and so might be a really poor question based on lack of knowledge. Thank you kindly!
I'm using Catboost, which seems excellent, to fit a trivial dataset. The results are ...
- 127
0
votes
1
answer
65
views
Feature importance by random forest and boosting tree when two features are heavy correlated [closed]
I have asked this question here but seems no one is interested in it.
Here is my understanding, pls correct me if there is any misunderstanding:
Tree models is used ...
- 127
1
vote
1
answer
22
views
If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree
I have asked this question here, but seems no one was interested in it:
https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the
...
- 127
0
votes
1
answer
81
views
Steps of multiclass classification problem
So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
1
vote
0
answers
26
views
How are regression trees fitted in gradient boosting for classification?
What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds).
For the rest of the trees, we try to fit ...
- 111
5
votes
1
answer
391
views
Multi-target regression tree with additional constraint
I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$):
$$ (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \...
- 7,182
1
vote
0
answers
14
views
Why is the average prediction moving away from average response for a reg:gamma model
I'm predicting a response that I would typically model under a gamma distribution, with relatively simple paramters, I'm just using the default other than these:
learning_rate = 0.01
max_depth = 6
...
1
vote
1
answer
909
views
Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?
I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
- 35
2
votes
1
answer
86
views
What if root of a such tree is pruned in xgboost?
Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
- 123
4
votes
1
answer
2k
views
XGBoost - Imputing Vs keeping NaN
What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
- 683
1
vote
1
answer
260
views
Tree complexity and gamma parameter in xgboost
According to xgboost paper, regularization is given by:
$$\Omega(f) = \gamma T + \lambda || w||^2$$
where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree).
The parameter ...
- 45
0
votes
1
answer
383
views
Chossing between gradient boosting algorithms
I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,...
- 65
0
votes
1
answer
643
views
What is "Missing" in output of plot_tree API of XGBoost
What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0.
I tried searching on ...
6
votes
1
answer
2k
views
What is Pruning & Truncation in Decision Trees?
Pruning & Truncation
As per my understanding
Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set ...
- 2,846