Questions tagged [gradient-boosting-decision-trees]

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

Understanding feature_parallel distributed learning algorithm in LightGBMClassifier

I want to understand feature_parallel algorithm in LightGBMClassifier. It describes how it is done traditionally and how LightGBM...
2
votes
1answer
30 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
1
vote
1answer
24 views

Reasons for a model predicting probability of being class 1 at x value

All, This is a general question. I have a binary classification which predicts if someone is rich or not. I had a question from someone asking that if the probability someone is rich is 0.6 and ...
0
votes
0answers
19 views

Understanding Residuals Plots

I have a residuals plot: Definitions: let's call "blue_line" the line that would exist if I were to draw a straight line by fitting to the blue dots (predictions). My expectation is that if ...
0
votes
1answer
30 views

Catboost not able to handle a very simple dataset?

This is a post from a newbie and so might be a really poor question based on lack of knowledge. Thank you kindly! I'm using Catboost, which seems excellent, to fit a trivial dataset. The results are ...
0
votes
1answer
31 views

Feature importance by random forest and boosting tree when two features are heavy correlated [closed]

I have asked this question here but seems no one is interested in it. Here is my understanding, pls correct me if there is any misunderstanding: Tree models is used ...
1
vote
1answer
18 views

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

I have asked this question here, but seems no one was interested in it: https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the ...
0
votes
0answers
23 views

Determining the effect of combinations of independent variables (customer charateristics) on dependent variables (customer value)

I have lots of transactional and demographic (etc.) data about my customers and I want to understand: "What are the characteristics (age, profession etc.) of valuable customers?" To do this ...
0
votes
1answer
33 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
1
vote
0answers
20 views

How are regression trees fitted in gradient boosting for classification?

What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds). For the rest of the trees, we try to fit ...
0
votes
0answers
11 views

Gradient Boosting - Why predicting negative gradients provide better generalizatrion

I am struggling to fully understand why training a tree to predict negative gradient of a loss function provides better generalization than applying steepest gradient which was described in "...
5
votes
1answer
69 views

Multi-target regression tree with additional constraint

I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$): $$ (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \...
1
vote
0answers
13 views

Why is the average prediction moving away from average response for a reg:gamma model

I'm predicting a response that I would typically model under a gamma distribution, with relatively simple paramters, I'm just using the default other than these: learning_rate = 0.01 max_depth = 6 ...
0
votes
0answers
37 views

LightGBM boosting and bagging parameters

When training a gradient boosted decision tree model, I can use the LightGBM package to efficiently train my model. It's possible to define the hyperparameter search space with eg. ...
0
votes
0answers
18 views

Uncertainty prediction in Gradient Boosted Tree based Quantile Regression

For an application, I am using a Gradient boosting Tree based quantile regression model (LightGBM, Catboot) to predict the 5th percentile of the target variable. The model predicts point estimates, ...
1
vote
1answer
100 views

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
2
votes
1answer
23 views

What if root of a such tree is pruned in xgboost?

Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
1
vote
1answer
327 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
1
vote
1answer
63 views

Tree complexity and gamma parameter in xgboost

According to xgboost paper, regularization is given by: $$\Omega(f) = \gamma T + \lambda || w||^2$$ where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree). The parameter ...
0
votes
1answer
107 views

Chossing between gradient boosting algorithms

I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,...
0
votes
1answer
222 views

What is "Missing" in output of plot_tree API of XGBoost

What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0. I tried searching on ...
5
votes
1answer
1k views

What is Pruning & Truncation in Decision Trees?

Pruning & Truncation As per my understanding Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set ...