Questions tagged [gradient-boosting-decision-trees]

Filter by
Sorted by
Tagged with
3 votes
3 answers
254 views

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing). I have ...
user avatar
0 votes
0 answers
12 views

How to Predict Binary Classification problem having less dominant Features

I have following dataset. Total 31 columns including Target. Target column has value of either 1 or 0. This is balanced dataset. All 30 Feature columns also have value of either 1 or 0. All these 30 ...
user avatar
  • 101
0 votes
0 answers
16 views

Prediction intervals is the correct way for upper bound prediction?

I was tasked with a relatively straightforward problem at work: Given an already preprocessed training dataframe X and its corresponding target vector y, find the estimated upper bound in performance ...
user avatar
0 votes
1 answer
47 views

How does Catboost regressor deal with categorical features at predict time?

I understand that Catboost regressor uses target-based encoding to convert categorical features to numerical features when training. But how does Catboost deal with categorical features at predict ...
user avatar
  • 3
0 votes
0 answers
32 views

Is it possible to embed a neural network layer into decision tree/random forest?

I want to do a classification task. I designed a customed layer for it. I also want to try decision tree/random forest, but as far as I know there is no way to embed my layer into a decsion tree/...
user avatar
1 vote
0 answers
27 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
user avatar
  • 36
0 votes
1 answer
11 views

GridSearch multiplying the number of trees in XGboost?

I'm having an issue: after running an XGboost in a HalvingGridSearchCV, I receive a certain number of estimators (50 for example), but the number of trees is constantly being multiplied by 3. I don't ...
user avatar
0 votes
0 answers
33 views

splitting point in LightGBM?

I am not able to understand how the first root node is selected in LightGBM and how the splitting at nodes happens further. I read blogs and related documents and I understand that in this histogram-...
user avatar
  • 27
1 vote
0 answers
18 views

Catboost not working properly when I remove non important variables (source of randomness?)

I was wondering if anyone has encountered the same. The thing is, when I run a catboost boosting model, delete non important variables (feature importance by prediction importance = 0, in fact these ...
user avatar
  • 111
0 votes
0 answers
22 views

Analysis of prediction shift problem in gradient boosting

I was going through the Catboost paper section 4.1 where they talk about the 'Analysis of prediction shift' using an example consisting of 2 features which are bernoulli random variables. I am unable ...
user avatar
1 vote
1 answer
35 views

How is the probability prediction of a binary classifier predicted

I have a trained BDT and with sklearn predict_proba(X), I can get a probability between 0 and 1 for a predicted feature. I am now wondering, how this probability is calculated? Any ideas?
user avatar
  • 129
0 votes
0 answers
60 views

Decision tree from boosted tree regressor Google bigquery ML

If I set the num_parallel tree to 1 and max_iteration to 1 in boosted_tree_regressor of Google Big Query ML will it work as Decision tree regressor ? Also can such decision tree give negative ...
user avatar
1 vote
1 answer
70 views

Random LightGBM Forest

I'm not completly sure about the bias/variance of boosted decision trees (LightGBM especially), thus I wonder if we generally would expect a performance boost by creating an ensemble of multiple ...
user avatar
0 votes
1 answer
59 views

XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values

I thought the consensus was that XGBoost was largely scale-invariant and scaling of features isn't really necessary but something's going wrong and I don't understand what. I have a range of features ...
user avatar
  • 103
1 vote
0 answers
12 views

references on how to use shap values without the shap package

I am familiar with the shap python package and how to use it, I also have a pretty good idea about shap values in general, but it is still new to me. What I'm requesting are references (ideally python ...
user avatar
1 vote
0 answers
13 views

How to compare new model to current production model?

Given new data, I trained the same model architecture and same hyperparameters (for example a random forest) as the current production model. How do I know that the new model that I trained is better ...
user avatar
0 votes
0 answers
32 views

How to have absolute importance of predictor variables in boosted regression trees (BRT) model?

Thank you very much for your time. I would like to compare the contribution of predictor variables (X1) from different BRT (Boosted regression trees) models, which are from different spatial scales (...
user avatar
0 votes
1 answer
59 views

Low accuracy on the test set

I have a dataset with 16 features and 32 class labels, which shows the following behavior: Neural network classification: high accuracy on train 100%, but low accuracy on the test set 3% (almost like ...
user avatar
  • 163
1 vote
1 answer
56 views

Understanding feature_parallel distributed learning algorithm in LightGBMClassifier

I want to understand feature_parallel algorithm in LightGBMClassifier. It describes how it is done traditionally and how LightGBM...
user avatar
2 votes
3 answers
59 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
user avatar
  • 173
1 vote
1 answer
28 views

Reasons for a model predicting probability of being class 1 at x value

All, This is a general question. I have a binary classification which predicts if someone is rich or not. I had a question from someone asking that if the probability someone is rich is 0.6 and ...
user avatar
  • 466
0 votes
0 answers
20 views

Understanding Residuals Plots

I have a residuals plot: Definitions: let's call "blue_line" the line that would exist if I were to draw a straight line by fitting to the blue dots (predictions). My expectation is that if ...
user avatar
0 votes
1 answer
45 views

Catboost not able to handle a very simple dataset?

This is a post from a newbie and so might be a really poor question based on lack of knowledge. Thank you kindly! I'm using Catboost, which seems excellent, to fit a trivial dataset. The results are ...
user avatar
0 votes
1 answer
43 views

Feature importance by random forest and boosting tree when two features are heavy correlated [closed]

I have asked this question here but seems no one is interested in it. Here is my understanding, pls correct me if there is any misunderstanding: Tree models is used ...
user avatar
1 vote
1 answer
21 views

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

I have asked this question here, but seems no one was interested in it: https://stats.stackexchange.com/questions/550994/if-a-feature-has-already-split-will-it-hardly-be-selected-to-split-again-in-the ...
user avatar
0 votes
0 answers
24 views

Determining the effect of combinations of independent variables (customer charateristics) on dependent variables (customer value)

I have lots of transactional and demographic (etc.) data about my customers and I want to understand: "What are the characteristics (age, profession etc.) of valuable customers?" To do this ...
user avatar
0 votes
1 answer
42 views

Steps of multiclass classification problem

So this question is more theoretical, than a practical one. I got a dataframe with 4 classes of cars' body types (e.g. sedan, hatchback, etc.) and different characteristics (doors, seats, maximum ...
user avatar
1 vote
0 answers
21 views

How are regression trees fitted in gradient boosting for classification?

What I understood is that even gradient boosting for binary classification uses regression trees. The first value we calculate is constant = log(odds). For the rest of the trees, we try to fit ...
user avatar
0 votes
0 answers
16 views

Gradient Boosting - Why predicting negative gradients provide better generalizatrion

I am struggling to fully understand why training a tree to predict negative gradient of a loss function provides better generalization than applying steepest gradient which was described in "...
user avatar
5 votes
1 answer
179 views

Multi-target regression tree with additional constraint

I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$): $$ (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \...
user avatar
  • 6,912
1 vote
0 answers
13 views

Why is the average prediction moving away from average response for a reg:gamma model

I'm predicting a response that I would typically model under a gamma distribution, with relatively simple paramters, I'm just using the default other than these: learning_rate = 0.01 max_depth = 6 ...
user avatar
1 vote
1 answer
375 views

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

I trained multiple models for my problem and most ensemble algorithms resulted in lengthy fit and train time and huge model size on disk (approx 10GB for RandomForest) but when I tried ...
user avatar
  • 35
2 votes
1 answer
54 views

What if root of a such tree is pruned in xgboost?

Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
user avatar
1 vote
1 answer
843 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
user avatar
1 vote
1 answer
148 views

Tree complexity and gamma parameter in xgboost

According to xgboost paper, regularization is given by: $$\Omega(f) = \gamma T + \lambda || w||^2$$ where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree). The parameter ...
user avatar
  • 45
0 votes
1 answer
248 views

Chossing between gradient boosting algorithms

I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,...
user avatar
0 votes
1 answer
418 views

What is "Missing" in output of plot_tree API of XGBoost

What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0. I tried searching on ...
user avatar
5 votes
1 answer
1k views

What is Pruning & Truncation in Decision Trees?

Pruning & Truncation As per my understanding Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set ...
user avatar
  • 2,376