Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gbm]

Gradient Boosting Machine

0
votes
1answer
26 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
0
votes
0answers
16 views

Confused about Output of lightGBM.cv

Do you know what the output of lgb.cv means? From my understanding, if we do 5 fold cross validation, there should be 5 iteration history, including the model metric on both train and validation. ...
1
vote
0answers
27 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
3
votes
1answer
66 views

Representing cyclical features as sin/cos components

I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc. After some searching I decided to follow the advice here. Now ...
0
votes
0answers
72 views

Dropping one of the one-hot encoded columns for Gradient Boost Methods/Decision Trees?

If I have the categorical variable like favorite_color and it has unique values red, green, ...
1
vote
0answers
29 views

Predicted Values in machine learning [closed]

I remember reading an article comparing predicted probabilities of boosted, stacked encoders, random forests, and neural network algorithms. One important point I got from the article was that gbm ...
0
votes
1answer
32 views

Why is there a big drop off in performance in my GBM?

I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 ...
0
votes
1answer
22 views

How does the number of trees effect the prediction time in gradient boost classification trees?

After tuning hyper-parameters for a gradient boosted model, I have found that the best tree count (iterations) is a few thousand. I'm worried that such a high count might impact prediction ...
1
vote
2answers
922 views

Decision Trees Nodes vs Leaves Definition

I am having a little trouble understanding the difference between what a "Node" of a tree and a "Leaf" of a tree. Suppose I am trying to decide the size of coffee a person may like. There are three ...
0
votes
0answers
85 views

Log-odds transformation in gbm binary classification

This question relates to fitting a model for a binary classification problem. The chosen approach is as follows: Structure of model: decision trees, ensemble (built/optimised using gradient boosting ...
0
votes
0answers
285 views

XGBoost vs Gradient Tree Boosting

In Gradient Tree Boosting algorithm we fit a regression tree to the negative gradient and then for each terminal node of the tree we use a single Newton-Raphson step by utilizing the second derivative ...
0
votes
1answer
307 views

How to find the residuals of a classification tree in xgboost

So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on ...
3
votes
2answers
744 views

Adding feature leads to worse results

I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables. I compare the results of a GBM model (using python xgboost and light GBM) and I ...
2
votes
0answers
201 views

What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?

I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set. What stopping metric can be used to optimize the sensitivity ...
1
vote
1answer
201 views

How much data is needed for a GBM to be more reliable than logistic regression for binary classification?

When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (...
3
votes
1answer
1k views

Residuals in a gradient boosted classification

I know that we iteratively model the residuals in case of a gradient boosted regression problem. The intuition is very well explained at kaggle. Can someone explain what are the residuals that are ...
1
vote
0answers
42 views

Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome

I have a binomial outcome that I am trying to predict using a gbm in h2o. I have set quite a low min_rows value for each node and it appears to be overfitting. See plots below. When I group the ...
4
votes
3answers
2k views

How to determine if my GBM model is overfitting?

Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length. The example yields an r2 value of 0.93, which seems ...
-1
votes
1answer
311 views

gbm function not working

I am working on a classification problem, and am applying gradient boosted tress on the dataset, to classify the items into two classes (Fraud -> 1 and No Fraud -> 0) I am using the below code for my ...
1
vote
0answers
37 views

Injecting random values as one input feature for feature selection results in a odd beaviour

I am trying to find a cutoff value, in the feature importance space to eliminate spurious features. So I am injecting a completely random generated feature (as one of the input features) and I cut the ...
4
votes
1answer
673 views

what is init_score in lightGBM?

in the tutorial of Boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I ...
30
votes
4answers
30k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key difference between GBM and XGBOOST. I tried to google it but could not find any good answer explaining the difference between the two algos and why does xgboost ...
4
votes
2answers
2k views

Xgboost quantile regression via custom objective

This is my first time posting, so please bare with me if I miss giving necessary info... I'm new to GBM and xgboost, and I'm currently using xgboost_0.6-2 in R. The modeling runs well with the ...
2
votes
1answer
413 views

Fit Decision Tree to Gradient Boosted Trees for Interpretability

I was wondering if there is literature on or someone could explain how to fit a decision tree to a gradient boosted trees classifier in order to derive more interpretable results. This is apparently ...
-1
votes
1answer
393 views

Comparing Categorical and Continuous Features using Splits in GBM

In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well ...
18
votes
4answers
7k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
2
votes
1answer
113 views

Searching interactions with RandomForest and/or GBM

I'm trying to explain a count variable and a continious variable > 0 with GLM, using R. In order to improve the quality of the regression, I want to add some interactions that can be useful for the ...
4
votes
1answer
1k views

Gradient Boosted Trees or Neural Networks Using Model Averaging?

I am working on a certain insurance claims related data-set to classify newly acquired customers as either claim or non-claim. The basic problem with the training set is the extremely large imbalance ...
2
votes
1answer
330 views

Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result?

I am working with a large data set (~9M rows with 20+ features). Is it ok to tune via grid search on a fraction of the data (~100k rows) to determine optimal hyperparameters? This is mostly for ...
8
votes
2answers
2k views

Need help understanding xgboost's approximate split points proposal

background: in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective: $$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$ where $g_i,...
22
votes
1answer
17k views

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while <...
3
votes
1answer
1k views

difference between logistic regression and binary logistic regression

In xgboost R package, there are two objectives given with booster gbtree. reg:logistic ...
4
votes
2answers
2k views

Can xgboost (or any other algorithm) give bad results with some bad features?

till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data. In one of my problems, there are around ...
2
votes
0answers
97 views

Why is each successive tree in GBM fit on the negative gradient of the loss function?

Page 359 of Elements Of Statistical Learning 2nd edition says the below. Can someone explain the intuition & simplify it in layman terms? Questions What is the reason/intuition & math ...
7
votes
2answers
9k views

How fit pairwise ranking models in xgBoost?

As far as I know, to train learning to rank models, you need to have three things in the dataset: label or relevance group or query id feature vector For example, the Microsoft Learning to Rank ...
1
vote
0answers
18 views

Convert Out-of-bag (OOB) estimate to quad weighted kappa score

Is there a way to directly calculate an approximate quad weighted kappa measure from an OOB estimate, obtained from a gradient boosting model with subsampling without going through cross validation?
1
vote
0answers
176 views

Unbalanced data fit in gbm

I'm trying to build a model using GBM in r in order to get probability of two classes ( 'yes','no'). My data are unbalanced, and because of this I trained my model using a balanced data(undersampling ...
3
votes
1answer
649 views

What loss function does the 'multinomial' distribution with the gbm package in R use?

All distributions in the gbm package in R are associated with a loss function. For example, when we set distribution = 'binomial'...
6
votes
2answers
150 views

Why isn't dimension sampling used with gradient boosting machines (GBM)?

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used ...
4
votes
2answers
2k views

Kappa near to 60% in unbalanced (1:10) data set

As mentioned before, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples. I have trained a Generalized Boosted Regression model using ...