Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gbm]

Gradient Boosting Machine

2
votes
1answer
32 views

What happens if GBM parameters (e.g., learning rate) vary as the training progresses?

In neural networks there is an idea of a "learning rate schedule" which changes the learning rate as training progresses. This made me ask the question, what would be the impact of varying ...
1
vote
0answers
31 views

How does L1 Regularization work in lightGBM

From the paper, lightGBM does a subsampling according to sorted $|g_i|$, where $g_i$ is the gradient (for the loss function) at a data instance. My question is that, when the objective is L1 loss/...
3
votes
3answers
129 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
2
votes
2answers
203 views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
0
votes
1answer
108 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
1
vote
0answers
31 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
3
votes
1answer
84 views

Representing cyclical features as sin/cos components

I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc. After some searching I decided to follow the advice here. Now ...
1
vote
0answers
32 views

Predicted Values in machine learning [closed]

I remember reading an article comparing predicted probabilities of boosted, stacked encoders, random forests, and neural network algorithms. One important point I got from the article was that gbm ...
0
votes
1answer
37 views

Why is there a big drop off in performance in my GBM?

I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 ...
0
votes
1answer
24 views

How does the number of trees effect the prediction time in gradient boost classification trees?

After tuning hyper-parameters for a gradient boosted model, I have found that the best tree count (iterations) is a few thousand. I'm worried that such a high count might impact prediction ...
3
votes
2answers
3k views

Decision Trees Nodes vs Leaves Definition

I am having a little trouble understanding the difference between what a "Node" of a tree and a "Leaf" of a tree. Suppose I am trying to decide the size of coffee a person may like. There are three ...
0
votes
1answer
536 views

How to find the residuals of a classification tree in xgboost

So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on ...
4
votes
2answers
2k views

Adding feature leads to worse results

I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables. I compare the results of a GBM model (using python xgboost and light GBM) and I ...
2
votes
0answers
292 views

What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?

I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set. What stopping metric can be used to optimize the sensitivity ...
2
votes
1answer
303 views

How much data is needed for a GBM to be more reliable than logistic regression for binary classification?

When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (...
3
votes
1answer
2k views

Residuals in a gradient boosted classification

I know that we iteratively model the residuals in case of a gradient boosted regression problem. The intuition is very well explained at kaggle. Can someone explain what are the residuals that are ...
1
vote
0answers
51 views

Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome

I have a binomial outcome that I am trying to predict using a gbm in h2o. I have set quite a low min_rows value for each node and it appears to be overfitting. See plots below. When I group the ...
4
votes
3answers
3k views

How to determine if my GBM model is overfitting?

Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length. The example yields an r2 value of 0.93, which seems ...
-1
votes
1answer
412 views

gbm function not working

I am working on a classification problem, and am applying gradient boosted tress on the dataset, to classify the items into two classes (Fraud -> 1 and No Fraud -> 0) I am using the below code for my ...
1
vote
0answers
50 views

Injecting random values as one input feature for feature selection results in a odd beaviour

I am trying to find a cutoff value, in the feature importance space to eliminate spurious features. So I am injecting a completely random generated feature (as one of the input features) and I cut the ...
5
votes
1answer
957 views

what is init_score in lightGBM?

in the tutorial of Boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I ...
34
votes
4answers
38k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key difference between GBM and XGBOOST. I tried to google it but could not find any good answer explaining the difference between the two algos and why does xgboost ...
4
votes
2answers
3k views

Xgboost quantile regression via custom objective

This is my first time posting, so please bare with me if I miss giving necessary info... I'm new to GBM and xgboost, and I'm currently using xgboost_0.6-2 in R. The modeling runs well with the ...
2
votes
1answer
442 views

Fit Decision Tree to Gradient Boosted Trees for Interpretability

I was wondering if there is literature on or someone could explain how to fit a decision tree to a gradient boosted trees classifier in order to derive more interpretable results. This is apparently ...
-1
votes
1answer
465 views

Comparing Categorical and Continuous Features using Splits in GBM

In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well ...
20
votes
4answers
10k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
2
votes
1answer
125 views

Searching interactions with RandomForest and/or GBM

I'm trying to explain a count variable and a continious variable > 0 with GLM, using R. In order to improve the quality of the regression, I want to add some interactions that can be useful for the ...
4
votes
1answer
2k views

Gradient Boosted Trees or Neural Networks Using Model Averaging?

I am working on a certain insurance claims related data-set to classify newly acquired customers as either claim or non-claim. The basic problem with the training set is the extremely large imbalance ...
2
votes
1answer
411 views

Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result?

I am working with a large data set (~9M rows with 20+ features). Is it ok to tune via grid search on a fraction of the data (~100k rows) to determine optimal hyperparameters? This is mostly for ...
10
votes
2answers
2k views

Need help understanding xgboost's approximate split points proposal

background: in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective: $$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$ where $g_i,...
29
votes
1answer
19k views

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while <...
3
votes
1answer
2k views

difference between logistic regression and binary logistic regression

In xgboost R package, there are two objectives given with booster gbtree. reg:logistic ...
4
votes
2answers
2k views

Can xgboost (or any other algorithm) give bad results with some bad features?

till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data. In one of my problems, there are around ...
2
votes
0answers
108 views

Why is each successive tree in GBM fit on the negative gradient of the loss function?

Page 359 of Elements Of Statistical Learning 2nd edition says the below. Can someone explain the intuition & simplify it in layman terms? Questions What is the reason/intuition & math ...
8
votes
2answers
11k views

How fit pairwise ranking models in xgBoost?

As far as I know, to train learning to rank models, you need to have three things in the dataset: label or relevance group or query id feature vector For example, the Microsoft Learning to Rank ...
1
vote
0answers
19 views

Convert Out-of-bag (OOB) estimate to quad weighted kappa score

Is there a way to directly calculate an approximate quad weighted kappa measure from an OOB estimate, obtained from a gradient boosting model with subsampling without going through cross validation?
1
vote
0answers
226 views

Unbalanced data fit in gbm

I'm trying to build a model using GBM in r in order to get probability of two classes ( 'yes','no'). My data are unbalanced, and because of this I trained my model using a balanced data(undersampling ...
3
votes
1answer
699 views

What loss function does the 'multinomial' distribution with the gbm package in R use?

All distributions in the gbm package in R are associated with a loss function. For example, when we set distribution = 'binomial'...
6
votes
2answers
159 views

Why isn't dimension sampling used with gradient boosting machines (GBM)?

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used ...
5
votes
2answers
3k views

Kappa near to 60% in unbalanced (1:10) data set

As mentioned before, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples. I have trained a Generalized Boosted Regression model using ...