Questions tagged [gbm]

Gradient Boosting Machine

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

Is it possible to create nonbinary trees in XGBoost?

I'm looking through the documentation for XGBoost, and I'm not seeing any parameters relating to number of branches per node.
0
votes
0answers
10 views

How to use interaction variables and indicator variable with GBM model in R?

I am new to Machine Learning and i have been working on a classification model which predicts donor is available or not. There is a variable which specifies whether user registered online voluntarily ...
0
votes
1answer
42 views

GBM: small change in the trainset causes radical change in predictions

I have build a model using transactions data trying to predict the value of future transactions. The main algorithm is Gradient Boosting Machine. The overall accuracy on the testset is fine and there ...
1
vote
0answers
28 views

using word embedding features with linear prediction models

I have been seeing that word embedding features (e.g. here or there) are used on classification or regression tasks where the classifier/regressor is a linear one: e.g. Linear/Logistic Regressor or ...
1
vote
0answers
156 views

Bayesian optimization for a Light GBM Model

I am able to successfully improve the performance of my XGBoost model through Bayesian optimization, but the best I can achieve through Bayesian optimization when using Light GBM (my preferred choice) ...
4
votes
1answer
192 views

What happens if GBM parameters (e.g., learning rate) vary as the training progresses?

In neural networks there is an idea of a "learning rate schedule" which changes the learning rate as training progresses. This made me ask the question, what would be the impact of varying ...
1
vote
0answers
92 views

How does L1 Loss work in lightGBM

From the paper, lightGBM does a subsampling according to sorted $|g_i|$, where $g_i$ is the gradient (for the loss function) at a data instance. My question is that, when the objective is L1 loss/...
4
votes
3answers
208 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
4
votes
2answers
384 views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
0
votes
1answer
205 views

Target feature in training set or not?

If I analyse a random forest in python with scikit I do: ...
1
vote
0answers
32 views

Is this random forest logical correct and correct implemented with R and gbm?

For professional reasons I want to learn and understand random forests. I feel unsafe if my understanding is the correct or if I am doing logical errors. I got a data set with 15 million entries and ...
3
votes
1answer
123 views

Representing cyclical features as sin/cos components

I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc. After some searching I decided to follow the advice here. Now ...
1
vote
0answers
32 views

Predicted Values in machine learning [closed]

I remember reading an article comparing predicted probabilities of boosted, stacked encoders, random forests, and neural network algorithms. One important point I got from the article was that gbm ...
0
votes
1answer
50 views

Why is there a big drop off in performance in my GBM?

I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 ...
0
votes
1answer
28 views

How does the number of trees effect the prediction time in gradient boost classification trees?

After tuning hyper-parameters for a gradient boosted model, I have found that the best tree count (iterations) is a few thousand. I'm worried that such a high count might impact prediction ...
3
votes
2answers
4k views

Decision Trees Nodes vs Leaves Definition

I am having a little trouble understanding the difference between what a "Node" of a tree and a "Leaf" of a tree. Suppose I am trying to decide the size of coffee a person may like. There are three ...
0
votes
1answer
738 views

How to find the residuals of a classification tree in xgboost

So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on ...
3
votes
2answers
3k views

Adding feature leads to worse results

I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables. I compare the results of a GBM model (using python xgboost and light GBM) and I ...
2
votes
0answers
357 views

What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?

I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set. What stopping metric can be used to optimize the sensitivity ...
2
votes
1answer
383 views

How much data is needed for a GBM to be more reliable than logistic regression for binary classification?

When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (...
3
votes
1answer
2k views

Residuals in a gradient boosted classification

I know that we iteratively model the residuals in case of a gradient boosted regression problem. The intuition is very well explained at kaggle. Can someone explain what are the residuals that are ...
5
votes
0answers
887 views

AdaBoost implementation and tuning for high dimensional feature space in R

I am trying to implement the AdaBoost.M1 algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in R. ...
1
vote
0answers
56 views

Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome

I have a binomial outcome that I am trying to predict using a gbm in h2o. I have set quite a low min_rows value for each node and it appears to be overfitting. See plots below. When I group the ...
4
votes
3answers
4k views

How to determine if my GBM model is overfitting?

Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length. The example yields an r2 value of 0.93, which seems ...
-1
votes
1answer
465 views

gbm function not working

I am working on a classification problem, and am applying gradient boosted tress on the dataset, to classify the items into two classes (Fraud -> 1 and No Fraud -> 0) I am using the below code for my ...
1
vote
0answers
58 views

Injecting random values as one input feature for feature selection results in a odd beaviour

I am trying to find a cutoff value, in the feature importance space to eliminate spurious features. So I am injecting a completely random generated feature (as one of the input features) and I cut the ...
4
votes
1answer
1k views

what is init_score in lightGBM?

in the tutorial of Boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I ...
37
votes
4answers
45k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost ...
4
votes
2answers
3k views

Xgboost quantile regression via custom objective

This is my first time posting, so please bare with me if I miss giving necessary info... I'm new to GBM and xgboost, and I'm currently using xgboost_0.6-2 in R. The modeling runs well with the ...
2
votes
1answer
460 views

Fit Decision Tree to Gradient Boosted Trees for Interpretability

I was wondering if there is literature on or someone could explain how to fit a decision tree to a gradient boosted trees classifier in order to derive more interpretable results. This is apparently ...
-1
votes
1answer
492 views

Comparing Categorical and Continuous Features using Splits in GBM

In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well ...
20
votes
4answers
13k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
2
votes
1answer
130 views

Searching interactions with RandomForest and/or GBM

I'm trying to explain a count variable and a continious variable > 0 with GLM, using R. In order to improve the quality of the regression, I want to add some interactions that can be useful for the ...
3
votes
1answer
2k views

Gradient Boosted Trees or Neural Networks Using Model Averaging?

I am working on a certain insurance claims related data-set to classify newly acquired customers as either claim or non-claim. The basic problem with the training set is the extremely large imbalance ...
2
votes
1answer
438 views

Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result?

I am working with a large data set (~9M rows with 20+ features). Is it ok to tune via grid search on a fraction of the data (~100k rows) to determine optimal hyperparameters? This is mostly for ...
11
votes
3answers
3k views

Need help understanding xgboost's approximate split points proposal

background: in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective: $$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$ where $g_i,...
29
votes
1answer
20k views

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while <...
3
votes
1answer
2k views

difference between logistic regression and binary logistic regression

In xgboost R package, there are two objectives given with booster gbtree. reg:logistic ...
4
votes
2answers
2k views

Can xgboost (or any other algorithm) give bad results with some bad features?

till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data. In one of my problems, there are around ...
2
votes
0answers
113 views

Why is each successive tree in GBM fit on the negative gradient of the loss function?

Page 359 of Elements Of Statistical Learning 2nd edition says the below. Can someone explain the intuition & simplify it in layman terms? Questions What is the reason/intuition & math ...
10
votes
2answers
12k views

How fit pairwise ranking models in xgBoost?

As far as I know, to train learning to rank models, you need to have three things in the dataset: label or relevance group or query id feature vector For example, the Microsoft Learning to Rank ...
1
vote
0answers
20 views

Convert Out-of-bag (OOB) estimate to quad weighted kappa score

Is there a way to directly calculate an approximate quad weighted kappa measure from an OOB estimate, obtained from a gradient boosting model with subsampling without going through cross validation?
1
vote
0answers
256 views

Unbalanced data fit in gbm

I'm trying to build a model using GBM in r in order to get probability of two classes ( 'yes','no'). My data are unbalanced, and because of this I trained my model using a balanced data(undersampling ...
3
votes
1answer
745 views

What loss function does the 'multinomial' distribution with the gbm package in R use?

All distributions in the gbm package in R are associated with a loss function. For example, when we set distribution = 'binomial'...
6
votes
2answers
164 views

Why isn't dimension sampling used with gradient boosting machines (GBM)?

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used ...
5
votes
2answers
3k views

Kappa near to 60% in unbalanced (1:10) data set

As mentioned before, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples. I have trained a Generalized Boosted Regression model using ...