Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

Filter by
Sorted by
Tagged with
42
votes
5answers
51k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost ...
37
votes
2answers
31k views

How to interpret the output of XGBoost importance?

I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance. What is the meaning of Gain, Cover, and Frequency and how do we ...
29
votes
1answer
21k views

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. XGBClassifier handles 500 trees within 43 seconds on my machine, while <...
27
votes
3answers
38k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
25
votes
3answers
19k views

Why do we need XGBoost and Random Forest?

I wasn't clear on couple of concepts: XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ? ...
25
votes
2answers
14k views

LightGBM vs XGBoost

I'm trying to understand which is better (more accurate, especially in classification problems) I've been searching articles comparing LightGBM and XGBoost but found only two: https://medium.com/...
23
votes
4answers
15k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
22
votes
3answers
27k views

xgboost: give more importance to recent samples

Is there a way to add more importance to points which are more recent when analyzing data with xgboost?
16
votes
4answers
15k views

Unbalanced multiclass data with XGBoost

I have 3 classes with this distribution: Class 0: 0.1169 Class 1: 0.7668 Class 2: 0.1163 And I am using xgboost for ...
13
votes
1answer
23k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
13
votes
1answer
4k views

Decision trees: leaf-wise (best-first) and level-wise tree traverse

Issue 1: I am confused by the description of LightGBM regarding the way the tree is expanded. They state: Most decision tree learning algorithms grow tree by level (depth)-wise, like the ...
12
votes
2answers
13k views

How fit pairwise ranking models in xgBoost?

As far as I know, to train learning to rank models, you need to have three things in the dataset: label or relevance group or query id feature vector For example, the Microsoft Learning to Rank ...
12
votes
1answer
3k views

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
11
votes
3answers
3k views

Need help understanding xgboost's approximate split points proposal

background: in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective: $$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$ where $g_i,...
11
votes
3answers
561 views

XGBoost outputs tend towards the extremes

I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature ...
10
votes
3answers
413 views

XGboost - Choice made by model

i am using XGboost to predict a 2 classes target variable on insurance claims. I have a model ( training with cross validation, hyper parameters tuning etc...) i run on another dataset. My question ...
10
votes
1answer
3k views

Lightgbm vs xgboost vs catboost

I've seen that in Kaggle competitions people are using lightgbms where they used to use xgboost. My question is: when would you rather use xgboost instead of lightgbm? What about catboost?
10
votes
1answer
3k views

What is the difference in xgboost binary:logistic and reg:logistic

What is the difference in R in xgboost between binary:logistic and reg:logistic? Is it only in evaluation metric? If yes, how does RMSE on binary classification compare to error rate? Is the ...
9
votes
3answers
19k views

Pandas Dataframe to DMatrix

I am trying to run xgboost in scikit learn. And I only use Pandas to load data into dataframe. How am i supposed to use pandas df with xgboost. I am confused by the DMatrix routine required to run ...
9
votes
1answer
12k views

XGBoost Linear Regression output incorrect

I am a newbie to XGBoost so pardon my ignorance. Here is the python code : ...
9
votes
1answer
8k views

Gradient Boosting Tree: “the more variable the better”?

From the tutorial of the XGBoost, I think when each tree grows, all the variables are scanned to be selected to split nodes, and the one with the maximum gain split will be chosen. So my question is ...
9
votes
1answer
131 views

What is meant by Distributed for a gradient boosting library?

I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library. What is meant by distributed? Have a nice day
8
votes
4answers
11k views

Is feature engineering still useful when using XGBoost?

I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, ...
8
votes
4answers
311 views

Why is there a difference between predicting on Validation set and Test set?

I have a XGBoost model trying to predict if a currency will go up or down next period (5 min). I have a dataset from 2004 to 2018. I split the data randomized into 95% train and 5% validation and the ...
8
votes
1answer
10k views

XGBoost for binary classification: choosing the right threshold

I am working on a highly-imbalanced binary-labeled dataset, where number of true labels is just 7% from the whole dataset. But some combination of features could yield higher than average number of ...
7
votes
2answers
4k views

XGBoost Feature importance - Gain and Cover are high but Frequency is low

I have read this question: How do i interpret the output of XGBoost importance? about the three different types of feature importances: frequency (called "weight" in Python XGBoost), gain, and cover. ...
7
votes
1answer
6k views

How to get a confidence score for predictions?

In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?
7
votes
0answers
543 views

Layman's Interpretation of XGBoost Importance [duplicate]

I'm trying to come up with a good way to explain the 3 importance metrics (Gain, Cover, Frequency) to a layman with only a basic understanding of XGBoost and trees in general. How best would you ...
6
votes
1answer
5k views

How does Xgboost learn what are the inputs for missing values?

So from Algorithm 3 of https://arxiv.org/pdf/1603.02754v3.pdf, it says that an optimum default direction is determined and the missing values will go in that direction. However, or perhaps I have ...
6
votes
3answers
5k views

Gradient boosting algorithm example

I'm trying to fully understand the gradient boosting (GB) method. I've read some wiki pages and papers about it, but it would really help me to see a full simple example carried out step-by-step. Can ...
6
votes
2answers
10k views

Would you recommend feature normalization when using boosting trees?

For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using ...
6
votes
3answers
153 views

xgboost: Is there a way to perform regression on rates/percentages data?

I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific ...
6
votes
1answer
1k views

How to place XGBoost in a full stack for ML?

Is XGBoost complete by itself for prod-strength machine learning? If not, with which other tools or libs is it typically combined, and how? (I recently read a description of a stack that included ca ...
5
votes
2answers
7k views

Why don't tree ensembles require one-hot-encoding?

I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
5
votes
2answers
9k views

In XGBoost would we evaluate results with a Precision Recall curve vs ROC?

I am using XGBoost for payment fraud detection. The objective is binary classification, and the data is very unbalanced. One out of every 3-4k transactions is fraud. I would expect the best way to ...
5
votes
1answer
2k views

What are the “extra nodes” in XGboost?

When training an XGboost model some of the information printed regards "extra nodes". I can't find an explanation of these anywhere in the documentation. What exactly are extra nodes? ...
5
votes
2answers
734 views

What algorithms will stuck in the local minimum?

Algorithms like neural network are easily getting stuck in local minimum because the shape of the loss function (so there are parameters like momentum are designed to solve this type of problem). ...
5
votes
2answers
181 views

XGBoost, binary classification: uneven number of observations per user

I'm working on a binary classification problem with XGBoost and I have a dataset, which has uneven number of observations per user. For some users there are over 100 observations, whereas for some ...
5
votes
1answer
2k views

What does it mean to “warm-start” XGBoost?

In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are ...
4
votes
3answers
14k views

Xgboost - How to use feature_importances_ with XGBRegressor()?

How could we get feature_importances when we are performing regression with XGBRegressor()? There is something like ...
4
votes
2answers
3k views

Can xgboost (or any other algorithm) give bad results with some bad features?

till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data. In one of my problems, there are around ...
4
votes
1answer
3k views

Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event ...
4
votes
2answers
9k views

How to determine feature importance while using xgboost in pipeline?

How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline? ...
4
votes
1answer
159 views

Interpretability of RMSE and R squared scores on cross validation

I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that ...
4
votes
2answers
549 views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
4
votes
2answers
673 views

L1 & L2 Regularization in Light GBM

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation: reg_alpha (float, optional (default=0.)) – L1 ...
4
votes
1answer
663 views

Handling unbalanced datasets with XG boosting

Suppose you want to model (predict) a rare disease, and you use the parameter "pos scale weight" as a hyperparameter in XG boost . For example I have 20 times more positive cases, can I then use pos ...
4
votes
2answers
2k views

Why do we use gradients instead of residuals in Gradient Boosting?

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
4
votes
1answer
58 views

SHAP value can explain right?

I face a problem with using SHAP value to interpret the Tree-based model. (https://github.com/slundberg/shapsd) First, I have input around 30 features and I have 2 features that have high positive ...
4
votes
1answer
347 views

How to extract trees in XGBoost?

I want to extract each tree so that I can feed it with any data, and see the output. ...