Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

Filter by
Sorted by
Tagged with
13
votes
2answers
9k views

L1 & L2 Regularization in Light GBM

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation: reg_alpha (float, optional (default=0.)) – L1 ...
45
votes
2answers
47k views

How to interpret the output of XGBoost importance?

I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance. What is the meaning of Gain, Cover, and Frequency and how do we ...
30
votes
4answers
31k views

Unbalanced multiclass data with XGBoost

I have 3 classes with this distribution: Class 0: 0.1169 Class 1: 0.7668 Class 2: 0.1163 And I am using xgboost for ...
2
votes
4answers
423 views

Is it possible to get worse model after optimization?

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to ...
39
votes
6answers
31k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
6
votes
3answers
1k views

xgboost: Is there a way to perform regression on rates/percentages data?

I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific ...
31
votes
3answers
42k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
59
votes
5answers
84k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost ...
28
votes
3answers
38k views

xgboost: give more importance to recent samples

Is there a way to add more importance to points which are more recent when analyzing data with xgboost?
15
votes
4answers
4k views

Need help understanding xgboost's approximate split points proposal

background: in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective: $$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$ where $g_i,...
19
votes
1answer
8k views

Lightgbm vs xgboost vs catboost

I've seen that in Kaggle competitions people are using lightgbms where they used to use xgboost. My question is: when would you rather use xgboost instead of lightgbm? What about catboost?
13
votes
1answer
6k views

What is the difference in xgboost binary:logistic and reg:logistic

What is the difference in R in xgboost between binary:logistic and reg:logistic? Is it only in evaluation metric? If yes, how does RMSE on binary classification compare to error rate? Is the ...
2
votes
2answers
608 views

Lowering learning rate makes my accuracy on the validation set go down

I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase ...
34
votes
3answers
33k views

Why do we need XGBoost and Random Forest?

I wasn't clear on couple of concepts: XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ? ...
37
votes
2answers
20k views

LightGBM vs XGBoost

I'm trying to understand which is better (more accurate, especially in classification problems) I've been searching articles comparing LightGBM and XGBoost but found only two: https://medium.com/...
20
votes
2answers
21k views

How fit pairwise ranking models in xgBoost?

As far as I know, to train learning to rank models, you need to have three things in the dataset: label or relevance group or query id feature vector For example, the Microsoft Learning to Rank ...
12
votes
1answer
4k views

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
7
votes
2answers
14k views

Why don't tree ensembles require one-hot-encoding?

I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
18
votes
1answer
30k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
8
votes
1answer
7k views

Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event ...
10
votes
2answers
18k views

Would you recommend feature normalization when using boosting trees?

For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using ...
9
votes
3answers
2k views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
9
votes
4answers
2k views

Will cross validation performance be an accurate indication for predicting the true performance on an independent data set?

I feel that this question is related to the theory behind cross-validation. I present my empirical finding here and wrote a question related to the theory of cross-validation at there. I have two ...
7
votes
3answers
29k views

Xgboost - How to use feature_importances_ with XGBRegressor()?

How could we get feature_importances when we are performing regression with XGBRegressor()? There is something like ...
5
votes
2answers
9k views

Prediction Intervals Using XGBoost

I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this blog, and not really understanding how the ...
5
votes
2answers
12k views

Xgboost predict probabilities

When using the python / sklearn API of xgboost are the probabilities obtained via the predict_proba method "real probabilities" or do I have to use ...
3
votes
1answer
862 views

What does xgb's scale_pos_weight parameter do for regression?

From other posts (see Unbalanced multiclass data with XGBoost) and the documentation, scale_pos_weight in XGBoost appears to balance positive and negative cases, ...
8
votes
1answer
5k views

XGBoost Feature importance - Gain and Cover are high but Frequency is low

I have read this question: How do i interpret the output of XGBoost importance? about the three different types of feature importances: frequency (called "weight" in Python XGBoost), gain, and cover. ...
3
votes
2answers
8k views

XGBoost change loss function

I'm using XGBoost (through the sklearn API) and I'm trying to do a binary classification. False Positives are much worse for me than False Negatives, how can I take this into account? The API ...
6
votes
1answer
187 views

XGBoost results are not invariant under monotone predictor transformations?

It is believed by many that tree-based methods are invariant under monotone transformations of the predictors. But recently I've read a paper (https://arxiv.org/pdf/1611.04561.pdf, referred to as the ...
6
votes
3answers
5k views

How to train a xgboost model on data that is too big for the memory?

What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more ...
3
votes
1answer
3k views

Classification using xgboost - predictions

I was trying to build a 0-1 classifier using xgboost R package. My question is how predictions are made? For example in random forests, trees "vote" against each option and the final prediction is ...
2
votes
1answer
1k views

Balancing XGboost still skews towards the majority class

I have unbalanced dataset for multiclass classification and I tried to use the class weights option in XGboost and the classifier still tends to favor the majority class. I am not sure if I need to ...
2
votes
2answers
91 views

High Recall but too low Precision result in imbalanced data

I was training a model using XGBoost Classifier on a heavy imbalanced database with 232:1 of binary class. Because my training data contains 750k rows and 320 features (after doing many feature ...
-3
votes
1answer
2k views

Training XGBoost sequentially

I'm currently tring to train a model with XGBoost. My dataset has ~7 million records and 61 columns. The problem I'm currently having is that I get a MemoryError on python when I try to train the ...
4
votes
3answers
3k views

Can I fine tune the xgboost model instead of re-training it?

I am using the xgboost library. My system runs a cronjob each night, where it pulls the data from the database and trains the model. However, I would like to remove the re-training of the model again ...
4
votes
1answer
4k views

changing cost function in xgboost [closed]

I'm using the newest version of xgboost package in python 2.7 and based on my problem, I'm going to change xgboost cost function to use my own defined cost function. Couple of questions: In which ...
3
votes
2answers
824 views

XGBoost validation for number of trees

I have a simple Question: I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2 2.) With 500 ...
3
votes
1answer
186 views

Average of importance gain for a categorical variable

Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc). I train an XGBoost model on a numeric ...
2
votes
1answer
580 views

my xgboost model accuracy decreases after grid search with [duplicate]

I tried grid search for hyperparameter tuning in XGBoost classifier but the best accuracy is less than the accuracy without any tuning ...
2
votes
2answers
40 views

Boosted tree regression loss function when data has occasionally very large values to predict?

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious ...
2
votes
1answer
535 views

eta and learning_rate different in xgboost

I am creating a classification model using xgboost in python. I am using different eta values to check its effect on the model. My code is- ...
1
vote
1answer
31 views

Time series data and ML - separating training/test data

I am using XGBoost to try to predict the direction of the stock market based on social media sentiment. Having read through some studies, I was planning to separate the training/test data by time ...
1
vote
1answer
61 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
1
vote
0answers
62 views

xgboost calibration kde plots (isotonic) not smooth

i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
1
vote
1answer
5k views

How to reach continue training in xgboost

I read the paper but found nothing talking about how to implement incremental learning. Can someone share some basic or deep knowledge? not in coding way. I know how to write code snippet to train ...
1
vote
1answer
345 views

Understanding python XGBoost model dump output of a very simple tree

I am trying to understand the model dump output from XGBoost. I would like to step through and see exactly how the model arrived at it's prediction. To simplify I trained a model with 1 tree and 1 max ...