Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

135 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
11
votes
3answers
560 views

XGBoost outputs tend towards the extremes

I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature ...
4
votes
1answer
243 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
4
votes
1answer
102 views

xgboost or lightgbm to handle Binomial problems

I have a dataset containing a column of trials, a column of successes and other features; and, obviously, I can generate a probability column. I would like to use gradient boosting methods (like ...
3
votes
1answer
645 views

Imbalanced dataset with 3 classes xgboost scale_pos_weight parameter

The xgboost classifier states the use of parameter scale_pos_weight for 2-class problems. I have a highly imbalanced dataset with 3 classes. Classes '1' and '-1' ...
3
votes
0answers
274 views

Target transformation for tree models

Can anybody explain why/if target variable transformations could help when dealing with tree based models? I've seen this excellent reply which explains quite well why it shouldn't affect if ...
3
votes
0answers
101 views

XGBoost results are not invariant under monotone predictor transformations?

It is believed by many that tree-based methods are invariant under monotone transformations of the predictors. But recently I've read a paper (https://arxiv.org/pdf/1611.04561.pdf, referred to as the ...
3
votes
0answers
66 views

Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning. Should I include the prediction matrix (ie. df containing columns of prediction results ...
3
votes
0answers
1k views

In XGBoost, how to change eval function and keeping same objective?

I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows. ...
3
votes
0answers
510 views

Sales Dataset to determine best model for predicting future sales

We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data ...
3
votes
0answers
10k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
2
votes
0answers
28 views

Can I specify the root node splitting feature in XGBoost?

Just what the title says. Suppose I know the feature that I want to be used for splitting for the root node in a tree model in XGBoost; is there a way for me to tell XGBoost to split on this feature ...
2
votes
0answers
173 views

Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )

I have a dataset of 23k rows of an unbalanced dataset 85/15 ratio, 10 variables ( 9 of which are categorical ) , i'm using CatBoost and XGBoost for a binary classification. I applied cv (5 iteration ...
2
votes
1answer
52 views

Training a model where each response in the observation data has a different known varience

I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a ...
2
votes
0answers
241 views

Adjust class weights due to class imbalance and class importance Multi class classification XGBoost

With respect to this question and the answer given by @Esmailian, Would anyone be able to let me know if Class B has a higher importance or the positive class ( i.e. it needs to have a higher ...
2
votes
0answers
40 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
2
votes
0answers
589 views

how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to ...
2
votes
2answers
781 views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for XGBoost, ...
2
votes
0answers
1k views

How to perform an actual time series prediction using xgboost- python

I have train data for 5 months and test data for one month which i am using to validate my model.Here is the xgboost code i wrote in python-\ ...
2
votes
0answers
65 views

Prevent overffitting in model stacking with training on the same target

I'm trying to solve Quora Question Pairs with model stacking. My first layers are: CNN trained to predict the same target as whole model should "Magic features" like question frequency in whole ...
2
votes
0answers
141 views

What is a good objective function for allowing close to 0 predictions?

Let's say we want to predict the probability of rain. So just the binary case: rain or no rain. In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this ...
2
votes
0answers
547 views

Custom objective function in xgboost for Regression

I am working on a regression problem, where I want to modify the loss function in xgboost library such that my predictions should never be lesser than the actual value. I have written this code: <...
2
votes
1answer
561 views

What does xgb's scale_pos_weight parameter do for regression?

From other posts (see Unbalanced multiclass data with XGBoost) and the documentation, scale_pos_weight in XGBoost appears to balance positive and negative cases, ...
2
votes
0answers
346 views

Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
2
votes
0answers
283 views

Stratified Sampling for XGboost

I have a multiclass-classification dataset with the target (dependent) variable highly imbalanced. While using the randomForest package in R, I usually use the parameters ...
2
votes
0answers
2k views

How to set weights in multi-class classification in xgboost for imbalanced data?

From this post, I know you can set scale_pos_weight for an imbalanced dataset. However, for the multi-classification problem in the imbalanced dataset, I don't ...
2
votes
0answers
108 views

What scale does LightGBM use for output?

Let's assume I'm modeling a process like: $$y=log(x1+x2+x3+N)$$ Where $x_i$ are features and $N$ is some error/noise value. With the way that decision trees work and the way that LightGBM works, ...
2
votes
0answers
599 views

How does XGBoost compute the probabilities in predict_proba()?

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed. In random forest for example, I ...
2
votes
0answers
852 views

XgBoost error: contrasts can be applied only to factors with 2 or more levels

...
2
votes
0answers
455 views

Why does xgboost give this unexpected result?

This is a really simple example where my training data has a single feature vector (1,2,3) and an equivalent target vector (1,2,3). I can get xgboost to build a ...
1
vote
0answers
20 views
1
vote
1answer
32 views

Average of importance gain for a categorical variable

Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc). I train an XGBoost model on a numeric ...
1
vote
0answers
15 views

SHAP Explanations in case of repeated train/test split

I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
1
vote
2answers
23 views

Adding extra variables to XGboost model is worsening the train and test accuracy

I am fitting a multi class model using Xgboost. I am getting an accuracy of 96% on Train and 95% on test. I am using the 80-20 train/test split. However, when I am adding two new features , the ...
1
vote
0answers
22 views

Feature Vectors representation

I would like to know I how you represent a feature vector like this dataset wise. The vector length is dynamic but the each element has a fixed length (9). For xgboost implementation, do I just create ...
1
vote
0answers
43 views

Hyperparameter Tuning using Bayesian Techniques

I've been looking into Bayesian optimization for hyperparameter tuning and trying to compare the results I get to those I get using different methods (random grid search). I came across this site, ...
1
vote
0answers
26 views

XGBoost speed issues

I'm trying to optimize the hyperparameters for XGBoost, thus needing to run it multiple times with different parameters. However the time needed to run single XGBoost with the parameters provided ...
1
vote
1answer
25 views

xgboost in R have different results compared to boosted decision tree in Azure ML

I have a small data set (4000 records with 10 features) and I used XGBOOST in R as well as Boosted Decision Tree model in Azure ML studio. Unfortunately the results are different. I like to optimize ...
1
vote
0answers
19 views

What is the best approach to train a multi-category regression model?

What is the best approach to train a multi-category regression model (using XBoost decision trees ensemble)? What are the ups and downs of each one? For example, if I want to train a model to predict ...
1
vote
0answers
53 views

Trying to beat random forest with xgboost

I have a small time series dataset of about 3000 samples and 5 features. With xgboost, my predictions seem biased (consistently overestimating the target). No matter how many estimators I throw at the ...
1
vote
0answers
77 views

Use LightGBM or FFM - imbalanced dataset

I have a highly imabalanced dataset but one that is not sparse. In train there are 1328 positives out of 104000. In validation ...
1
vote
0answers
51 views

XGBoost predicting everything as null when sample weights are passed

I am trying to build an Uplift model using observational data. The data is consists of collections calls to customers and my objective is to predict the incremental probability due to the treatment (...
1
vote
1answer
51 views

Explaining XGBoost functioning to non-technical people

I have been tasked to explain the principle of the XGBoost algorithm to non-technical people (think 1-2 slides in a powerpoint presentation to upper management). I am currently working with the ...
1
vote
1answer
520 views

XGBOOST : model.predict_proba() and model.predict() conflicting behaviour

I have two classes : 1 and 2 The output of model.predict_proba() -> [0.333,0.6667] The output of model.predict() -> 1 This is happening for around 200 test values out of the test data of 10 lac. ...
1
vote
0answers
91 views

Improving recall in XGBoost algorithm

I have highly imbalanced dataset. I am using XGBoost and I got the following results without balancing the dataset out: Precision: 0.87 Recall: 0.79 F1: 0.83 My ...
1
vote
0answers
18 views

Is it possible to create nonbinary trees in XGBoost?

I'm looking through the documentation for XGBoost, and I'm not seeing any parameters relating to number of branches per node.
1
vote
0answers
30 views

I have tried 5 different types of model but all returns really low training accuracy (~64%) and low testing accuracy (~14%). What should I do?

I am working with a typical regressor problem. There are $6$ features in the dataset that I am concerned with. There are about $800$ data points in my dataset. The features and the predicted values ...
1
vote
0answers
46 views

Feature Importance Scores from Gradient Boosting vs Random Forest

In sklearn, the feature_importances_ attribute exists for both RandomForestClassifier and GradientBoostingClassifier. Would like to know what are the fundamental differences in how this attribute is ...
1
vote
1answer
89 views

Tweedie Loss for Keras

We are currently using XGBoost model with Tweedie loss for solving a regression problem which works very good, now I wanted to move our model to Keras and experience with neural networks, do anybody ...
1
vote
0answers
35 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
1
vote
2answers
33 views

Determining threshold in an area with very few samples of positive label

I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...