Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

183 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
1answer
105 views

Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning. Should I include the prediction matrix (ie. df containing columns of prediction results ...
5
votes
0answers
443 views

XGBoost custom objective for regression in R

I implemented a custom objective and metric for a xgboost regression task. In order to see if I'm doing this correctly, I started with a quadratic loss. The ...
4
votes
1answer
193 views

What is a good objective function for allowing close to 0 predictions?

Let's say we want to predict the probability of rain. So just the binary case: rain or no rain. In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this ...
4
votes
1answer
1k views

In XGBoost, how to change eval function and keeping same objective?

I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows. ...
4
votes
0answers
813 views

How does XGBoost compute the probabilities in predict_proba()?

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed. In random forest for example, I ...
4
votes
0answers
11k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
3
votes
1answer
30 views

What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding: hessian ...
3
votes
0answers
89 views

weighted quantile sketch in xgboost

I am unable to understand what is weighted quantile sketch in xgboost. Can anyone help me give an intuitive understanding of this?
3
votes
0answers
608 views

Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )

I have a dataset of 23k rows of an unbalanced dataset 85/15 ratio, 10 variables ( 9 of which are categorical ) , i'm using CatBoost and XGBoost for a binary classification. I applied cv (5 iteration ...
3
votes
0answers
342 views

Adjust class weights due to class imbalance and class importance Multi class classification XGBoost

With respect to this question and the answer given by @Esmailian, Would anyone be able to let me know if Class B has a higher importance or the positive class ( i.e. it needs to have a higher ...
3
votes
0answers
531 views

Xgboost rank:ndcg learning per group or for all dataset

I'm trying to implement xgboost with an objective of rank:ndcg I want the target to be between 0-3. In my data for most of the groups, there is only 1 event per ...
3
votes
0answers
554 views

Target transformation for tree models

Can anybody explain why/if target variable transformations could help when dealing with tree based models? I've seen this excellent reply which explains quite well why it shouldn't affect if ...
3
votes
0answers
50 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
3
votes
0answers
1k views

how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to ...
3
votes
1answer
471 views

Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
3
votes
0answers
1k views

XgBoost error: contrasts can be applied only to factors with 2 or more levels

...
2
votes
0answers
20 views

Static ML model or Time-Series? How to model/predict a binary target when I have time variant features but most features are constant?

I have been working with Real World data from patients. I have a dataset with information about 10million patients; Collected over a span of varying duration (5 to 20 years). What I am predicting is ...
2
votes
0answers
19 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
2
votes
2answers
91 views

High Recall but too low Precision result in imbalanced data

I was training a model using XGBoost Classifier on a heavy imbalanced database with 232:1 of binary class. Because my training data contains 750k rows and 320 features (after doing many feature ...
2
votes
1answer
51 views

Why Continous Variable Buckets Overfitting model

I have a continuous (high cardinal discrete) variable 'numInteractionPoints' in my dataset during training model - I binned this feature in order to avoid overffing , first top bar chart is from ...
2
votes
0answers
21 views

Output value of a gradient boosting decision tree node that has just a single example in it

The general gradient boosting algorithm for tree-based classifiers is as follows: Input: training set ${\displaystyle \{(x_{i},y_{i})\}_{i=1}^{n},}{\displaystyle \{(x_{i},y_{i})\}_{i=1}^{n},}$a ...
2
votes
1answer
31 views

How to measure deviance resulting from different random seeds in machine learning?

I'm running an xgboost model to predict probabilities to a binary classification problem. Then I aggregate the results based on the Age variable (what is the aggregated risk of getting the sickness ...
2
votes
0answers
16 views

Understanding additive function approximation or Understanding matching pursuit

I am trying to read Greedy function approximation: A gradient boosting machine. On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach ...
2
votes
1answer
84 views

why does my calibration curve for platts and isotonic have less points than my uncalibrated model?

i train a model using grid search then i use the best parameters from this to define my chosen model. ...
2
votes
0answers
78 views

how can i plot probability distribution of my classes in the way below?

All, I would like to plot the following: I have a binary classification problem where I am using xgboost as my 'model' below: ...
2
votes
2answers
129 views

Which loss function is the best loss function when using XGB regression with highly skewed dataset?

Which loss function is the best loss function when using XGB regression with a highly skewed dataset? The skewness of the data is very high. I used XGBoost with objective function of linear ...
2
votes
0answers
122 views

Prediction issue with xgboost custom loss

I have an issue with xgboost custom objectives: I do not manage to get consistent forecasts. In other words, the scale of my forecasts is not in line with the values I would like to predict. I tried ...
2
votes
1answer
305 views

SHAP Explanations in case of repeated train/test split

I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
2
votes
2answers
119 views

Adding extra variables to XGboost model is worsening the train and test accuracy

I am fitting a multi class model using Xgboost. I am getting an accuracy of 96% on Train and 95% on test. I am using the 80-20 train/test split. However, when I am adding two new features , the ...
2
votes
0answers
532 views

XGBoost predicting everything as null when sample weights are passed

I am trying to build an Uplift model using observational data. The data is consists of collections calls to customers and my objective is to predict the incremental probability due to the treatment (...
2
votes
1answer
71 views

Training a model where each response in the observation data has a different known varience

I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a ...
2
votes
1answer
108 views

Multiple XGBoost models or just 1 for a cetain type of category?

I am building a model to predict, say house prices. Within my data I have sales and rentals. The Y variable is the price of either the sales or rentals. I also have ...
2
votes
2answers
3k views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for classifiers ...
2
votes
1answer
248 views

Can you reuse observations from your train data in your final test data?

For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data. Unlike in most machine learning problems where you ...
2
votes
0answers
78 views

Averaging CNN perform worse than boosting

I'm trying to solve Quora Question Pairs with model stacking. My first layers are: CNN trained to predict the same target as whole model should "Magic features" like question frequency in ...
2
votes
0answers
560 views

Sales Dataset to determine best model for predicting future sales

We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data ...
2
votes
0answers
536 views

Why does xgboost give this unexpected result?

This is a really simple example where my training data has a single feature vector (1,2,3) and an equivalent target vector (1,2,3). I can get xgboost to build a ...
2
votes
1answer
209 views

XGBoost - feature importance just depends on the location of the feature in the data

I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data ...
1
vote
1answer
11 views

XGboost with group-specific and individual-specific data

My dataset consists of a combination of two databases. One database consists of individual-level data on the characteristics and compensation of the top five executive officers of big American ...
1
vote
0answers
6 views

how could RFECV ever give the same score to all number of features selected?

I built an XGB and ran RFECV over 250 features. After an hour or so, I plotted the grid_scores_. All numbers of features are within 0.02, as clearly visible on the y-axis. To me this plot would ...
1
vote
0answers
14 views

Xgboost regressor : prediction overestimated or underestimated

Hi, Recently I am working on a project about synthesizing wind turbine power by the other wind turbines, like the graph above , axis X is wind turbine A power and axis Y is wind turbine B power. All ...
1
vote
0answers
23 views

XGBoost regressor hyperparameter tuning with hyperopt leads to overfit

Using hyperopt to hyperparameter tuning on XGBoost regressor, I am receiving overfiting on the train set. There is any suggestion how to solve it ? I have used cross validation with ...
1
vote
0answers
9 views

How to understand Xgboost model dump

Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump ...
1
vote
0answers
19 views

XGBoost with deep trees

I've been exploring the use of XGBoost in many different applications. Up to now, I always find the best results with shallow trees (from 1 to 3 levels), with the rest of the parameters very dependent ...
1
vote
0answers
49 views

XGBoost: Typical gamma and min_child_weight range

What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm? Is the range of min_child_weight correlated with the number of feature or samples in the training ...
1
vote
1answer
16 views

Prediction problem across a wide space of Clash Royale card games

I have assembled a database of Clash Royale games in an attempt to understand the outcomes of various match-ups. The game is composed of an 8 card deck drawn from 102 cards. As you can see from the ...
1
vote
0answers
22 views

How can I use a validation set to tune the hyperparameters of an XGBClassifier?

I'm currently building a ranking model using an XGBClassifier. I have training, testing, and validation sets. I want to use the validation set to tune the hyperparameters of the XGBClassifier before ...
1
vote
1answer
104 views

Negative R2_score Bad predictions for my Sales prediction problem using LightGBM

My project involves trying to predict the sales quantity for a specific item across a whole year. I've used the LightGBM package for making the predictions. The params I've set for it are as follows: <...
1
vote
0answers
45 views

Feature Selection before modeling with Boosting Trees

I have read in some papers that the subset of features chosen for a boosting tree algorithm will make a big difference on the performanceso I've been trying RFE, Boruta, Clustering variables, ...