Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

192 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
2 answers
146 views

Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning. Should I include the prediction matrix (ie. df containing columns of prediction results ...
  • 161
6 votes
1 answer
231 views

What is a good objective function for allowing close to 0 predictions?

Let's say we want to predict the probability of rain. So just the binary case: rain or no rain. In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this ...
  • 17.9k
6 votes
1 answer
2k views

In XGBoost, how to change eval function and keeping same objective?

I would like to keep the objective as "reg:linear" and eval_metric as customized RMSE as follows: ...
5 votes
2 answers
1k views

how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to ...
  • 61
5 votes
0 answers
1k views

How does XGBoost compute the probabilities in predict_proba()?

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed. In random forest for example, I ...
  • 51
5 votes
0 answers
11k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
3 votes
0 answers
46 views

Does ROC AUC different between crossval and test set indicate overfitting or other problem?

I am training a composite model (XGBoost, Linear Regression, and RandomForest) to predict injured people probability. Well, the results of cross-validation with 5 folds. Well, I can see any problem ...
3 votes
3 answers
68 views

Example for Boosting

Can someone exactly tell me how does boosting as implemented by LightGBM or XGBoost work in real case scenerio. Like I know it splits tree leaf wise instead of level wise, which will contribute to ...
  • 183
3 votes
1 answer
79 views

Output value of a gradient boosting decision tree node that has just a single example in it

The general gradient boosting algorithm for tree-based classifiers is as follows: Input: training set $\{(x_{i},y_{i})\}_{i=1}^{n}$, a differentiable loss function $L(y,F(x))$, and a number of ...
3 votes
0 answers
133 views

weighted quantile sketch in xgboost

I am unable to understand what is weighted quantile sketch in xgboost. Can anyone help me give an intuitive understanding of this?
3 votes
0 answers
707 views

Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )

I have a dataset of 23k rows of an unbalanced dataset 85/15 ratio, 10 variables ( 9 of which are categorical ) , i'm using CatBoost and XGBoost for a binary classification. I applied cv (5 iteration ...
  • 1,994
3 votes
0 answers
387 views

Adjust class weights due to class imbalance and class importance Multi class classification XGBoost

With respect to this question and the answer given by @Esmailian, Would anyone be able to let me know if Class B has a higher importance or the positive class ( i.e. it needs to have a higher ...
3 votes
0 answers
668 views

Xgboost rank:ndcg learning per group or for all dataset

I'm trying to implement xgboost with an objective of rank:ndcg I want the target to be between 0-3. In my data for most of the groups, there is only 1 event per ...
  • 161
3 votes
0 answers
679 views

Target transformation for tree models

Can anybody explain why/if target variable transformations could help when dealing with tree based models? I've seen this excellent reply which explains quite well why it shouldn't affect if ...
  • 221
3 votes
0 answers
52 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
  • 131
3 votes
1 answer
585 views

Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
  • 141
3 votes
0 answers
1k views

XgBoost error: contrasts can be applied only to factors with 2 or more levels

...
  • 907
2 votes
1 answer
63 views

Learning to Rank with Unlabelled Dataset

I have folder of about 60k PDF documents that I would like to learn to rank based on queries to surface the most relevant results. The goal is to surface and rank relevant documents, very much like a ...
  • 51
2 votes
1 answer
1k views

Xgboost Multiclass evaluation Metrics

Im training an Xgb Multiclass problem, but im having doubts about my evaluation metrics, heres my code + output ...
2 votes
0 answers
541 views

What is the difference between RandomForestClassifier and XGBRFClassifier?

What is the difference between RandomForestClassifier and XGBRFClassifier? There is no detailed explanation about what XGBRFClassifier exactly is so I was wondering.
  • 1,007
2 votes
1 answer
37 views

Static ML model or Time-Series? How to model/predict a binary target when I have time variant features but most features are constant?

I have been working with Real World data from patients. I have a dataset with information about 10million patients; Collected over a span of varying duration (5 to 20 years). What I am predicting is ...
  • 21
2 votes
0 answers
23 views

Understanding additive function approximation or Understanding matching pursuit

I am trying to read Greedy function approximation: A gradient boosting machine. On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach ...
2 votes
0 answers
294 views

how can i plot probability distribution of my classes in the way below?

All, I would like to plot the following: I have a binary classification problem where I am using xgboost as my 'model' below: ...
  • 486
2 votes
2 answers
438 views

Which loss function is the best loss function when using XGB regression with highly skewed dataset?

Which loss function is the best loss function when using XGB regression with a highly skewed dataset? The skewness of the data is very high. I used XGBoost with objective function of linear ...
  • 658
2 votes
1 answer
215 views

Prediction issue with xgboost custom loss

I have an issue with xgboost custom objectives: I do not manage to get consistent forecasts. In other words, the scale of my forecasts is not in line with the values I would like to predict. I tried ...
  • 21
2 votes
0 answers
787 views

XGBoost predicting everything as null when sample weights are passed

I am trying to build an Uplift model using observational data. The data is consists of collections calls to customers and my objective is to predict the incremental probability due to the treatment (...
2 votes
0 answers
1k views

Improving recall in XGBoost algorithm

I have highly imbalanced dataset. I am using XGBoost and I got the following results without balancing the dataset out: Precision: 0.87 Recall: 0.79 F1: 0.83 My ...
  • 151
2 votes
0 answers
176 views

How does the feval parameter influences the XGBoost training process?

In the package XGBoost, is possible to modify the feval (evaluated function) to a personalized one (as shown in the link: MAPE eval metric). I would like to know how is the training process of the ...
2 votes
1 answer
87 views

Training a model where each response in the observation data has a different known varience

I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a ...
  • 21
2 votes
1 answer
285 views

Multiple XGBoost models or just 1 for a cetain type of category?

I am building a model to predict, say house prices. Within my data I have sales and rentals. The Y variable is the price of either the sales or rentals. I also have ...
2 votes
2 answers
4k views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for classifiers ...
2 votes
0 answers
81 views

Averaging CNN perform worse than boosting

I'm trying to solve Quora Question Pairs with model stacking. My first layers are: CNN trained to predict the same target as whole model should "Magic features" like question frequency in ...
2 votes
0 answers
576 views

Sales Dataset to determine best model for predicting future sales

We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data ...
  • 29
2 votes
0 answers
562 views

Why does xgboost give this unexpected result?

This is a really simple example where my training data has a single feature vector (1,2,3) and an equivalent target vector (1,2,3). I can get xgboost to build a ...
  • 131
2 votes
1 answer
140 views

My own model trained on the full data is better than the best_estimator I get from GridSearchCV with refit=True?

I am using an XGBoost model to classify some data. I have cv splits (train, val) and a separate test set that I never use until the end. I have used GridSearchCV to determine the best parameters and ...
2 votes
1 answer
531 views

XGBoost - feature importance just depends on the location of the feature in the data

I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data ...
1 vote
0 answers
29 views

Is there a typo in the XGBoost example in the paper?

I am reading the XGBoost paper and I found this example below in the paper to calculate the quality tree structure: To calculate the quality tree structure, we use the following formula: My question,...
  • 113
1 vote
0 answers
17 views

How to convert machine learning XGBoost "R" binary model into CORE ML on iOS Swift?

How can we convert our sleep stage classification "R" XGBoost binary model on Windows into CORE ML on iOS to run model on iPhone? CORE ML doc says it inputs XGBoost -- need guide or tutorial....
  • 111
1 vote
1 answer
22 views

How can I manually add a root node in XGBoost tree(s)?

I have a binary feature {0, 1} and only when its value is 1, I would like my XGBoost model to "evaluate" a set of ...
  • 11
1 vote
1 answer
88 views

Linear regression returning negative values for house price prediction

I am trying to do a prediction of real estate (prices are in millions). The mean price for the dataset is 4 million. I do not have any negative values in my dataset,...
1 vote
0 answers
28 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
  • 36
1 vote
0 answers
75 views

Does it make sense to use target encoding together with tree-based models?

I'm working on a regression problem with a few high-cardinality categorical features (Forecasting different items with a single model). Someone suggested to use target-encoding (mean/median of the ...
  • 11
1 vote
1 answer
34 views

XGBoost results changing when one row is removed

I have a training dataset of 2,600 rows and 26 columns. I trained an XGBoost (1.3.1) Classification model using the data and evaluated it using a test set of c. 800 rows. Whilst experimenting I found ...
1 vote
0 answers
19 views

Predictions using calibrated classifer

I find myself asking alot of calibration related questions recently - but i cannot find adequate material on it! I am training a binary classifier to predict default. This probability will be used in ...
  • 486
1 vote
0 answers
193 views

Are predictive features with 0 SHAP values included in the model?

I have trained and XGBoost by enforcing no-feaure interaction and calculated Global Shap values: It looks like only 6 features have some SHAP values, whilst the remaining ones have a SHAP value of 0. ...
1 vote
0 answers
42 views

Do monotonic constraints prevent an XGboost to capture non-linear relationships in the data?

I have trained an XGBoost model (for a binary classification problem) and I have tested two scenarios: Scenario 1 - No Monotonic Constrained applied In this case I get a Gini on the training sample of ...
1 vote
0 answers
563 views

Differences between Feature Importance and SHAP variable importance graph

I have run an XGBClassifier using the following fields: ...
1 vote
0 answers
18 views

What is the formula of gradient boosting trees model?

I have been reading about gradient boosting trees (GBT) in some machine learning books and papers, but the references seem to only describe the training algorithms of GBT, but they do not describe the ...
1 vote
1 answer
248 views

Are there rules of thumb for xgboosts hyperperameter selection?

There are multiple parameters that need to be specified in the XGBClassifier. Certainly gridsearchcv could give some insight into optimal hyperparameters, but I would imagine there are some rules of ...
  • 165
1 vote
1 answer
28 views

Why does an unimportant feature has a big impact on R2 in XGBoost?

I am training an XGBoost model, xgbr, using xgb.XGBRegressor() with 13 features and one numeric target. The R2 on the test set ...