Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

Filter by
Sorted by
Tagged with
0
votes
2answers
23 views

XgBoost given targets its only feature but fails when test targets are outside the range of training targets?

I'm learning to use XgBoost, and I'm doing an exercise involving predicting prices. However I'm noticing some weird behavior where XgBoost's predictions deviate from the target value even if I'm ...
4
votes
1answer
33 views

What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding: hessian ...
2
votes
0answers
20 views

Static ML model or Time-Series? How to model/predict a binary target when I have time variant features but most features are constant?

I have been working with Real World data from patients. I have a dataset with information about 10million patients; Collected over a span of varying duration (5 to 20 years). What I am predicting is ...
4
votes
1answer
193 views

What is a good objective function for allowing close to 0 predictions?

Let's say we want to predict the probability of rain. So just the binary case: rain or no rain. In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this ...
3
votes
1answer
1k views

Small number of estimators in gradient boosting

I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models ...
2
votes
2answers
93 views

High Recall but too low Precision result in imbalanced data

I was training a model using XGBoost Classifier on a heavy imbalanced database with 232:1 of binary class. Because my training data contains 750k rows and 320 features (after doing many feature ...
1
vote
1answer
65 views

How to restructure my dataset for interpretability without losing performance?

What I am doing: I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format: What I want to do: I want to use SHAP TreeExplainer to interpret each prediction my model ...
0
votes
2answers
982 views

Specifying number of threads using XGBoost.train

When using the xgboost.train() function, all the threads are used. I would like to use a specific amount. Unfortunately, this function does not accept the ...
0
votes
1answer
716 views

On which algorithm for boosting is the method xgbLinear of the xgboost/caret- package based on?

In caret package of R, there is a method 'xgblinear'. What is the working algorithm behind this method.
0
votes
0answers
8 views

What if root of a such tree is pruned in xgboost?

Extreme Gradient Boosting stops to grow a tree if $\gamma$ is greater than impurity reduction given as eq (7) (see below) , what does happen if tree's root has a negative impurity? I think there is no ...
2
votes
2answers
129 views

Which loss function is the best loss function when using XGB regression with highly skewed dataset?

Which loss function is the best loss function when using XGB regression with a highly skewed dataset? The skewness of the data is very high. I used XGBoost with objective function of linear ...
4
votes
1answer
1k views

In XGBoost, how to change eval function and keeping same objective?

I want to keep objective as "reg:linear" and eval_metric as customised rmse as follows. ...
39
votes
6answers
31k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
6
votes
1answer
105 views

Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning. Should I include the prediction matrix (ie. df containing columns of prediction results ...
1
vote
1answer
63 views

One-hot & interaction one-hot on multiple categorical

I was wondering if there is any value to creating combined features out of multiple categorical variables when the individual categorical variables are already one-hot encoded? Simple example: there ...
1
vote
1answer
63 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
1
vote
1answer
434 views

Comparing XGBR with CatBoost performance

I saw on a CatBoost site that it supposed to outperform any other boosted training model and decided to try it myself on a Kaggle's https://www.kaggle.com/c/house-prices-advanced-regression-techniques....
2
votes
1answer
305 views

SHAP Explanations in case of repeated train/test split

I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
0
votes
1answer
51 views

How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in ...
1
vote
1answer
11 views

XGboost with group-specific and individual-specific data

My dataset consists of a combination of two databases. One database consists of individual-level data on the characteristics and compensation of the top five executive officers of big American ...
0
votes
2answers
36 views

Why is XGBClassifier in Python outputting different feature importance values with the same data across different repetitions?

I am fitting an XGBClassifier to a small dataset (32 subjects) and find that if I loop through the code 10 times the feature importances (gain) assigned to the features in the model varies slightly. I ...
3
votes
1answer
50 views

Any advantage of sklearn wrappers for xgboost over python API?

Are there any advantages of using the XGBoost sklearn wrappers XGBRegressor or XGBClassifier over using the Python API with the <...
1
vote
0answers
6 views

how could RFECV ever give the same score to all number of features selected?

I built an XGB and ran RFECV over 250 features. After an hour or so, I plotted the grid_scores_. All numbers of features are within 0.02, as clearly visible on the y-axis. To me this plot would ...
1
vote
0answers
14 views

Xgboost regressor : prediction overestimated or underestimated

Hi, Recently I am working on a project about synthesizing wind turbine power by the other wind turbines, like the graph above , axis X is wind turbine A power and axis Y is wind turbine B power. All ...
1
vote
0answers
23 views

XGBoost regressor hyperparameter tuning with hyperopt leads to overfit

Using hyperopt to hyperparameter tuning on XGBoost regressor, I am receiving overfiting on the train set. There is any suggestion how to solve it ? I have used cross validation with ...
2
votes
1answer
108 views

Multiple XGBoost models or just 1 for a cetain type of category?

I am building a model to predict, say house prices. Within my data I have sales and rentals. The Y variable is the price of either the sales or rentals. I also have ...
1
vote
1answer
1k views

Improving prediction accuracy with XGBoost

I have a 32x20 matrix for which I am trying to use XGBoost (Regression). I am looping through rows to produce an out of sample forecast. I'm surprised that XGBoost only returns an out of sample ...
4
votes
3answers
3k views

Can I fine tune the xgboost model instead of re-training it?

I am using the xgboost library. My system runs a cronjob each night, where it pulls the data from the database and trains the model. However, I would like to remove the re-training of the model again ...
0
votes
0answers
10 views

XGBoost:Logistic poor performance with Scaled and PCA data

Working on a data set similar to fraud but not monetary transactions. Here are the steps that I have taken on the modeling side: Convert Some of the categorical into numerical (One hot encode) Over ...
0
votes
0answers
22 views

Is my approach for a loss function that adds more importance to negative samples reasonable?

For my current project I'm using XGBoost Regression to predict values y_pred with mean = 0 and std = 1. I want my model to place more emphasis on predicting samples right, where the true value y_true ...
1
vote
0answers
9 views

How to understand Xgboost model dump

Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump ...
0
votes
1answer
19 views

Random Forest but keep only leaves with impurities below a threshold

Is there an algorithm out there that creates a random forest but then prunes all the leaves that have an impurity measure above a certain threshold that I would determine? In other words, if I set min ...
0
votes
0answers
28 views

XGBoost failing on highly imbalanced data!

I am working on a classification problem, where I am trying to predict a fraud login. The data is highly imbalanced i.e. 0 = non fraud logins , 1 = fraud logins 0 : 4538076 1 : 365 I have been trying ...
0
votes
1answer
218 views

Correct theoretical regularized objective function for XGB/LGBM (regression task)

I am writing an academic paper on the application of Machine Learning methods to Time Series Forecasting and I am unsure about how to write down the theoretical part about the regularized objective ...
2
votes
2answers
3k views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for classifiers ...
1
vote
1answer
22 views

Tree complexity in xgboost

According to xgboost paper, regularization is given by: $$\Omega(f) = \gamma T + \lambda || w||^2$$ where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree). The parameter ...
0
votes
0answers
29 views

Train/ Test split on small dataset along with SMOTE

I have a binary classification imbalanced dataset with 1000 samples ( 15% of class 1, 85% of the rest). My main goal is to build a robust classifier using the following approach. Wanted to know if ...
1
vote
1answer
100 views

scale_pos_weight using XGBoost's Learning API

I see it is possible to add a weight for unbalanced problems in XGBoost's Scikit-Learn API through scale_pos_weight. Does it have an equivalent in the Learning API? If not, is there a reason behind ...
0
votes
1answer
39 views

Is it possible to do Normalization before Xgboost?

Currently I am working on a project which uses Xgboost Regression. Before putting data into model, I implemented Normalization, the accuracy significantly increased compared with without Normalization....
0
votes
0answers
16 views

Techniques for Ordinal Classification/Regression with Gradient Boosted Trees

I did some research on how to run ordinal classification decision trees (such as lightgbm, xgboost), and found these articles to be helpful. Both use a k-1 binary classification technique to output ...
2
votes
0answers
19 views

XGBoost incremental training for big datasets

I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred). I have been trying to use some libraries such as Dask to deal with this problem, without any success due to ...
0
votes
0answers
25 views

What's the difference between multiclass categorical crossentropy, mlogloss and multi:softprob?

As far as I understand, an objective is something I'm trying to optimize and an evaluation statistic is something I use to look for overfitting. I stumbled upon 4 losses that seem to be the same, but ...
1
vote
1answer
16 views

Prediction problem across a wide space of Clash Royale card games

I have assembled a database of Clash Royale games in an attempt to understand the outcomes of various match-ups. The game is composed of an 8 card deck drawn from 102 cards. As you can see from the ...
0
votes
1answer
9k views

How to print feature names in conjunction with feature Importance using Imbalanced-learn library?

I used BalancedBaggingClassifier from imblearn library to do an unbalanced classification task. How can I get feature improtance of the estimator in conjunction ...
3
votes
1answer
471 views

Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes ...
0
votes
0answers
18 views

Is it possible that shap feature importance result will be more accurate than gain?

XGboost build the boosted tree in the following way: Each level of each tree (the phase of selecting the next feature with conditional value) selected according ...
2
votes
1answer
51 views

Why Continous Variable Buckets Overfitting model

I have a continuous (high cardinal discrete) variable 'numInteractionPoints' in my dataset during training model - I binned this feature in order to avoid overffing , first top bar chart is from ...
9
votes
3answers
2k views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
0
votes
2answers
32 views

select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations

I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, ...

1
2 3 4 5
12