Questions tagged [xgboost]
For questions related to the eXtreme Gradient Boosting algorithm.
701
questions
1
vote
0
answers
17
views
Weird Behavour in Skforcast
I applied XGBoost to forcast a univariante time series dataset, the first time I created my own lags features manually:
...
2
votes
0
answers
26
views
Tuning the learning rate parameter for GBDT models
I've always been taught that decreasing the learning rate parameter in gbdt models such as XGBoost, LightGBM and Catboost will improve the out-of-sample performance, assuming the number of iterations ...
0
votes
1
answer
19
views
How does deeper or shallower trees (higher or lower max_depth) affect xgboost model?
I am doing an xgboost model for landslides assessment and I am using max_depth as one of my hyperparameters, but I don't understand how does it affect model ...
0
votes
0
answers
16
views
What is the complexity of ICE (Individual conditional expectation) on ensemble trees
I'm evaluating different model-agnostic methods on gradient boosting or random forest model.
For Shapley, specifically TreeSHAP, the complexity is O(TLD^2) according to Lundberg et al. 2018. T: #trees,...
0
votes
0
answers
21
views
Difference in the value of evaluation-metric in xgb.train() and predict in R
I have trained a xgboost classifier with a custom metric (f1_xgb), that is, the F1 score. Here the important aspect is that I evaluated the classifier on the test set by setting:
...
0
votes
0
answers
44
views
Gradient function in LogisitcLoss class
I am going through a code for XGBoost from scratch and I am referring to this repository here
The log-loss function is given by
On differentiating the above function with respect to y_pred (referring ...
0
votes
1
answer
33
views
How is the weight of each new weak learner is calculated in Xgboost?
In Xgboost we have multiple sequential weak learner.
Let say I have weak learner WL1 and we fitted it on our data and we calulated the error. Now we have another weak learner WL2. And as I have read ...
0
votes
2
answers
145
views
Handling categorical variables for Xgboost?
Currently there seems to be two approaches for handling categorical variables in gbdts:
Xgboost as an option, but data need to be encoded properly (integers)
Catboost can handle everything provided ...
0
votes
1
answer
69
views
Xgboost multiclass monotonic constraints
I have a problem where i have a variable price and i need to classify this price as winning/non-winning. If price grows, probability should monotonically go down. I use a monotonic constraint that ...
0
votes
0
answers
62
views
How to do feature selection correctly in xgboost for time series forecasting after obtaining a good predictive model?
I have a very large dataset (~7 million rows) for which I have extracted ~500 features during feature engineering phase. I have trained an XGBoost which has a fairly good predictive capability (based ...
0
votes
0
answers
19
views
LightGBM Regressor miscalibratred/underestimating on high fitted values and overestimating on low fitted values
I'm training a pretty standard LightGBM regressor and noticing a strange pattern with the residuals (see images below--I'm bunching the predicted values and taking the observed average for the group). ...
0
votes
0
answers
53
views
How to deal with missing values in the output for XGBoost Regressor
So I am trying to create a regression model that takes two arrays as the input features and an array as an Output. However, some of the point in this dataset do not contain any value.
This is because ...
0
votes
0
answers
55
views
Custom objective function for xgboost to optimize lift in best decile
I've tried to define a custom objective function for xgboost to optimize the lift for a binary classification problem in the upper decile. The task is simply to concentrate the training effort to the ...
0
votes
1
answer
152
views
Understand and compute confidence interval and coefficient of variation for regression model
I would like to better understand the concepts of:
coefficient of variation and confidence interval.
Trivially taking the definitions from wikipedia:
confidence interval (CI)
In frequentist ...
0
votes
0
answers
52
views
Ordinal log-loss in a multiclass classification in XGBoost?
I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High).
What I would ...
0
votes
0
answers
108
views
Making ML Model (XGBoost) Output Smoother
I am using XGBoost for a classification problem. The model has multiple inputs and a probability as output.
For simplicity's sake, let's say the model only has one continous input and the function to ...
0
votes
0
answers
7
views
Histogram creation in lightgbm in the train API and the scikit-learn API. Is it always benefitial to use the train API?
In the LightGBM for python we have a scikit-learn API in which (either for regression or for classification) there is fit method whose documentation is
fit(X, y, sample_weight=None, init_score=None, ...
0
votes
0
answers
55
views
Xgboost giving different results on windows and Linux
I'm facing one issue with the results not matching on Windows and Linux. I'm using the same code on both machines. Using separate trains. csv and test. csv
The problem is due to sampling and ...
0
votes
1
answer
34
views
Validity of using raw time series data for training of xgboost/random forest classifier
I am currently working on a project aiming of classification of process states based on time series data. For this, we are looking at different models, such as XGBoost-based classifiers or ...
0
votes
0
answers
18
views
Minimize MAE loss for a target that is sum of two other targets
Working on a regression modelling task where my dataset have some feature columns, two more columns A, B and a target column T. The goal is to predict T, and minimize MAE, that is ...
0
votes
0
answers
24
views
Binary classification using xgboost
Why when adding new features in my ADS for a binary classification using XGBOOST my score and uplift has decreased ? What is the best way to treate categorical features or other features in order that ...
0
votes
0
answers
15
views
Parameter outside specified range using HGboost/Hyperopt library
I am trying to use the HGboost library which uses the Hyperopt library for doing hyperparameter optimization of an XGboost model. The script runs fine but the optimized parameter for "...
1
vote
2
answers
116
views
Xgboost model predicting extreme values for events and non-events | Overfitting
Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
0
votes
0
answers
38
views
XGBoost Classifier Evaluation Confusion on New Dataset Despite High Cross-Validation Scores
I have built an XGBoost classifier model with 90 features, trained on a dataset containing 760k samples. I took great care to separate the labels from the features in both the training and testing ...
1
vote
0
answers
51
views
0
votes
0
answers
60
views
XGBoost Classifier + Isotonic Regression leading to worse probability accuracy
I'm testing out an XGBoost Classifier with the goal of using the probabilities it predicts in production. I know that tree based model probabilities are often not calibrated well so I decided to test ...
0
votes
0
answers
231
views
XGBoost tweedie regression objetive from scratch
I'm trying to gain a deeper understanding of the tweedie loss function and how it is used in XGBoost. So, I tried to implement it from scratch. I started by examining the original implementation.
I ...
0
votes
0
answers
55
views
XGBRegressor underestimates sum of regression target (insurance application)
I'm trying to learn how to apply Boostig Algorithms (e.g. XGBoost) to insurance applications (premium calculation). As a starting point I used this tutorial from the scikit-learn website.
tldr: The ...
1
vote
1
answer
165
views
Quality of first tree prediction in xgboost
Up until now I have mostly been using neural nets for regression. There on each iteration (epoch) the weights are updated using the whole training data, and we expect the error to go down each time, ...
0
votes
0
answers
65
views
Problem calculating f-1 score when when running hyperparameter tuning using Optuna
I'm trying to run Stratified CV hyperparameter tuning on an XGBClassifier using Optuna. This is a multiclass problem with 3 classes (labelled 0 through 2 in the "classes" variable). These ...
1
vote
0
answers
64
views
Replication of XGBoost's binary:logistic loss
I am trying to replicate XGBoost's logistic loss function as a first step before implementing my own custom loss functions. Following from here and looking at the original code in git repository, I ...
1
vote
1
answer
67
views
Time series prediction on test set produces multiple cycles instead of following trend
I have a non-stationary time series where I am trying to build a model for forecasting. So far on test set it produces multiple cycles no matter which technique I use.
There's just one feature ...
0
votes
1
answer
102
views
XGBoost with objective as binary:logistic returning only binary value
I am training an xgboost model for binary classification using objective as 'binary:logistic'. The model should predict probability but it is outputting either 0 or 1. I want the model to output the ...
1
vote
3
answers
149
views
Feature importance score for a feature that contains mostly 0's in XGBoost
I have read that the feature importance scores are calculated based on how a split on that feature improves performance.
I have a binary classification dataset and am running XGBoost classifier on it. ...
0
votes
1
answer
65
views
eval_metric of XGBoost // ML model in general
Say I am using Xgboost on a binary classification task. eval_metric is one of the model parameter. How should I think about the impact of using different eval_metric(e.g rmse/mae/logloss) in general? ...
0
votes
0
answers
84
views
Can I decompose SHAP interaction values like a linear regression?
I had a question regarding the shap interaction matrix.
Suppose I have 500 samples with 2 features. Then my interaction matrix becomes (500,2,2).
I want to calculate the SHAP values of each feature ...
0
votes
0
answers
54
views
Build a model with cross-validation on entire dataset to learn insights?
Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed.
Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
0
votes
0
answers
64
views
XGBoost Architecture Diagram required
Good Day!
My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
0
votes
0
answers
47
views
weight derivation for XGBOOST in regression trees
I am working on a variation of the XGBOOST presented on the following paper by Chen and Guestrin: https://dl.acm.org/doi/abs/10.1145/2939672.2939785
My goal is to fit a linear regression in each leaf ...
0
votes
0
answers
62
views
How to predict an xgboost model outcome directly from the trained trees?
I want to train by Xgboost algorithm and predict directly using the trees while testing. Precisely, speaking I don't want to keep the model weights in any file like "joblib" and load it ...
1
vote
1
answer
172
views
Funny looking learning curve
I am training an XGBoost for binary classification. I am getting a very odd-looking learning curve. Any idea what is going on and how I might fix this? Thanks
1
vote
1
answer
286
views
Why is accuracy score suddenly becoming 1 on using XGBoost?
I am developing a music classification system based on a kaggle dataset: https://www.kaggle.com/datasets/vatsalmavani/spotify-dataset
I tried using K means classifier to classify the songs into 4 ...
0
votes
0
answers
48
views
Why am I getting differing "gain" feature importances from XGBoost?
I am currently training an XGBoost model for binary classification. I have fitted and predicted with the model but when I try to get the "gain" type feature importances, the results differ ...
0
votes
0
answers
242
views
How do the predict() and predict_proba() for xgbClassifier() work?
I am using xgboost classifier for a binary classification problem. I used the predict() to get the class predictions (0/1) but I also used the predict_proba() method to get the class probabilities (my ...
0
votes
1
answer
296
views
How to intrepret low F1 score and high AUC on training set?
I am currently working on a very imbalanced dataset:
24 million transactions (rows of data)
30,000 fraudulent transactions
(0.1% of total transactions)
and I am using XGBoost as the model to predict ...
0
votes
0
answers
57
views
Why does my accuracy score drop after hyperparameter tuning in XGBoost?
I am trying to tune the model I've built, but every time I change hyperparameters my accuracy score drops significantly. I'm using RandomizedSearchCV and best_params_ to determine which parameters I ...
0
votes
0
answers
86
views
Visualize Catboost and XGBoost training process + Cross Validation
I want to optimize Catboost and XGBoost models and visualize this process such that:
Use 3-fold cross-validation
Use my own pre-processing pipeline (Missing value imputation, over- or undersampling)
...
0
votes
0
answers
16
views
Data processing - how to input a pandas column that contains numbers and numpy arrays
I have a pandas df that contains numbers and strings. I use word2vec to convert all the strings into embeddings. The problem now is that these embeddings are all numpy arrays. So now my pandas df ...
1
vote
1
answer
165
views
Deleting part of saved XGBoost-Model (JSON) and reloading it
I am training an XGBoost-model on part of the ForestCover-Dataset. Then I save the trained model to json. Now I load the model and "update" the saved model with the data, that I previously ...
0
votes
1
answer
33
views
Validating ML regression model and predictions
I have a years worth of electricity power data on 15 minute intervals joined with weather data and time-of-week one hot dummy variables.
Is using train/test split an okay approach for validating the ...