Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

Filter by
Sorted by
Tagged with
2
votes
2answers
590 views

Ordinal classification with xgboost

I am working in the problem where the dependent variables are ordered classes, such as bad, good, ...
0
votes
0answers
66 views

XGBoost regression

I run XGBoost regression with tree as base learner. I have over 400 variables and more than 30000000 samples. I have generated most important features and was surprised to see that one feature is ...
1
vote
1answer
404 views

What are the limitations while using XGboost algorithm? [closed]

Will XGBoost pose any problem while dealing with categorical variables with more than 2 levels. For example, occupation variable can have values like doctor, engineer, lawyer, data scientist, farmer e....
1
vote
0answers
38 views

Using feature vectors from imagenet to train xgboost (vs a standard Conv net)?

I am planning to use feature vectors generated from imagenet to train an xgboost model. This is as opposed to training a standard convolutional network with the same image set. This is because we ...
3
votes
1answer
641 views

Imbalanced dataset with 3 classes xgboost scale_pos_weight parameter

The xgboost classifier states the use of parameter scale_pos_weight for 2-class problems. I have a highly imbalanced dataset with 3 classes. Classes '1' and '-1' ...
2
votes
1answer
109 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
2
votes
1answer
590 views

What does the limit of xgboost max_depth=1 represent?

In my mind, this means that each tree just takes one feature, and produces a step function based upon it. In the limit of n_estimators being very large and max_depth=1, does xgboost become a linear ...
3
votes
0answers
274 views

Target transformation for tree models

Can anybody explain why/if target variable transformations could help when dealing with tree based models? I've seen this excellent reply which explains quite well why it shouldn't affect if ...
2
votes
1answer
127 views

Should I create metafeatures for my XGBoost training set?

Say I've got two (not necessarily independent) features A and B for my dataset. Should I create metafeatures from them? say for ...
4
votes
2answers
543 views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
0
votes
1answer
93 views

Why xgboost can not deal with this simple sentence case?

There is only 1 feature dim. But the result is unreasonable. The code and data is below. The purpose of the code is to judge whether the two sentences are the same. In fact, the final input to the ...
9
votes
1answer
131 views

What is meant by Distributed for a gradient boosting library?

I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library. What is meant by distributed? Have a nice day
1
vote
2answers
978 views

Both train and test error are decreasing in XGBoost iterations

I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 ...
0
votes
1answer
462 views

does xgb multi-class require one-hot encoding?

I was trying an xgboost from python with a multiclass single-label problem and assumed the label can be an integer indicating my class (as opposed to eg one-hot) . ...
2
votes
0answers
40 views

Is linear regression on the trees of XGBoost (rather than taking their mean) useful/popular?

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the ...
3
votes
1answer
463 views

Can I use xgboost on a dataset with 1000 rows for classification problem?

I have used all types of classification algorithms on my dataset yet I couldn't improve my score no matter how I try. So I've read about Xgboost classifier. So I was wondering is it practical to use ...
0
votes
1answer
399 views

Improving prediction accuracy with XGBoost

I have a 32x20 matrix for which I am trying to use XGBoost (Regression). I am looping through rows to produce an out of sample forecast. I'm surprised that XGBoost only returns an out of sample ...
1
vote
0answers
76 views

What would be the equivalent of R's mboost in Python?

I am looking for the Python equivalent of R's mboost package ( mboost ). Would that be xgboost?
4
votes
1answer
656 views

Handling unbalanced datasets with XG boosting

Suppose you want to model (predict) a rare disease, and you use the parameter "pos scale weight" as a hyperparameter in XG boost . For example I have 20 times more positive cases, can I then use pos ...
4
votes
1answer
985 views

Credit scoring using scorecardpy with XGBoost

I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried ...
1
vote
0answers
291 views

Comparing XGBR with CatBoost performance

I saw on a CatBoost site that it supposed to outperform any other boosted training model and decided to try it myself on a Kaggle's https://www.kaggle.com/c/house-prices-advanced-regression-techniques....
4
votes
1answer
2k views

Sensitivity analysis of a machine learning model

Let’s say I have a set of input variables (A, B, C and D)...
3
votes
2answers
3k views

Xgboost performs significantly worse than Random Forest

I have a dataset of 3500 observations x 70 features which is my training set and I also have a dataset of 600 observations x 70 features which is the test set. The target is to classify observations ...
2
votes
1answer
296 views

boosting an xgboost classifier with another xgboost classifier using different sets of features

What I would like to do, is train a first model $f_{1}(\underline{x})$, where $\underline{x}$ is a set of features, fix what model 1 has learned, and then train a second model $f_{2}(\underline{y})$ ...
1
vote
0answers
274 views

xgboost gain vs kolmogorov smirnov

After running xgboost model with: objective = 'binary:logistic' eval_metric = 'logloss' I have a group of 3 variables that have the highest values of gain. Now, ...
1
vote
1answer
294 views

Impact of sparse features on tree-based models

Say you have a highly imbalanced binary classification problem. Some of the features are binary features, where they're false most of the time, but when they're true they tend to be highly predictive (...
0
votes
1answer
2k views

How to show progress of sklearn.multioutput.MultiOutputRegressor and XGBRegressor?

Is it possible to show the training progress of the MultiOutputRegressor in sklearn? When a huge dataset is processed, my program runs a long time and I have no clue how long it will take. I have ...
2
votes
0answers
589 views

how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to ...
2
votes
2answers
781 views

Minimum number of samples to train XGBoost without overfitting

When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron. Is there a similar rule of thumb for XGBoost, ...
2
votes
1answer
236 views

get_dump() leaf value and AUC

I have used Xgboost fitted a model with AUC around 0.73 and I printed out my last booster: ...
3
votes
1answer
4k views

XGBoost: Quantifying Feature Importances

I need to quantify the importance of the features in my model. However, when I use XGBoost to do this, I get completely different results depending on whether I use the variable importance plot or the ...
1
vote
0answers
85 views

Regression Decision Tree - Normalize or Split into Ranges a continuos feature

I have in my dataset a feature named distances which ranges goes from 200 to 12000 (more or less). Since the other features have got values under 50 I need to do ...
4
votes
2answers
4k views

Python XGBoost killing kernel

My Jupyter notebook's python kernel keeps dying when attempting to train an XGBoost logistic classifier. Previously, I have run all of the following code successfully. Presently, there are issues. ...
1
vote
2answers
85 views

How can machine learning algorithms solve this particular problem?

Let's think of a case of an e-commerce website which lists products for sale. Now a person can come on a particular product page and decide to add it to the shopping cart or not. If we look at it as a ...
2
votes
1answer
253 views

eta and learning_rate different in xgboost

I am creating a classification model using xgboost in python. I am using different eta values to check its effect on the model. My code is- ...
0
votes
2answers
3k views

R langauge how to create xgb.DMatrix object from data frame (newbe)

In R, how does one create an xgb.DMatrix object from an R data frame?
2
votes
1answer
60 views

How can I minimize features of the trainded model?

I have real technological process, that explained with complex model (xgboost). I.e. current mass of a product (y) depends on current temperature (x1), pressure (x2) and so on. I would like to solve ...
3
votes
1answer
3k views

XGBoost evaluation metric unbalanced data - custom eval metric

I have built a model using the xgboost package (in R), my data is unbalanced (5000 positives vs 95000 negatives), with a binary classification output (0,1). I have performed cross validation with the ...
10
votes
3answers
411 views

XGboost - Choice made by model

i am using XGboost to predict a 2 classes target variable on insurance claims. I have a model ( training with cross validation, hyper parameters tuning etc...) i run on another dataset. My question ...
1
vote
2answers
40 views

What to optimize for when asked to find the most important features

I have a binary classification problem, let's say people can buy or not buy a certain product. Now unlike a standard prediction task, I only want to find which features are the most important for the ...
5
votes
2answers
708 views

What algorithms will stuck in the local minimum?

Algorithms like neural network are easily getting stuck in local minimum because the shape of the loss function (so there are parameters like momentum are designed to solve this type of problem). ...
3
votes
0answers
101 views

XGBoost results are not invariant under monotone predictor transformations?

It is believed by many that tree-based methods are invariant under monotone transformations of the predictors. But recently I've read a paper (https://arxiv.org/pdf/1611.04561.pdf, referred to as the ...
1
vote
2answers
68 views

Numerical example of Confusion in understanding learning rate in xgboost

I fail to understand as to how learning rate is used in XGBoost? Can anyone explain using a numerical example?
2
votes
2answers
158 views

xgboost cannot identify perfectly fitting regression line

For a dataset I want to use xgboost for the optimal ensembling of $n$ forecasts instead of just using their arithmetic mean for combination. I found that xgboost generates forecasts that are worse ...
2
votes
0answers
1k views

How to perform an actual time series prediction using xgboost- python

I have train data for 5 months and test data for one month which i am using to validate my model.Here is the xgboost code i wrote in python-\ ...
3
votes
2answers
3k views

What's SHAP contribution dependency plots from xgboost package in R?

...
3
votes
0answers
66 views

Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning. Should I include the prediction matrix (ie. df containing columns of prediction results ...
1
vote
1answer
138 views

Can you reuse observations from your train data in your final test data?

For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data. Unlike in most machine learning problems where you ...
4
votes
2answers
2k views

Why do we use gradients instead of residuals in Gradient Boosting?

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
2
votes
0answers
65 views

Prevent overffitting in model stacking with training on the same target

I'm trying to solve Quora Question Pairs with model stacking. My first layers are: CNN trained to predict the same target as whole model should "Magic features" like question frequency in whole ...