Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

Filter by
Sorted by
Tagged with
2
votes
1answer
6k views

Gradient boosting vs logistic regression, for boolean features

I have a binary classification task where all of my features are boolean (0 or 1). I have been considering two possible supervised learning algorithms: Logistic regression Gradient boosting with ...
5
votes
2answers
8k views

Why don't tree ensembles require one-hot-encoding?

I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn'...
1
vote
0answers
62 views

Injecting random values as one input feature for feature selection results in a odd beaviour

I am trying to find a cutoff value, in the feature importance space to eliminate spurious features. So I am injecting a completely random generated feature (as one of the input features) and I cut the ...
8
votes
1answer
11k views

XGBoost for binary classification: choosing the right threshold

I am working on a highly-imbalanced binary-labeled dataset, where number of true labels is just 7% from the whole dataset. But some combination of features could yield higher than average number of ...
8
votes
4answers
11k views

Is feature engineering still useful when using XGBoost?

I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, ...
2
votes
0answers
615 views

How does XGBoost compute the probabilities in predict_proba()?

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed. In random forest for example, I ...
9
votes
1answer
8k views

Gradient Boosting Tree: “the more variable the better”?

From the tutorial of the XGBoost, I think when each tree grows, all the variables are scanned to be selected to split nodes, and the one with the maximum gain split will be chosen. So my question is ...
13
votes
1answer
24k views

XGBRegressor vs. xgboost.train huge speed difference?

If I train my model using the following code: ...
2
votes
1answer
1k views

More features hurts when underfitting?

I was training a binary classifier using XGBClassifier (basically boosted decision trees if I understand it correctly). I have 10K training examples. I have two distinct set of features (but they ...
1
vote
0answers
344 views

How to set subsample in lightgbm in R?

There is a subsample parameter for the XGBoost (xgb.train() function in R). What is the corresponding subsample parameter for ...
2
votes
0answers
877 views

XgBoost error: contrasts can be applied only to factors with 2 or more levels

...
44
votes
5answers
53k views

GBM vs XGBOOST? Key differences?

I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost ...
5
votes
1answer
2k views

What does it mean to “warm-start” XGBoost?

In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are ...
19
votes
4answers
16k views

Unbalanced multiclass data with XGBoost

I have 3 classes with this distribution: Class 0: 0.1169 Class 1: 0.7668 Class 2: 0.1163 And I am using xgboost for ...
5
votes
2answers
10k views

In XGBoost would we evaluate results with a Precision Recall curve vs ROC?

I am using XGBoost for payment fraud detection. The objective is binary classification, and the data is very unbalanced. One out of every 3-4k transactions is fraud. I would expect the best way to ...
6
votes
2answers
11k views

Would you recommend feature normalization when using boosting trees?

For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using ...
1
vote
2answers
983 views

Kelly Criterion in xgboost loss function

I have a model that predicts the outcome of ATP tennis matches. The quality of predictions varies, and I want to develop a second binary classification model that optimises the decision to bet (or not)...
0
votes
1answer
4k views

XGboost classification with very small data set

I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets. Normally I tune the n_rounds parameters by cross-validation, but what if you have too less ...
2
votes
1answer
411 views

Problem when loading a XGBoost model in a different computer

I'm working on a project and we are using XGBoost to make predictions. My colleague sent me the model file but when I load on my computer it don't run as expected. When I changed one variable from ...
4
votes
2answers
10k views

How to determine feature importance while using xgboost in pipeline?

How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline? ...
1
vote
0answers
672 views

Feeding data to Xgboost for recomender system

I am using xgboost for a recommender system. There are 3-4 recommended content on each page. My data consists of columns like page_id and advertisement_id. Currently for every page_id, there are 3-4 ...
1
vote
1answer
2k views

Xgboost (classification problem) feature importance per input not for the model

I have trained a xgboost model for a classification problem. I'm able to get the feature importance for the model as below. http://machinelearningmastery.com/feature-importance-and-feature-selection-...
1
vote
1answer
1k views

How to run XGBoostregressor using reg: tweedie as objective?

I installed XGBoost for anaconda on windows 10 based on the instructions provided here. It seems that xgboost 0.6 is already installed. It performs well using "...
4
votes
2answers
4k views

Xgboost quantile regression via custom objective

This is my first time posting, so please bare with me if I miss giving necessary info... I'm new to GBM and xgboost, and I'm currently using xgboost_0.6-2 in R. The modeling runs well with the ...
-1
votes
1answer
810 views

Feature engineering using XGBoost

I am participating in a kaggle competition. I am planning to use the XGBoost package (in R). I read the XGBoost documentation and understood the basics. Can someone explain how is feature engineering ...
3
votes
0answers
528 views

Sales Dataset to determine best model for predicting future sales

We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data ...
2
votes
1answer
2k views

roc_auc score GridSearch

I am experimenting with xgboost. I ran GridSearchCV with score='roc_auc' on xgboost. The best classificator scored ~0.935 (this is what I read from GS output). But now when I run best classificator ...
1
vote
1answer
797 views

Variance in cross validation score / model selection

Between cross-validation runs of a xgboost classification model, I gather different validation scores. This is normal, the Train/validation split and model state are different each time. ...
6
votes
1answer
6k views

How does Xgboost learn what are the inputs for missing values?

So from Algorithm 3 of https://arxiv.org/pdf/1603.02754v3.pdf, it says that an optimum default direction is determined and the missing values will go in that direction. However, or perhaps I have ...
2
votes
1answer
2k views

Boruta feature selection in R with custom importance (xgboost feature importance)

According to the documentation - CRAN Boruta is an all relevant feature selection wrapper algorithm, capable of working with any classification method that output variable importance measure (VIM); ...
1
vote
0answers
280 views

What does “gblinear” do in XGBoost?

For example, if I run a single round (nrounds=1), how does XGBoost go about making predictions? I thought it would simply return a linear regression model, but I ...
2
votes
1answer
412 views

XGBoost in final model what does “yes=3,no=4” mean?

XGBoost v0.6 Binary Classification $ cat predin.txt 0 1:10 2:10 3:10 1 1:10 2:10 3:100 1 1:10 2:100 3:10 1 1:100 2:10 3:10 In other words, the negative instance ...
0
votes
1answer
946 views

Dynamic learning rates in XGBoost cross-validation

XGBoost's xgb.train() method takes a learning_rates parameter, which can take a custom function to apply a dynamic learning rate,...
3
votes
5answers
3k views

Machine learning algorithm for ranking

I am working on a ranking question, recommending k out of m items to the users. The evaluation metric is average precision at K. Both R and Python have xgboost can be used for pairwise comparison ...
0
votes
1answer
147 views

High overestimation on prediction data

I am building lost sales estimation model for out of stock days etc. using XGBoost. I am using simple logic of training model on data of normal days with ample inventory (when sales and demand are ...
3
votes
2answers
8k views

Xgboost predict probabilities

When using the python / sklearn API of xgboost are the probabilities obtained via the predict_proba method "real probabilities" or do I have to use ...
3
votes
0answers
10k views

Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model: ...
1
vote
3answers
2k views

Forecasting sales of next year using sales of past years?

I have a dataset of sales of a company for different products throughout 3 years. I have to forecast the sales for each of these products for next year. A sample of the dataset is: Here I have to ...
2
votes
2answers
2k views

xgboost Predictions from R and Python don't match

Thought maybe somebody here could help us solve a mystery (https://github.com/dmlc/xgboost/issues/1623): We are trying to build a xgboost prediction function in R for a model that was trained in ...
0
votes
1answer
205 views

Does anyone have memory utilization benchmark for random forest and xgboost?

I want to compare which technique has higher memory utilization while training on the same dataset
2
votes
1answer
178 views

xgboost speed difference per API

How can it be that a xgboost.cv cross-validation operation where n-folds are evaluated is quicker than a single ...
4
votes
2answers
1k views

When is Gradient Descent invoked on the objective function while running XGboost?

is it at the end of every tree? or only after all trees are build? I tried to think in both ways but didn't get a clear picture. Can we focus more the part "the loss function is applied between ...
2
votes
2answers
3k views

XGBoost increase the error when changing evaluation function

I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does ...
2
votes
1answer
2k views

How does xgboost work with linear booster?

I know how to implement linear objective function and linear boosts in XGBoost. My concrete question is: when the algorithm it fits the residual (or the negative gradient) is it using one feature at ...
11
votes
3answers
20k views

Pandas Dataframe to DMatrix

I am trying to run xgboost in scikit learn. And I only use Pandas to load data into dataframe. How am i supposed to use pandas df with xgboost. I am confused by the DMatrix routine required to run ...
-1
votes
1answer
534 views

Comparing Categorical and Continuous Features using Splits in GBM

In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well ...
2
votes
0answers
460 views

Why does xgboost give this unexpected result?

This is a really simple example where my training data has a single feature vector (1,2,3) and an equivalent target vector (1,2,3). I can get xgboost to build a ...
24
votes
4answers
16k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
1
vote
0answers
508 views

Why initialization of Xgboost DMatrix reducec features number?

I am trying to understand following case: when I create new xgbost DMatrix xgX = xgb.DMatrix(X, label=Y, missing=np.nan) ...
3
votes
1answer
2k views

How can the process of hypertuning of XGBoost parameters be automated?

I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here. After performing grid search and some manual settings, I ...