Questions tagged [xgboost]

For questions related to the eXtreme Gradient Boosting algorithm.

Filter by
Sorted by
Tagged with
2
votes
0answers
65 views

Prevent overffitting in model stacking with training on the same target

I'm trying to solve Quora Question Pairs with model stacking. My first layers are: CNN trained to predict the same target as whole model should "Magic features" like question frequency in whole ...
3
votes
1answer
2k views

changing cost function in xgboost

I'm using the newest version of xgboost package in python 2.7 and based on my problem, I'm going to change xgboost cost function to use my own defined cost function. Couple of questions: In which ...
4
votes
2answers
5k views

Is the early stopping of xgboost using correct

I'm using xgboost package in R with early stopping at 75 rounds. To monitor the progress the algorithm I print the F1 score from the training and test set after each round. After the algorithm has ...
2
votes
0answers
141 views

What is a good objective function for allowing close to 0 predictions?

Let's say we want to predict the probability of rain. So just the binary case: rain or no rain. In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this ...
2
votes
2answers
3k views

difference between XGBRegressor and XGBClassifier

I'm trying to understand the difference between xgboost.XGBRegressor and xgboost.sklearn.XGBClassifier. Can someone explain the difference in a concise manner? Because when I fit both classifiers ...
3
votes
1answer
226 views

XGBoost equations (for dummies)

I am having a hard time trying to understand the MSE loss function given in the Introduction to Boosted Trees (beware! My maths skills are the equivalent of a very sparse matrix): $ \begin{split}\...
1
vote
1answer
2k views

Using L1 penalty in XGBoost

I'm trying to use L1 regularization to select features in XGBoost classifier. However, I don't see any example code on how to specify the penalty of l1. This is how I do in sklearn's ...
1
vote
1answer
548 views

Is XGBoost better with numeric predictors?

I have a categorical feature that I one-hot encoded and used in my XGBoost model, but it consistently underperforms as a predictor compared to the other predictors. Then I created a new variable ...
3
votes
2answers
5k views

How to optimize XGBoost performance accuracy?

I have dataset to predict customers dropout(yes,no), with 5 numerical features and 2 categorical features. I have applied a scaler to the numerical data and transformed the categorical features into ...
4
votes
2answers
335 views

Tactics to avoid feeling overwhelmed by machine learning

Short version: despite lots of reading, machine learning still feels like being a monkey in the dark. Any advice? For background, I'm a researcher in computer science, in a field non-related to ...
4
votes
1answer
3k views

Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event ...
1
vote
0answers
42 views

AdaBoost - Ensemble model perform poor than best weak classifier

Can Adaboost's ensemble classifier perform worse than the best of the weak learners considered? If so when in what case of weak learner the ensemble learning does not perform better?
0
votes
1answer
639 views

Interpretation of tuning parameters (shrinkage and nrounds) in XGBoost

I'm using XGBoost for a multiclass classification problem in R. I'm trying different combinations of the shrinkage parameter and number of iterations to try and settle on optimal values for these ...
4
votes
1answer
6k views

What is the best way to deal with imbalanced data for XGBoost? [closed]

There are a lot of way to deal with class-imbalanced data like undersampling, oversampling, changing cost function etc. https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-...
3
votes
1answer
406 views

Correcting log-bias in the output of an XGB

I have previously worked with GAMs, where I was trying to do regression on a log-transformed variable. The log-transformation introduced a negative bias in the average of the predicted variable, and I ...
0
votes
1answer
944 views

How to find the residuals of a classification tree in xgboost

So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on ...
1
vote
1answer
40 views

Help with training XGB

I work for a highly-regulated entity, so I have to obfuscate what I'm working on; I'll provide the following as examples on what I'm doing. I am training an XGB model for NLP comments about breeds ...
1
vote
1answer
933 views

Configuring Incremental XGBoost model

I have a large dataset which can't be loaded in memory, hence I decided to use incremental learning using Xgboost. What I have done currently is: Tuned num_boosting_rounds using a chunk of data Set ...
2
votes
2answers
1k views

Xgboost interpretation: shouldn't cover, frequency, and gain be similar?

I was surprised to see the results of my feature importance table from my xgboost model. Based on the tutorials that I've seen online, gain/cover/frequency seems to be somewhat similar (as I would ...
0
votes
1answer
112 views

XGBoost: predictive, descriptive (or both) model?

I have trained an XGBoost model for prediction. The algorithm is able to calculate variable importances. I was asked why I have not analyzed these variable importances and I did not because as I ...
1
vote
0answers
2k views

xgboost f-score

I'm using xgboost package on python 3.5 for time base predication The result of the f-score Partial dependence on the order of the columns in data frame. The rmse of the predication is a same. for ...
11
votes
3answers
563 views

XGBoost outputs tend towards the extremes

I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature ...
3
votes
2answers
4k views

Cost-sensitive Logloss for XGBoost

I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost. $$ \begin{array} \\ p &...
1
vote
2answers
4k views

How to find and use the top features for XGBoost?

Suppose I have data with X_train, X_test, y_train, y_test given. As it is a classification problem I want to use XGBoost. The issue is that there are more than 300 features. I have found online ...
1
vote
2answers
463 views

Is it possible to use the saved xgboost model (with one-hot encoding features) on unseen data (without one-hot encoding) for prediction?

I think the question is self-explanatory. But let's say you have a data with a few features with categorical data, and when building a model for example XGBoost you one-hot encode categorical features....
13
votes
1answer
4k views

Decision trees: leaf-wise (best-first) and level-wise tree traverse

Issue 1: I am confused by the description of LightGBM regarding the way the tree is expanded. They state: Most decision tree learning algorithms grow tree by level (depth)-wise, like the ...
2
votes
2answers
92 views

What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)?

I have a data set I'm trying to create a predictor model for. The 5 features and outcome are all categorical data. One of the features contains 5,000 unique levels. While the other 4 are all under ...
7
votes
0answers
543 views

Layman's Interpretation of XGBoost Importance [duplicate]

I'm trying to come up with a good way to explain the 3 importance metrics (Gain, Cover, Frequency) to a layman with only a basic understanding of XGBoost and trees in general. How best would you ...
7
votes
1answer
6k views

How to get a confidence score for predictions?

In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?
1
vote
0answers
98 views

separate decision tree for categorical feature values

Given either, different decision trees each based on a particular feature value (like separate models for each male and female) or a single decision tree, should both give the same result?
2
votes
0answers
548 views

Custom objective function in xgboost for Regression

I am working on a regression problem, where I want to modify the loss function in xgboost library such that my predictions should never be lesser than the actual value. I have written this code: <...
1
vote
0answers
192 views

How does XGBoost implement MAE loss?

As we all know, XGBoost constructs trees based on gradient. I wonder how does XGBoost define gradient of MAE loss, as MAE itself is not differentiable. After some digging of the source code, I found ...
0
votes
1answer
6k views

How to print feature names in conjunction with feature Importance using Imbalanced-learn library?

I used BalancedBaggingClassifier from imblearn library to do an unbalanced classification task. How can I get feature improtance of the estimator in conjunction ...
2
votes
1answer
562 views

What does xgb's scale_pos_weight parameter do for regression?

From other posts (see Unbalanced multiclass data with XGBoost) and the documentation, scale_pos_weight in XGBoost appears to balance positive and negative cases, ...
3
votes
2answers
1k views

XGBoost Classification Probabilities higher than RF or SVM?

I am using Random Forests, XGBoost and SVMs to classify whether the home team wins or the away team wins their bowl game (in college football). I trained the models on all the games during the season. ...
1
vote
1answer
3k views

How to select features based on feature importance using SelectFromModel?

I would appreciate if you could let me know how to select features based on feature importance using SelectFromModel. I wrote: ...
4
votes
2answers
4k views

Prediction Intervals Using XGBoost

I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this blog, and not really understanding how the ...
1
vote
0answers
33 views

What gradient boosting library is the easiest to implement on mobile?

I think of implementing training and predicting in an app (both Android and iOS), but existing packages I found not seem to be very mobile-friendly (scikit-learn, xgboost, lightgbm). Random forest ...
3
votes
3answers
3k views

How to train a xgboost model on data that is too big for the memory?

What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more ...
0
votes
1answer
158 views

XGBoost Predictions

I am working on a multi-class classification task for 24 classes using XGBoost. I am training the model as follows: ...
2
votes
1answer
2k views

XGBoost Predictions all the same

When I evaluate the model I seem to be getting a decent RMSE score but when I try to actually see the predictions when I call the model all my values are the same. ...
0
votes
1answer
41 views

The best w_j confusion in xgboost

from XGBoost tutorial, it described: In this equation $w_j$ are independent with respect to each other, the form $G_j w_j + \frac{1}{2}(H_j+λ)w_j^2$ is quadratic and the best $w_j$ for a given ...
1
vote
2answers
845 views

XGBoost ranking file format

The xgboost package has two files that must be used for ranking: train.txt with the data ...
7
votes
2answers
4k views

XGBoost Feature importance - Gain and Cover are high but Frequency is low

I have read this question: How do i interpret the output of XGBoost importance? about the three different types of feature importances: frequency (called "weight" in Python XGBoost), gain, and cover. ...
-2
votes
1answer
1k views

Training XGBoost sequentially

I'm currently tring to train a model with XGBoost. My dataset has ~7 million records and 61 columns. The problem I'm currently having is that I get a MemoryError on python when I try to train the ...
25
votes
3answers
19k views

Why do we need XGBoost and Random Forest?

I wasn't clear on couple of concepts: XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ? ...
0
votes
1answer
7k views

XGBoost Fit vs Train

I am trying to do a grid searching using the methodology that mentioned in this post. However, I found that XGBClassifier().fit() is using much more memory than <...
1
vote
1answer
1k views

xgboost with tree_method = 'hist' in R

According to a benchmark of GBM vs. xgboost vs. LightGBM (https://www.kaggle.com/nschneider/gbm-vs-xgboost-vs-lightgbm) it is possible to implenet xgboost with the argument ...
3
votes
2answers
4k views

Overfitting XGBoost

I try to classify data from a dataset of 315 lines and 17 (real data) features (315x17). The target value is either "good" or "bad" (binary classification). I used XGBoost to classify these data, ...
0
votes
1answer
197 views

Do xgboost and random forests in general handle multiple splits of the same numeric feature in a single branch?

For example, let's say that the age (say $x$) of a person for $12< x< 25$ can be used to predict computer usage to a high degree of certainty. In a decision tree, this could be represented by a ...