43
From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees.
The meaning of the importance data table is as follows:
The Gain implies the relative contribution of ...
42
Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned.
Caret
See this answer on Cross Validated for a thorough explanation on how to use the caret package for hyperparameter search on xgboost.
How to tune hyperparameters of xgboost trees?
Custom Grid Search
I ...
42
Quote from the author of xgboost:
Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.
We have updated a comprehensive tutorial on introduction to the model, ...
38
It's easier to start with your second question and then go to the first.
Bagging
Random Forest is a bagging algorithm. It reduces variance.
Say that you have very unreliable models, such as Decision Trees. (Why unreliable? Because if you change your data a little bit, the decision tree created can be very different.) In such a case, you can build a robust ...
34
Just add weights based on your time labels to your xgb.DMatrix.
The following example is written in R but the same principle applies to xgboost on Python or Julia.
data <- data.frame(feature = rep(5, 5),
year = seq(2011, 2015),
target = c(1, 0, 1, 0, 0))
weightsData <- 1 + (data$year - max(data$year)) * 5 * 0.01
...
33
Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features.
Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. ...
25
Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate.
My guess is that the biggest effect ...
22
LightGBM is a great implementation that is similar to XGBoost but varies in a few specific ways, especially in how it creates the trees.
It offers some different parameters but most of them are very similar to their XGBoost counterparts.
If you use the same parameters, you almost always get a very close score. In most cases, the training will be 2-10 times ...
21
It seems that XGBoost uses regression trees as base learners by default. XGBoost (or Gradient boosting in general) work by combining multiple of these base learners. Regression trees can not extrapolate the patterns in the training data, so any input above 3 or below 1 will not be predicted correctly in your case. Your model is trained to predict outputs for ...
20
xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In xgboost.train, boosting iterations (i.e. n_estimators) is controlled by num_boost_round(default: 10)
In your case, the first code will do 10 iterations (by default), but the second one will do 1000 iterations. There won't be any big difference if you try to change clf = ...
19
You can use the dataframe's .values method to access raw data once you have manipulated the columns as you need them.
E.g.
train = pd.read_csv("train.csv")
target = train['target']
train = train.drop(['ID','target'],axis=1)
test = pd.read_csv("test.csv")
test = test.drop(['ID'],axis=1)
xgtrain = xgb.DMatrix(train.values, target.values)
xgtest = xgb....
19
scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is:
sum(negative instances) / sum(positive instances)
For your specific case, there is another option in order to weight individual data points and take their weights ...
18
On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but ...
15
On Python you have a nice scikit-learn wrapper, so you can write just like this:
import xgboost as xgb
exgb_classifier = xgb.XGBClassifier()
exgb_classifier.fit(X, y, sample_weight=sample_weights_data)
More information you can receive from this:
http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier.fit
15
Just use predict_proba instead of predict. You can leave the objective as binary:logistic.
14
Know I'm a bit late, but to get probabilities from xgboost you should specify multi:softmax objective like this:
xgboost(param, data = x_mat, label = y_mat,nround = 3000, objective='multi:softprob')
From the ?xgb.train:
multi:softprob same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The ...
13
In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization:
Use of sparse matrices with sparsity aware algorithms
Improved data structures for better processor cache utilization which makes ...
12
The procedure is described in their paper, section 3.4: Sparsity aware split-finding.
Assume you're at your node with 50 observations and, for the sake of simplicity, that there's only one split point possible.
For example, you have only one binary feature $x$, and your data can be split in three groups:
Group $B$: 20 observations such that $x=B$,
Group $...
11
One very important difference is xgboost has implemented DART, the dropout regularization for regression trees.
References
Rashmi, K. V., & Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.
11
Let's define first Feature Engineering:
Feature selection
Feature extraction
Adding features through domain expertise
XGBoost does (1) for you.
XGBoost does not do (2)/(3) for you.
So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.
11
XGBoost now has a histogram binning option for tree growth similar to the one LightGBM uses. It provides about the same level of speedup and similar accuracy characteristics, although the algorithms are still not exactly the same.
There are some plots and tables here showing how they are right on top of each other now.
https://github.com/dmlc/xgboost/issues/...
11
If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow trees to their full depth, order matters: application of early stopping criteria and pruning methods can result in very different trees. Because leaf-wise ...
10
You could try building multiple xgboost models, with some of them being limited to more recent data, then weighting those results together. Another idea would be to make a customized evaluation metric that penalizes recent points more heavily which would give them more importance.
10
My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth.
This is only perfectly correct for very large, near infinite data sets, where the number of samples in your training set gives good coverage of all variations. In practice, with enough dimensions ...
9
According to the XGBoost documentation, XGboost expects:
the examples of a same group to be consecutive examples,
a list with the size of each group (which you can set with set_group method of DMatrix in Python).
9
I was curious about this and made a few tests.
I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is higher than a certain threshold.
Then, I’ve added multiple columns highly correlated to x, ran the same model, and observed the same values.
It seems that when ...
9
There is an answer from Tianqi Chen (2018).
This difference has an impact on a corner case in feature importance analysis: the correlated features.
Imagine two features perfectly correlated, feature A and feature B. For one specific tree, if the algorithm needs one of them, it will choose randomly (true in both boosting and Random Forests™).
...
9
XGBRegressor is for continuous target/outcome variables. These are often called "regression problems."
XGBClassifier is for categorical target/outcome variables. These are often called "classification problems."
9
You can use the ELI5 library to explain the feature contributions to individual predictions for XGBoost models.
See Explaining Predictions in the docs, copied below:
To get a better idea of how our classifier works, let’s examine
individual predictions with eli5.show_prediction():
from eli5 import show_prediction
show_prediction(clf, valid_xs[1], vec=...
8
I won't go into details but the following should help you grasp the idea.
They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ (sorted), you can try the $10$-quantiles split points $\{x_{10}, x_{20}, \cdots, x_{90}\}$ and have a good approximation already. This is what the $\epsilon$ ...
Only top voted, non community-wiki answers of a minimum length are eligible
