Hot answers tagged

56

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance. We have updated a comprehensive tutorial on introduction to the model, which ...


49

From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees. The meaning of the importance data table is as follows: The Gain implies the relative contribution of ...


48

It's easier to start with your second question and then go to the first. Bagging Random Forest is a bagging algorithm. It reduces variance. Say that you have very unreliable models, such as Decision Trees. (Why unreliable? Because if you change your data a little bit, the decision tree created can be very different.) In such a case, you can build a robust ...


45

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. ...


42

Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned. Caret See this answer on Cross Validated for a thorough explanation on how to use the caret package for hyperparameter search on xgboost. How to tune hyperparameters of xgboost trees? Custom Grid Search I ...


40

Just add weights based on your time labels to your xgb.DMatrix. The following example is written in R but the same principle applies to xgboost on Python or Julia. data <- data.frame(feature = rep(5, 5), year = seq(2011, 2015), target = c(1, 0, 1, 0, 0)) weightsData <- 1 + (data$year - max(data$year)) * 5 * 0.01 ...


32

xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In xgboost.train, boosting iterations (i.e. n_estimators) is controlled by num_boost_round(default: 10) In your case, the first code will do 10 iterations (by default), but the second one will do 1000 iterations. There won't be any big difference if you try to change clf = ...


30

LightGBM is a great implementation that is similar to XGBoost but varies in a few specific ways, especially in how it creates the trees. It offers some different parameters but most of them are very similar to their XGBoost counterparts. If you use the same parameters, you almost always get a very close score. In most cases, the training will be 2-10 times ...


30

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but ...


27

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate. My guess is that the biggest effect ...


25

scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is: sum(negative instances) / sum(positive instances) For your specific case, there is another option in order to weight individual data points and take their weights ...


24

You can use the dataframe's .values method to access raw data once you have manipulated the columns as you need them. E.g. train = pd.read_csv("train.csv") target = train['target'] train = train.drop(['ID','target'],axis=1) test = pd.read_csv("test.csv") test = test.drop(['ID'],axis=1) xgtrain = xgb.DMatrix(train.values, target.values) xgtest = xgb....


23

It seems that XGBoost uses regression trees as base learners by default. XGBoost (or Gradient boosting in general) work by combining multiple of these base learners. Regression trees can not extrapolate the patterns in the training data, so any input above 3 or below 1 will not be predicted correctly in your case. Your model is trained to predict outputs for ...


22

Just use predict_proba instead of predict. You can leave the objective as binary:logistic.


21

On Python you have a nice scikit-learn wrapper, so you can write just like this: import xgboost as xgb exgb_classifier = xgb.XGBClassifier() exgb_classifier.fit(X, y, sample_weight=sample_weights_data) More information you can receive from this: http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier.fit


21

Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require normalization for the inputs either. For corroboration, see also the thread Is Normalization necessary? at the XGBoost Github repo, where the answer by the lead ...


18

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is higher than a certain threshold. Then, I’ve added multiple columns highly correlated to x, ran the same model, and observed the same values. It seems that when ...


18

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, feature A and feature B. For one specific tree, if the algorithm needs one of them, it will choose randomly (true in both boosting and Random Forests™). ...


18

The procedure is described in their paper, section 3.4: Sparsity aware split-finding. Assume you're at your node with 50 observations and, for the sake of simplicity, that there's only one split point possible. For example, you have only one binary feature $x$, and your data can be split in three groups: Group $B$: 20 observations such that $x=B$, Group $...


18

This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation. weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing. For example, if we have three imbalanced classes with ratios class A = 10% class ...


17

Know I'm a bit late, but to get probabilities from xgboost you should specify multi:softmax objective like this: xgboost(param, data = x_mat, label = y_mat,nround = 3000, objective='multi:softprob') From the ?xgb.train: multi:softprob same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The ...


17

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization: Use of sparse matrices with sparsity aware algorithms Improved data structures for better processor cache utilization which makes ...


17

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So using both isn't unprecedented. Second, XGBoost and LightGBM have quite a number of hyperparameters that overlap in their purpose. Tree complexity can be ...


15

For sklearn version < 0.19 Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of sklearn then assign each row of the train data its appropriate weight. I assume here that the train data has the column class containing the class number. I assumed also that there are nb_classes that ...


15

Let's define first Feature Engineering: Feature selection Feature extraction Adding features through domain expertise XGBoost does (1) for you. XGBoost does not do (2)/(3) for you. So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.


14

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.


13

I am answering my own question, almost two years later. XGBoost now has a new eval metric aucpr. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters


13

Feature selection: XGBoost does the feature selection up to a level. In my experience, I always do feature selection by a round of xgboost with parameters different than what I use for the final model. I typically use low numbers for row and feature sampling, and trees that are not deep and only keep the features that enter to the model. Then fine tune with ...


13

XGBoost now has a histogram binning option for tree growth similar to the one LightGBM uses. It provides about the same level of speedup and similar accuracy characteristics, although the algorithms are still not exactly the same. There are some plots and tables here showing how they are right on top of each other now. https://github.com/dmlc/xgboost/issues/...


13

If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow trees to their full depth, order matters: application of early stopping criteria and pruning methods can result in very different trees. Because leaf-wise ...


Only top voted, non community-wiki answers of a minimum length are eligible