Hot answers tagged

60 votes
Accepted

GBM vs XGBOOST? Key differences?

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
user avatar
  • 4,046
52 votes
Accepted

Does XGBoost handle multicollinearity by itself?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
user avatar
51 votes
Accepted

How to interpret the output of XGBoost importance?

From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names ...
user avatar
50 votes
Accepted

Why do we need XGBoost and Random Forest?

It's easier to start with your second question and then go to the first. Bagging Random Forest is a bagging algorithm. It reduces variance. Say that you have very unreliable models, such as ...
user avatar
42 votes
Accepted

Hypertuning XGBoost parameters

Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned. Caret See this answer on Cross Validated for a ...
user avatar
  • 3,260
41 votes

xgboost: give more importance to recent samples

Just add weights based on your time labels to your xgb.DMatrix. The following example is written in R but the same principle applies to xgboost on Python or Julia. ...
user avatar
  • 3,260
33 votes
Accepted

XGBRegressor vs. xgboost.train huge speed difference?

xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In <...
user avatar
  • 4,046
30 votes
Accepted

LightGBM vs XGBoost

LightGBM is a great implementation that is similar to XGBoost but varies in a few specific ways, especially in how it creates the trees. It offers some different parameters but most of them are very ...
user avatar
30 votes
Accepted

Lightgbm vs xgboost vs catboost

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do ...
user avatar
  • 1,834
28 votes
Accepted

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In ...
user avatar
  • 1,316
28 votes
Accepted

Is it necessary to normalize data for XGBoost?

Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require ...
user avatar
  • 1,800
26 votes
Accepted

Unbalanced multiclass data with XGBoost

scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a ...
user avatar
  • 411
24 votes

Pandas Dataframe to DMatrix

You can use the dataframe's .values method to access raw data once you have manipulated the columns as you need them. E.g. ...
user avatar
  • 27.4k
23 votes
Accepted

XGBoost Linear Regression output incorrect

It seems that XGBoost uses regression trees as base learners by default. XGBoost (or Gradient boosting in general) work by combining multiple of these base learners. Regression trees can not ...
user avatar
  • 1,861
23 votes

Does XGBoost handle multicollinearity by itself?

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
user avatar
22 votes

xgboost: give more importance to recent samples

On Python you have a nice scikit-learn wrapper, so you can write just like this: ...
user avatar
  • 321
22 votes

How to predict probabilities in xgboost using R?

Just use predict_proba instead of predict. You can leave the objective as binary:logistic.
user avatar
  • 1,277
22 votes

Does XGBoost handle multicollinearity by itself?

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, ...
user avatar
  • 330
20 votes
Accepted

L1 & L2 Regularization in Light GBM

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...
user avatar
  • 9,698
19 votes
Accepted

How does Xgboost learn what are the inputs for missing values?

The procedure is described in their paper, section 3.4: Sparsity aware split-finding. Assume you're at your node with 50 observations and, for the sake of simplicity, that there's only one split ...
user avatar
  • 1,316
18 votes

Unbalanced multiclass data with XGBoost

This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation. weight parameter in XGBoost is per ...
user avatar
  • 8,629
18 votes

GBM vs XGBOOST? Key differences?

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
user avatar
18 votes
Accepted

Decision trees: leaf-wise (best-first) and level-wise tree traverse

If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow ...
user avatar
  • 3,028
17 votes

How to predict probabilities in xgboost using R?

Know I'm a bit late, but to get probabilities from xgboost you should specify multi:softmax objective like this: ...
user avatar
  • 271
16 votes

Unbalanced multiclass data with XGBoost

For sklearn version < 0.19 Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of ...
user avatar
15 votes

Is feature engineering still useful when using XGBoost?

Let's define first Feature Engineering: Feature selection Feature extraction Adding features through domain expertise XGBoost does (1) for you. XGBoost does not do (2)/(3) for you. So you still ...
user avatar
14 votes

In XGBoost would we evaluate results with a Precision Recall curve vs ROC?

I am answering my own question, almost two years later. XGBoost now has a new eval metric aucpr. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters
user avatar
  • 415
14 votes

GBM vs XGBOOST? Key differences?

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, ...
user avatar
  • 1,320
14 votes

Is feature engineering still useful when using XGBoost?

Feature selection: XGBoost does the feature selection up to a level. In my experience, I always do feature selection by a round of xgboost with parameters different than what I use for the final model....
user avatar
  • 411
14 votes
Accepted

Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

It depends on the definition of accurate model, but in general the answer to your question 1) is No. Regarding your second question (based on results in the paper of Niculescu-Mizil & Caruana ...
user avatar
  • 1,470

Only top scored, non community-wiki answers of a minimum length are eligible