I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, interactions. And it can handle both numerical and categorical variables and it also seems that redundant variables does not affect this method too much.

Usually, in predictive modeling, you may do some selection among all the features you have and you may also create some new features from the set of features you have. So select a subset of features means you think there are some redundancy in your set of features; create some new features from the current feature set means you do some functional transformations on your current features. Then, both of these two points should be covered in XGBoost. Then, does it mean that to use XGBoost, you only need to choose those tunning parameters wisely? What is the value of doing feature engineering using XGBoost?


Let's define first Feature Engineering:

  1. Feature selection
  2. Feature extraction
  3. Adding features through domain expertise

XGBoost does (1) for you. XGBoost does not do (2)/(3) for you.

So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.

  1. Feature selection: XGBoost does the feature selection up to a level. In my experience, I always do feature selection by a round of xgboost with parameters different than what I use for the final model. I typically use low numbers for row and feature sampling, and trees that are not deep and only keep the features that enter to the model. Then fine tune with another model. This prevented overfitting for me when the number of features was very high.
  2. Feature generation: XGBoost (classification, booster=gbtree) uses tree based methods. This means that the model would have hard time on picking relations such as ab, a/b and a+b for features a and b. I usually add the interaction between features by hand or select the right ones with some heuristics. Depending on the application, this can really boost the performance.

What is the value of doing feature engineering using XGBoost?

Performance maybe?

(Note we don't use XGBoost, but another gradient boosting library - though XGBoost's performance probably also depends on the dimensionality of the data in some way.)

We have a dataset where each item consists of 3 signals, each 6000 samples long - that's 18k features. Using these features directly takes ages (days), so we did some manual feature engineering to reduce the number of features to about 200. Now training (including parameter tuning) is a matter of a few hours.

For comparison: a short time ago we also started training ConvNets with the same data and the whole 18k features (no feature engineering). They reach the same accuracy as the gradient boosting models after only about 2 hours of training.


This is probably the best answer to your question from the guys who uses too much of xgboost and stacking: http://blog.kaggle.com/2017/03/17/outbrain-click-prediction-competition-winners-interview-2nd-place-team-brain-afk-darragh-marios-mathias-alexey/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.