Is feature engineering still useful when using XGBoost?

Question

I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, interactions. And it can handle both numerical and categorical variables and it also seems that redundant variables does not affect this method too much.

Usually, in predictive modeling, you may do some selection among all the features you have and you may also create some new features from the set of features you have. So select a subset of features means you think there are some redundancy in your set of features; create some new features from the current feature set means you do some functional transformations on your current features. Then, both of these two points should be covered in XGBoost. Then, does it mean that to use XGBoost, you only need to choose those tunning parameters wisely? What is the value of doing feature engineering using XGBoost?

Answer 1

Let's define first Feature Engineering:

1. Feature selection
2. Feature extraction
3. Adding features through domain expertise

XGBoost does (1) for you. XGBoost does not do (2)/(3) for you.

So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.

Answer 2

1. Feature selection: XGBoost does the feature selection up to a level. In my experience, I always do feature selection by a round of xgboost with parameters different than what I use for the final model. I typically use low numbers for row and feature sampling, and trees that are not deep and only keep the features that enter to the model. Then fine tune with another model. This prevented overfitting for me when the number of features was very high.
2. Feature generation: XGBoost (classification, booster=gbtree) uses tree based methods. This means that the model would have hard time on picking relations such as ab, a/b and a+b for features a and b. I usually add the interaction between features by hand or select the right ones with some heuristics. Depending on the application, this can really boost the performance.

Answer 3

What is the value of doing feature engineering using XGBoost?

Performance maybe?

(Note we don't use XGBoost, but another gradient boosting library - though XGBoost's performance probably also depends on the dimensionality of the data in some way.)

We have a dataset where each item consists of 3 signals, each 6000 samples long - that's 18k features. Using these features directly takes ages (days), so we did some manual feature engineering to reduce the number of features to about 200. Now training (including parameter tuning) is a matter of a few hours.

For comparison: a short time ago we also started training ConvNets with the same data and the whole 18k features (no feature engineering). They reach the same accuracy as the gradient boosting models after only about 2 hours of training.

Answer 4

An empirical answer to that question woud be to look at public kaggle competitions / notebooks (see here), where xgboost is heavily used as state of the art for tabular data problems.

The answer is yes without a doubt. Notably in competitions, feature engineering is the main way to make a difference (followed maybe by parameter tuning) with everyone else. If everyone was dumping the same dataset in the same xgboost model they would have the same results.

I think this can be restated as a Data Science "no free lunch theorem". It is fairly easy to install R / Python with the associated XGBoost library. But, if it was that easy to deal with Data Science problems, any one would be able to do it and there would not be so much people training or working in Data Science. Genereally speaking that means Data Science has hard parts you need to deal with. Feature engineering is one of those hard parts of Data Science that has no universal solution.