I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 classes. I have now about 300 features in the model.
Then I take an output model and count for each feature
- In how many trees a feature was present
- In how many splits a feature was present
Then I sort the features from the ones with most trees to least trees.
I get some results that I am not sure of. Features at the top of the list with most trees and splits basically appear in each single xgboost tree multiple times. For example in xgboost with 100 rounds and colsample_bytree=1.0
and max_depth=6
I would see a feature A
appear in 100 trees and in ~400 splits. Feature B
appears in 98 trees and 350 splits etc... Basically it seems that all my trees are based on all the same same top features in different configurations.
Question:
Does it mean:
- It normal
- Do those features overfit my model
- I tried to force model to take other features by decreasing
colsample_bytree
orcolsample_bylevel
and it helps but somewhat, but model performance does not dramattically improve.
Any other suggestions?
Update Observations (Feb 2019)
- The features are continuous not categorical
- When I sort all features based on their total gain (sum of gain in all nodes that split on the feature), the features that appear in all trees multiple times can have highest total gain or quite low total gain. There is not rule here.