I am working on a XGBoost model for fraud detection (2 class classification) using XGBoost v0.7 on Spark. I am looking at different aspects of the model to find important features but also to understand if my model is not doing something weird. There is much more tools for this in R or python, but spark version 0.7 is quite poor in this regard.
One of the aspects I looked at were trees themselves and their gain. Precisely I sum gain of all nodes in a tree and plot it in the function of a tree id. I get some surprisingly different plots for different types of fraud and different datasets. For example for three models of 400 trees on 3 different datasets:
Naively I would expect a normal plot to look like Model 2 where the first tree has highest gain and then each subsequent tree improves on all predecessors but its gain decreases as we are converging. But I might be wrong.
- How to interpret these kinds of plots? What do they tell me?
- Is it a useful metric to look at?
- Is it normal that some of the plots have a maximum not at the first trees but later around tree 170?