Xgboost interpretation: shouldn't cover, frequency, and gain be similar?

Question

I was surprised to see the results of my feature importance table from my xgboost model. Based on the tutorials that I've seen online, gain/cover/frequency seems to be somewhat similar (as I would expect because if a variable improves accuracy, shouldn't it increase in frequency as well?) but my numbers are drastically different. Am I perhaps doing something wrong or is my intuition wrong? Thank you in advance!

Feature       Gain      Cover   Frequency

1:    Var1 0.21943765 0.02821822 0.009433962
2:    Var2 0.18207910 0.05509272 0.066037736
3:    Var3 0.10746529 0.22117710 0.216981132
4:    Var4 0.10294292 0.05267401 0.018867925
5:    Var5 0.06928732 0.10185434 0.141509434
6:    Var6 0.05745753 0.05482397 0.047169811

Answer 1

My layman's understanding of those metrics as follows:

• Gain = (some measure of) improvement in overall model accuracy by using the feature
• Frequency = how often the feature is used in the model.

It's important to remember that the algorithm builds sequentially, so the two metrics are not always directly comparable / correlated.

An example (2 scenarios):

1. Var1 is relatively predictive of the response. It is included by the algorithm and its "Gain" is relatively high. Once its link to the response has been captured it might not be used again - e.g. there may be other features which are more predictive at later stages of modelling or all of Var1's link to the response may have been captured - and so its "Frequency" is low.

2. Var1 is extremely predictive across the whole range of response values. We can expect that Var1 will have high "Gain". Now, since Var1 is so predictive it might be fitted repeatedly (each time using a different split) and so will also have a high "Frequency".

In most cases, we prioritise accuracy and so will likely prioritise "Gain" over "Frequency", but if you're using the algorithm for feature selection then it may be a good idea to use a mixture of both to inform your decision, much like @bbennett36 suggested.

Answer 2

Gain = Total gains of splits which use the feature. (In my opinion, features with high gain are usually the most important features)

Frequency = Numbers of times the feature is used in a model.

In my experience, these values are not usually correlated all of the time. I have had situations where a feature has the most gain but it was barely checked so there wasn't alot of 'frequency'.

Also, binary coded variables don't usually have high frequency because there is only 2 possible values. When it comes continuous variables, the model usually is checking for certain ranges so it needs to look at this feature multiple times usually resulting in high frequency.

For future reference, I usually just check the top 20 features by gain, and top 20 by frequency. If a feature appears in both then it is important in my opinion. Also, I wouldn't really worry about 'cover'. I don't think there is much to learn from that.