Feature importance score for a feature that contains mostly 0's in XGBoost

Question

I have read that the feature importance scores are calculated based on how a split on that feature improves performance.

I have a binary classification dataset and am running XGBoost classifier on it. There is 1 feature that has the value 0 for 1425 rows out of the 1431 rows. Out of the 5 rows for which it has data, 3 of them have a label 1 and 2 have the label 0. I would assume this feature, due to just how sparse it is and the fact that it has only 0's would have a very low feature importance score. However relative to other features, it has the 2nd highest score of 0.14416. In the test set, the model predictions are as follows:

	Class 0	Class 1
Class 0	227	23
Class 1	18	19

Could anyone please explain the intuition as to why this feature ends up getting a high importance? Does feature importance get biased for features that have high number of the same value?

Some context:

The target is if the response time of a server is high or low and this feature is the run time of a particular dunction. But that function turns out to run at very few hours this having 0's for most rows

Answer 1

Some documentation on the exact meaning of feature importance is here:https://xgboost.readthedocs.io/en/stable/R-package/discoverYourData.html#feature-importance . The default

There are a couple of things that should be mentioned here. First, you say that this feature has a value of 0 for 1425 rows, and that it "has data" for 5 rows. The value of 0 is also data, unless this is actually some kind of "missing data" label! In any case, XGBoost will just consider the value 0 to be a normal value.

Secondly, you don't give the distribution of the class labels for the 0 valued items, but of the other, 3/5 have class 1, i.e. 0.6 probability of class 1. If most of the items are class 0 then the overall probability of class 1 will be much lower - less than 0.1 in the test sample - so this is a relatively good predictive feature for these 5 cases. As for this having the 2nd highest feature importance, it might just be that the other features increased the accuracy by even less. You should try to see if that makes sense by exploring the data.

Answer 2

Feature importance has nothing to do with sparsity, but as its name suggest, how important it is when predicting. Also remember that sometimes even if a feature does not show direct correlation with the target, combining it with other features may.

To verify, try remove that feature, train another model and compare the performance.

Answer 3

If it correctly helps predict 3 one out of 19, it might be an extremely important feature. Now that you didn't provide us with any detail about his feature an about the target, we can really help you that much; Is there any rationale that would link that feature to the target ? is the question you'll need to answer.