1
$\begingroup$

I have read that the feature importance scores are calculated based on how a split on that feature improves performance.

I have a binary classification dataset and am running XGBoost classifier on it. There is 1 feature that has the value 0 for 1425 rows out of the 1431 rows. Out of the 5 rows for which it has data, 3 of them have a label 1 and 2 have the label 0. I would assume this feature, due to just how sparse it is and the fact that it has only 0's would have a very low feature importance score. However relative to other features, it has the 2nd highest score of 0.14416. In the test set, the model predictions are as follows:

Class 0 Class 1
Class 0 227 23
Class 1 18 19

Could anyone please explain the intuition as to why this feature ends up getting a high importance? Does feature importance get biased for features that have high number of the same value?

Some context:

The target is if the response time of a server is high or low and this feature is the run time of a particular dunction. But that function turns out to run at very few hours this having 0's for most rows

$\endgroup$
4
  • $\begingroup$ What do the predictions look like on the training set? $\endgroup$
    – Ben Reiniger
    Aug 4, 2023 at 18:26
  • $\begingroup$ Is the feature negatively weighted? $\endgroup$
    – M__
    Aug 4, 2023 at 18:34
  • $\begingroup$ Predictions are binary either class 0 or 1 $\endgroup$
    – Vjs
    Aug 5, 2023 at 3:59
  • $\begingroup$ @Vjs I meant, how's the confusion matrix? $\endgroup$
    – Ben Reiniger
    Aug 9, 2023 at 2:31

3 Answers 3

1
$\begingroup$

Some documentation on the exact meaning of feature importance is here:https://xgboost.readthedocs.io/en/stable/R-package/discoverYourData.html#feature-importance . The default

There are a couple of things that should be mentioned here. First, you say that this feature has a value of 0 for 1425 rows, and that it "has data" for 5 rows. The value of 0 is also data, unless this is actually some kind of "missing data" label! In any case, XGBoost will just consider the value 0 to be a normal value.

Secondly, you don't give the distribution of the class labels for the 0 valued items, but of the other, 3/5 have class 1, i.e. 0.6 probability of class 1. If most of the items are class 0 then the overall probability of class 1 will be much lower - less than 0.1 in the test sample - so this is a relatively good predictive feature for these 5 cases. As for this having the 2nd highest feature importance, it might just be that the other features increased the accuracy by even less. You should try to see if that makes sense by exploring the data.

$\endgroup$
1
$\begingroup$

Feature importance has nothing to do with sparsity, but as its name suggest, how important it is when predicting. Also remember that sometimes even if a feature does not show direct correlation with the target, combining it with other features may.

To verify, try remove that feature, train another model and compare the performance.

$\endgroup$
1
$\begingroup$

If it correctly helps predict 3 one out of 19, it might be an extremely important feature. Now that you didn't provide us with any detail about his feature an about the target, we can really help you that much; Is there any rationale that would link that feature to the target ? is the question you'll need to answer.

$\endgroup$
1
  • $\begingroup$ I'm sorry i did not understand what you mean by 3 one out of 19 ? The target is if the response time of a server is high or low and this feature is the run time of a particular dunction. But that function turns out to run at very few hours this having 0's for most rows $\endgroup$
    – Vjs
    Aug 5, 2023 at 4:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.