5
$\begingroup$

I am training an XGboost model for binary classification on around 60 sparse numeric features. After training, the feature importance distribution has one feature with importance > 0.6, and all the rest with importance <0.05.

I remove the most important feature, and retrain. The same distribution forms; the most important feature has importance > 0.6, and the rest have < 0.05. I continued to remove the most important feature and retrain, remove and retrain, remove and retrain, etc. My f1-score started to drop, but every time there was one feature more important than the rest.

Also worth noting, when I removed the most important feature and retrained, the new most important feature was not the second most important feature from the previous training.

I cannot explain this behaviour intuitively. Does anyone know why this pattern arises?

Improve this question
$\endgroup$
2
  • $\begingroup$ what metric are you using for feature importance? there are lot:r “gain”, “weight”, “cover”, “total_gain” or “total_cover”. From xgboost.readthedocs.io/en/latest/python/python_api.html $\endgroup$
    – Carlos Mougan
    Commented Aug 10, 2020 at 16:32
  • 1
    $\begingroup$ Thanks @CarlosMougan, I am using the default parameter, so 'weight', according to the docs. $\endgroup$
    – Alan
    Commented Aug 11, 2020 at 9:25

1 Answer 1

Reset to default
3
$\begingroup$

According to the docs "gain" is the default, which is calculated by this formula, where I don't think intuition would help. General consensus is, that feature importance is a tricky concept, as long as your model performs fine, you shouldn't worry about it. Especially with the linked formula for gain you can see that the first split may be the most important by far, depending on gamma, so that every other feature importance is low by default.

If, on the other hand you chose "weight", that would be just be the number of times the feature is chosen to split. That may mean that one feature contains a lot of unique values and can be chosen a lot of times and still improve accuracy.

All of this depends heavily on the data and parameters you are using. There isn't any general answer to this pattern.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.