I'm using XGBoost and all its doing is using the feature in the first column of my data. My feature importance chart correlates perfectly to the position of the feature in my xtrain. If I shuffle the columns in xtrain, the feature chart changes along witht he shuffle after re-running the model.

This suggests that XGBoost is "getting stuck" on the first feature. Is there a way to prevent this? I was thinking by limiting how many times the same feature can appear in a tree?

  • $\begingroup$ I don't think trying to force xgb to limit usage of a feature is the right approach here. If it always focuses on the first feature regardless of what that feature actually is, you've got a serious bug. $\endgroup$ – Ben Reiniger Jun 20 at 1:02
  • $\begingroup$ What happens if you fit a decision tree and you plot it? What do you see? $\endgroup$ – Carlos Mougan Jun 20 at 6:59

If you mean "how many times the same feature can appear in an [individual] tree", then you can use max_depth to indirectly limit the number of features included in a single tree, even down to one feature. Since XGBoost is designed to use weak learners, having a lower depth value is ok.

model = XGBClassifier(max_depth=n)

However, I think the problem is not that XGBoost is getting stuck on a single feature. For example, maybe the other columns contain little or no correlation to the label. How does your model perform when you remove that column entirely. That should tell you if you only have one good feature in your dataset.

| improve this answer | |
  • $\begingroup$ There is very little correlation between my features. And there is lower correlation betwen my features and labels - I only expect to achieve results that are a little better than a random guess. $\endgroup$ – xxanissrxx Jun 20 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.