# Why does an unimportant feature has a big impact on R2 in XGBoost?

I am training an XGBoost model, xgbr, using xgb.XGBRegressor() with 13 features and one numeric target. The R2 on the test set is 0.935, which is good. I am checking the feature importance by

for col,score in zip(X_train.columns,xgbr.feature_importances_):
print(col,score)


When I check the importance type by xgbr.importance_type, the result is gain.

I have a feature, x1, whose importance seems to be 0.0068, not so high. x1 is a categorical feature with a cardinality of 5122, and I apply LabelEncoder before training the model.

I remove this feature from training set, and retrain the model with the same hyperparameters and the same training-testing set. The R2 seems to have a big hit and falls down to 0.885.

Why does a seemingly unimportant feature have such a big impact?

• What are (some of) the other feature importances, and how do their removal affect R2? Jan 15 at 5:07

With no more context, my main guess is that it is an effectt not of the features but the metric you are using. Remember that $$R^2$$ is nondecreasing, so it will be greater or equal as you add more predictors