I am training an XGBoost model, xgbr, using xgb.XGBRegressor() with 13 features and one numeric target. The R2 on the test set is 0.935, which is good. I am checking the feature importance by

for col,score in zip(X_train.columns,xgbr.feature_importances_):

When I check the importance type by xgbr.importance_type, the result is gain.

I have a feature, x1, whose importance seems to be 0.0068, not so high. x1 is a categorical feature with a cardinality of 5122, and I apply LabelEncoder before training the model.

I remove this feature from training set, and retrain the model with the same hyperparameters and the same training-testing set. The R2 seems to have a big hit and falls down to 0.885.

Why does a seemingly unimportant feature have such a big impact?

  • $\begingroup$ What are (some of) the other feature importances, and how do their removal affect R2? $\endgroup$ Jan 15 at 5:07

With no more context, my main guess is that it is an effectt not of the features but the metric you are using. Remember that $R^2$ is nondecreasing, so it will be greater or equal as you add more predictors

I can recommend to use another metric like mean squared error and repeat the model evaluation.

For reference check: https://stats.stackexchange.com/questions/133089/why-does-adding-more-terms-into-a-linear-model-always-increase-the-r-squared-val

  • 2
    $\begingroup$ The question specifies "R2 on the test set", which doesn't need to be monotonic. And R2 is just a scaled mse, so switching to mse won't change the comparison. $\endgroup$ Jan 15 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.