I am training an XGBoost model,
xgb.XGBRegressor() with 13 features and one numeric target. The R2 on the test set is 0.935, which is good. I am checking the feature importance by
for col,score in zip(X_train.columns,xgbr.feature_importances_): print(col,score)
When I check the importance type by
xgbr.importance_type, the result is
I have a feature,
x1, whose importance seems to be 0.0068, not so high.
x1 is a categorical feature with a cardinality of 5122, and I apply
LabelEncoder before training the model.
I remove this feature from training set, and retrain the model with the same hyperparameters and the same training-testing set. The R2 seems to have a big hit and falls down to 0.885.
Why does a seemingly unimportant feature have such a big impact?