0
$\begingroup$

I am fitting an XGBClassifier to a small dataset (32 subjects) and find that if I loop through the code 10 times the feature importances (gain) assigned to the features in the model varies slightly.

I am using the same hyperparameter values between each iteration, and have subsample and colsample set to the default of 1 to prevent any random variation between executions. I am using the scikit learn feature_importance_ function to extract the values from the fitted model.

Any ideas as to why this variation in feature importance could be occurring? Does this mean that some of my features may be correlated and is there a way to ensure the XGBoost outputs the same importance values each time it is called? Note that the prediction and the predicted probabilities are all constant across iterations: it is just the feature importances.

Thanks in advance!

$\endgroup$
2
  • 2
    $\begingroup$ Can you share the model definition code ? Probably it is because of random_state is not set. $\endgroup$ – Wickkiey Apr 22 at 6:14
  • $\begingroup$ Thanks for the reply! I do not set random_state in the model definition. However I did this deliberately thinking that the XGBoost will output the same results given the same data regardless of random_state. Do you have any idea where the variation could be coming from (i.e what the random state could be stabilizing)? $\endgroup$ – user15733888 Apr 22 at 15:17
0
$\begingroup$

Look at the 'max_features' parameter in the GradientBoosting Classifier in scikit learn, here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier

That means that the algorithm is fitting different trees to try and account for the residual information after each sequence of trees, and that the features for each tree can be randomly permuted based on this parameter's selection.

Scikit learn does say that this can be made deterministic by setting random_state...

$\endgroup$
0
$\begingroup$

According to the XGBClassifier parameters (link) some operations will be happens on top of randomness, like subsample feature_selector etc.

If we didn't set seed for random value everything different value will be chosen and different result we will get. (Not abrupt change is expected).

So to reproduce the same result, it is a best practice to set the seed parameter in XGBoost Classifiers.

Most of the SkLearn classes will have random_state parameter for similar purposes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.