XGboost Classifier predicits different results for same samples depending on test dataset size

Question

I train a simple xgboost classifier model with the following lines.

xgb_model = xgb.XGBClassifier(objective="binary:logistic", random_state=42)
xgb_model.fit(X_train, y_train)
ypred_1 = xgb_model.predict(X_test_1)
ypred_2 = xgb_model.predict(X_test_2)

Then I use two test data sets, of which X_test_2 is a subset of X_test_1. When predicting the two test data sets, the model gives different predictions for some samples (which are identical in both data sets). Even if I run the predictions in batches, the different predicted samples differ depending on the batch size. Only when I predict both test data sets in samples, the predictions are the same for all samples. The same behavior was also observed when using XGboost.DMatrix.

Does anyone have an explanation for this behavior?

Answer 1

So the mistake was to use the DMatrix datatype while using categorical data in the dataset. I split the data and then stored it in a DMatrix with enable_categorical=True. This encoded the categorical data differently in the sub-dataset than in the entire test dataset, and due to that the predictions were different.

Better way to go: Encode before splitting and then store in DMatrix with enable_categorical=False.