XGBoost Classification Probabilities higher than RF or SVM?

Question

I am using Random Forests, XGBoost and SVMs to classify whether the home team wins or the away team wins their bowl game (in college football). I trained the models on all the games during the season.

I've come across something that is a bit weird and can't explain. I calculated a prediction confidence by subtracting the class probabilities. The XGBoost confidence values are consistency higher than both Random Forests and SVM's. I've attached the image below.

I did some hyper-parameter tuning for all of my models and used the best parameters based on testing accuracy.

• Random Forest:
  • 700 trees
  • 15 variables randomly sampled (mtries)
  • minimum split criteria of 5 rows.
• XGBoost:
  • 0.5, Learn rate
  • gbtree as my booster
  • max depth of 6
• SVM:
  • RBF kernel
  • C (slack) of 1
  • 0.01, Sigma

I wasn't clear with my question: Why exactly does XGBoost prefer one class greatly to the other? In comparison to these other methods. I'm trying to figure out why my prediction confidences of a class are so high for XGboost.

Answer 1

Rather than answering why XGBoost give very confident predictions, I will answer why random forest and SVM give not-so-confident predictions.

Random forest probability estimates are given by the percentage of the forest that predicted a particular class. For example, if you have $100$ trees in your forest and $81$ of them predict some class for some example, the probability estimate for that example belonging to that class is calculated to be $\frac{81}{100} = 0.81$. Because of the random nature of the ensemble members, it's very unlikely that each individual tree will end up with the correct prediction, even if the majority do. This makes probability estimates from random forests shy away from the extreme ends of the scale.

SVM is a slightly different case, because they are unable to produce probability estimates directly. Typically, Platt scaling (essentially logistic regression) is used to scale the SVM output to a probability estimate. This has the added benefit of calibrating the probability estimates, meaning the predicted probability is quite accurate - in other words, if a probability of $0.8$ is given for a prediction, it actually has approximately an $80\%$ chance of being correct. For a problem like this where there's a lot of noise (underdog teams do win sometimes, and there's a lot of evenly matched games that are hard to predict), these predictions will tend not to be overconfident.

I don't have a good reason as to why XGBoost is possibly overconfident, but it has been observed in the past that additive boosting models tend to provide distorted probability estimates without applying post-training calibration e.g. here, here & here.

Answer 2

As a side note, you have not mentioned any regularisation parameters of xgboost, so I understand you don’t use any. In general, it is not good and might lead to overfitting.

Regarding your question, my hypothesis is that in your case xgboost classifier is simply more powerful than other two approaches and thus is more confident, which is indicated by higher probabilities assigned to particular classes.

Maybe xgboost is even overconfident, i.e. overfits, but one cannot be sure without thorough testing on unseen test data.