Probabilities predicted by XGBoost

Question

I looking at football data and trying to predict whether a goal will occur using xgboost with objective binary: logistic.

My data is 1:10 unbalanced with no goals being more dominant. I have used smote or task.over in mlr package to oversample (with a factor of 4).

I train the model, tune and cross validate but the predictions seems reasonable (low aucpr of 30% but high in other stats).

However when I look at the probabilities predicted by the model, it much larger than the actual average. Is there anything that could cause this?

What are the probabilities? i.e class 1 probs are the probabilities that they are in class 1, so may not truly represent the probability to score a goal.

Thanks in advance

Answer 1

By oversampling, you have told your model that goals are more prevalent than they are, so of course its predicted probabilities are too high.

You can undo the shift in probabilities induced by resampling, see e.g. this post or this CV.SE one, or go for more general "probability calibration" methods, e.g. with our tag probability-calibration.

Answer 2

If the predicted outcome targets(i.e., probabilities in this example), are consistently higher than actual, it implies you have statistical bias model. Statistical bias is when there is a systematically different result from the population parameter being estimated.

There are many ways to reduce statistical bias:

• Resample at the more extreme ratio
• Change the loss function to more heavily penalize mispredictions
• Change the algorithm