0
$\begingroup$

I looking at football data and trying to predict whether a goal will occur using xgboost with objective binary: logistic.

My data is 1:10 unbalanced with no goals being more dominant. I have used smote or task.over in mlr package to oversample (with a factor of 4).

I train the model, tune and cross validate but the predictions seems reasonable (low aucpr of 30% but high in other stats).

However when I look at the probabilities predicted by the model, it much larger than the actual average. Is there anything that could cause this?

What are the probabilities? i.e class 1 probs are the probabilities that they are in class 1, so may not truly represent the probability to score a goal.

Thanks in advance

$\endgroup$
1
  • $\begingroup$ Can you share your code to see how these ideas were implemented? $\endgroup$ – Brian Spiering Feb 25 at 18:30
0
$\begingroup$

If the predicted outcome targets(i.e., probabilities in this example), are consistently higher than actual, it implies you have statistical bias model. Statistical bias is when there is a systematically different result from the population parameter being estimated.

There are many ways to reduce statistical bias:

  • Resample at more extreme ratio
  • Change the loss function to more heavily penalize mispredictions
  • Change algorithm
$\endgroup$
0
$\begingroup$

By oversampling, you have told your model that goals are more prevalent than they are, so of course its predicted probabilities are too high.

You can undo the shift in probabilities induced by resampling, see e.g. this post or this CV.SE one, or go for more general "probability calibration" methods, e.g. with our tag .

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.