Skip to main content
14 votes
Accepted

Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

It depends on the definition of accurate model, but in general the answer to your question 1) is No. Regarding your second question (based on results in the paper of Niculescu-Mizil & Caruana ...
aivanov's user avatar
  • 1,510
6 votes
Accepted

Probability calibration is worsening my model performance

The probability calibration is just stacking a logistic or isotonic regression on top of the base classifier. The default is logistic, and since the sigmoid is a strictly increasing function, the ...
Ben Reiniger's user avatar
  • 11.8k
6 votes

Probability Calibration : role of hidden layer in Neural Network

It makes sense that a network with ReLU activations produces worse probabilities than a sigmoid activation when you think about how these activation functions interact with the unit in the output ...
timleathart's user avatar
  • 3,940
5 votes

Are calibrated probabilities always more reliable?

The results seem quite reasonable to me, but I cannot be sure based on your given information. In your table of results, you have shown 10 instances where the uncalibrated probability for the mixed ...
Meelis Kull's user avatar
4 votes

Probability Calibration : role of hidden layer in Neural Network

I spent way too long attempting to calibrate my keras probability outputs. It turned out to be very simple. The isotonic model does a great job (so much so it risks overfitting) ...
ErichBSchulz's user avatar
3 votes

Is there any way to artificially create a probability calibration for data coming from another model?

Sound like you are interested in controlling for some of the time varying coefficients in the Cox model using priors using Bayesian methods. Here are some references that might help you: Bayesian ...
Ralph Winters's user avatar
3 votes
Accepted

Xgboost model predicting extreme values for events and non-events | Overfitting

This is not necessarily overfitting, but it may indicate data leakage i.e you are passing information to the model that is not supposed to be there it may be: Information that is generated after the ...
Multivac's user avatar
  • 2,969
3 votes

convert predict_proba results using class_weight in training

The correction that you are talking about is called probability calibration -- you want the "real" probability of an observation being in each class, right? The two most common approaches to ...
Maia's user avatar
  • 51
2 votes

Does PMML support probability calibration?

I'm not too familiar with PMML, but probability calibration (or at least, the most well-known methods, Platt scaling and isotonic regression) can be viewed as a stacked ensemble, with the output of ...
Ben Reiniger's user avatar
  • 11.8k
2 votes
Accepted

Adjust predicted probability after smote

No, any adjustment to the probabilities will presumably be monotonic, so the rank-ordering of the predictions will be the same, so the AUC will be the same. See, e.g., https://datascience....
Ben Reiniger's user avatar
  • 11.8k
2 votes

XGBoost probability distribution tending towards the extreme

Since you're seeing probabilities concentrated near 0 and 1 (as is expected in gradient boosting), make sure you're not using Platt calibration; isotonic calibration is a better choice, or if you're ...
Ben Reiniger's user avatar
  • 11.8k
2 votes
Accepted

why does my calibration curve for platts and isotonic have less points than my uncalibrated model?

The default strategy for calibration_curve is 'uniform', i.e. each of the bins has equal ...
Ben Reiniger's user avatar
  • 11.8k
2 votes
Accepted

How to determine the correct target for classification probability when the observed samples are probabilities of each class?

Imho the "cleanest" option would be to train a probabilistic model on the original categorical target, then obtain the predicted probabilities for every category as the final "...
Erwan's user avatar
  • 25.4k
2 votes

convert predict_proba results using class_weight in training

First, you should consider not balancing the dataset. It may well be unnecessary for this problem. Now to your actual question. In sklearn, each decision tree ...
Ben Reiniger's user avatar
  • 11.8k
1 vote
Accepted

Medical Predictive AI Model, Calibration Curve axes representation order

The argument for putting predicted probability on the horizontal axis seems to be: When we plot, we typically put the predicted value on the vertical axis, and the horizontal axis is for the value ...
Dave's user avatar
  • 3,863
1 vote

Xgboost model predicting extreme values for events and non-events | Overfitting

A high score on the test set does not indicate overfitting. See Why 100% accuracy on test data is not good? ; you're not quite reaching perfect performance, but you're quite close, and in that seeing ...
Ben Reiniger's user avatar
  • 11.8k
1 vote

Precision vs probability

Precision is usually defined as a quality metric and describes (in binary classification) the fraction of correctly classified instances among all instances that it classified as True. There is a ...
Robin van Hoorn's user avatar
1 vote

Calibration Curve Error

It's hard to say without your data. I'd comment this, but it's a bit long and better formatted as an answer. The part of the source code that's relevant is quite short, so you can go through step by ...
Ben Reiniger's user avatar
  • 11.8k
1 vote
Accepted

Calibrating probability thresholds for multiclass classification

Disclaimer: This answer describes the thoughts I had about this problem, I don't offer any guarantee about their validity so use at your own risks ;) There are two distinct parts in this problem: ...
Erwan's user avatar
  • 25.4k
1 vote

I have 3 graphs of a binary Logistic Regression that I want to understand better what is happening and learn of a strategy to make the model better

Given a confusion matrix: ...
Noah Weber's user avatar
  • 5,669
1 vote

XGBoost: how to adjust the probabilities of a binary classifier to match training data?

The first (and easiest) option is to make sure that your model is calibrated in probabilites. In Python, it means that you should pass the option binary:logistic in your fitting method. The ...
Lucas Morin's user avatar
  • 2,196
1 vote

Compare scores of models

You included that probability-calibration tag, which is prescient: there are a few techniques, all called "probability calibration," which adjust the scores output ...
Ben Reiniger's user avatar
  • 11.8k
1 vote

Compare scores of models

I'm going to assume your using python and scikit-learn mostly because it has a method for providing model metrics. ...
Tasty213's user avatar
  • 439
1 vote
Accepted

Predict_proba on a binary classification problem

sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using ...
pcko1's user avatar
  • 3,940
1 vote

Calibrate the predicted class probability to make it represent a true probability?

"Not all classification models are naturally probabilistic" is the first line of this chapter [1] . This means that some models do output probabilities, and some do not. This is highly dependant on ...
CharlesG's user avatar
  • 267
1 vote

Do I need to correct predict_proba by training fraction?

It depends on the model, whether your training data is representative of the testing data, and possibly on how "easy" the classification problem is. But the short answer is that (probably) no ...
Ben Reiniger's user avatar
  • 11.8k
1 vote

XGBoost outputs tend towards the extremes

XGBoost is not naturally calibrated in probabilities. You need to use something like: objective = "binary:logistic" to make sure that the model's output can be ...
Lucas Morin's user avatar
  • 2,196
1 vote

XGBoost outputs tend towards the extremes

The first I'd ask would be "What is positive/negative ratios?". Just because I had a similar issue multiple times when classes were very imbalanced. If it's your case you can try to balance dataset or ...
i1bgv's user avatar
  • 116
1 vote

which loss function (if any) optimizes the calibration graph

Log-loss optimizes your predictions in terms of their probability so in essence, yes it should be optimizing your calibration curve. i.e. if you predict probability of 0% but it's actually labelled ...
gkennos's user avatar
  • 131

Only top scored, non community-wiki answers of a minimum length are eligible