14
votes
Accepted
Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?
It depends on the definition of accurate model, but in general the answer to your question 1) is No.
Regarding your second question (based on results in the paper of Niculescu-Mizil & Caruana ...
- 1,490
6
votes
Accepted
Probability calibration is worsening my model performance
The probability calibration is just stacking a logistic or isotonic regression on top of the base classifier. The default is logistic, and since the sigmoid is a strictly increasing function, the ...
- 11.2k
5
votes
Are calibrated probabilities always more reliable?
The results seem quite reasonable to me, but I cannot be sure based on your given information. In your table of results, you have shown 10 instances where the uncalibrated probability for the mixed ...
- 51
5
votes
Probability Calibration : role of hidden layer in Neural Network
It makes sense that a network with ReLU activations produces worse probabilities than a sigmoid activation when you think about how these activation functions interact with the unit in the output ...
- 3,920
3
votes
Is there any way to artificially create a probability calibration for data coming from another model?
Sound like you are interested in controlling for some of the time varying coefficients in the Cox model using priors using Bayesian methods. Here are some references that might help you:
Bayesian ...
- 421
3
votes
Accepted
Xgboost model predicting extreme values for events and non-events | Overfitting
This is not necessarily overfitting, but it may indicate data leakage i.e you are passing information to the model that is not supposed to be there it may be:
Information that is generated after the ...
- 2,909
3
votes
Probability Calibration : role of hidden layer in Neural Network
I spent way too long attempting to calibrate my keras probability outputs. It turned out to be very simple.
The isotonic model does a great job (so much so it risks overfitting)
...
- 131
3
votes
convert predict_proba results using class_weight in training
The correction that you are talking about is called probability calibration -- you want the "real" probability of an observation being in each class, right?
The two most common approaches to ...
- 51
2
votes
Does PMML support probability calibration?
I'm not too familiar with PMML, but probability calibration (or at least, the most well-known methods, Platt scaling and isotonic regression) can be viewed as a stacked ensemble, with the output of ...
- 11.2k
2
votes
Accepted
Adjust predicted probability after smote
No, any adjustment to the probabilities will presumably be monotonic, so the rank-ordering of the predictions will be the same, so the AUC will be the same.
See, e.g., https://datascience....
- 11.2k
2
votes
XGBoost probability distribution tending towards the extreme
Since you're seeing probabilities concentrated near 0 and 1 (as is expected in gradient boosting), make sure you're not using Platt calibration; isotonic calibration is a better choice, or if you're ...
- 11.2k
2
votes
convert predict_proba results using class_weight in training
First, you should consider not balancing the dataset. It may well be unnecessary for this problem.
Now to your actual question. In sklearn, each decision tree ...
- 11.2k
2
votes
Accepted
why does my calibration curve for platts and isotonic have less points than my uncalibrated model?
The default strategy for calibration_curve is 'uniform', i.e. each of the bins has equal ...
- 11.2k
2
votes
Accepted
How to determine the correct target for classification probability when the observed samples are probabilities of each class?
Imho the "cleanest" option would be to train a probabilistic model on the original categorical target, then obtain the predicted probabilities for every category as the final "...
- 25k
1
vote
Xgboost model predicting extreme values for events and non-events | Overfitting
A high score on the test set does not indicate overfitting.
See Why 100% accuracy on test data is not good? ; you're not quite reaching perfect performance, but you're quite close, and in that seeing ...
- 11.2k
1
vote
Precision vs probability
Precision is usually defined as a quality metric and describes (in binary classification) the fraction of correctly classified instances among all instances that it classified as True. There is a ...
- 164
1
vote
Calibration Curve Error
It's hard to say without your data. I'd comment this, but it's a bit long and better formatted as an answer.
The part of the source code that's relevant is quite short, so you can go through step by ...
- 11.2k
1
vote
Accepted
Calibrating probability thresholds for multiclass classification
Disclaimer: This answer describes the thoughts I had about this problem, I don't offer any guarantee about their validity so use at your own risks ;)
There are two distinct parts in this problem:
...
- 25k
1
vote
1
vote
XGBoost: how to adjust the probabilities of a binary classifier to match training data?
The first (and easiest) option is to make sure that your model is calibrated in probabilites. In Python, it means that you should pass the option binary:logistic in your fitting method.
The ...
- 2,567
1
vote
Compare scores of models
You included that probability-calibration tag, which is prescient: there are a few techniques, all called "probability calibration," which adjust the scores output ...
- 11.2k
1
vote
Compare scores of models
I'm going to assume your using python and scikit-learn mostly because it has a method for providing model metrics.
...
- 439
1
vote
Accepted
Predict_proba on a binary classification problem
sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using ...
- 3,920
1
vote
Calibrate the predicted class probability to make it represent a true probability?
"Not all classification models are naturally probabilistic" is the first line of this chapter [1] .
This means that some models do output probabilities, and some do not. This is highly dependant on ...
- 257
1
vote
Do I need to correct predict_proba by training fraction?
It depends on the model, whether your training data is representative of the testing data, and possibly on how "easy" the classification problem is. But the short answer is that (probably) no ...
- 11.2k
1
vote
XGBoost outputs tend towards the extremes
XGBoost is not naturally calibrated in probabilities. You need to use something like:
objective = "binary:logistic"
to make sure that the model's output can be ...
- 2,567
1
vote
XGBoost outputs tend towards the extremes
The first I'd ask would be "What is positive/negative ratios?". Just because I had a similar issue multiple times when classes were very imbalanced. If it's your case you can try to balance dataset or ...
- 116
1
vote
which loss function (if any) optimizes the calibration graph
Log-loss optimizes your predictions in terms of their probability so in essence, yes it should be optimizing your calibration curve. i.e. if you predict probability of 0% but it's actually labelled ...
- 131
Only top scored, non community-wiki answers of a minimum length are eligible