Questions tagged [probability-calibration]
The probability-calibration tag has no usage guidance.
34
questions
0
votes
0answers
28 views
Implementing Smoothed Isotonic Regression
In the paper here the authors suggest a new way of calibrating classifiers, called Smoothed Isotonic Regression (Algorithm 1).
As I follow the algorithm along, I noticed a problem in lines 19-20: ...
1
vote
0answers
14 views
Account for imbalanced data in a Neural Network using prior distribution
I have a dataset with 4 classes, say their distribution in the training-set is
$P_{prior}(C1) = 60\% $
$P_{prior}(C2) = 25\% $
$P_{prior}(C3) = 10\% $
$P_{prior}(C4) = 5\% $
After training a Neural ...
0
votes
0answers
17 views
Calibration of a few binary classifiers is not perfect - why?
I am working on a binary classifier using LightGBM. I try to see the results of the classifiers when changing the costs of false positives and false negatives, still working on the same training and ...
0
votes
0answers
29 views
Calibration curve motivation
I struggle to understand the mathematical motivation for the binary classification model calibration curve.
Why do we assume that the predicted probabilities should be consistent with the proportion ...
0
votes
1answer
66 views
Calibration Curve Error
I want to calibrate probability outputs of a model. I'm using Isotonic Regression. After calibration, when I called calibration_curve function of sklearn calibration module I got this error: ...
2
votes
1answer
204 views
Calibrating probability thresholds for multiclass classification
I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch ...
0
votes
0answers
22 views
How to ouput buckets of probabilities?
I am dealing with an unbalanced binary classification problem. The problem is so unbalanced (2:98) and hard to predict that I am interested in probability of the positive outcome instead of trying to ...
1
vote
0answers
73 views
xgboost calibration kde plots (isotonic) not smooth
i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
2
votes
1answer
62 views
How can i tell if my model is overfitting from the distribution of predicted probabilities?
all,
i am training light gradient boosting and have used all of the necessary parameters to help in over fitting.i plot the predicted probabilities (i..e probabililty has cancer) distribution from the ...
2
votes
1answer
114 views
why does my calibration curve for platts and isotonic have less points than my uncalibrated model?
i train a model using grid search then i use the best parameters from this to define my chosen model.
...
1
vote
0answers
34 views
calibrating classifier probabilities for unbalanced data when class ratios are unknown
I've built a binary classification convolutional neutral network, trained on simulated data with equal numbers of simulations for each class. I've obtained good results for a validation set with equal ...
2
votes
1answer
34 views
How to determine the correct target for classification probability when the observed samples are probabilities of each class?
I have data in which each event's outcome can be described by a probability of a categorical occurrence. For example, if all of the possible class outcomes are A, B, C, or D suppose in one event 7/10 ...
1
vote
0answers
63 views
Imbalanced text classification by oversampling: correction of class predicted probability by prior probability
My dataset has 3 class and 900 examples for training. Class distribution is 255, 185, and 460.
I found that if I oversample (random) the training data then I have to correct/calibrate the predicted ...
2
votes
0answers
130 views
Platt Scaling vs Isotonic Regression for reliability curve
I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
3
votes
1answer
38 views
I have 3 graphs of a binary Logistic Regression that I want to understand better what is happening and learn of a strategy to make the model better
My problem is the following: I have a binary Logistic Regression model with a very imbalanced dataset that outputs the percentage of the prediction. As can be seen in the images, as the threshold is ...
5
votes
1answer
910 views
XGBoost: how to adjust the probabilities of a binary classifier to match training data?
Training and testing data have around 1% positives, but the model predicts only around 0.1% as positives.
The model is an xgboost classifier.
I’ve tried calibration but it didn’t improve much. I ...
7
votes
1answer
263 views
Are calibrated probabilities always more reliable?
EDIT: Based on the answer below, I have updated the question and added more detail.
I have applied Dirichlet calibration to my fast-bert sentiment classification model, and I am struggling to really ...
1
vote
1answer
344 views
Adjust predicted probability after smote
i have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model.
I have two ...
1
vote
1answer
961 views
XGBoost probability distribution tending towards the extreme
I am using an XGBoost classifier to make risk predictions, and I see that even if it has very good binary classification results, the probability outputs are mainly under $0.05$ or over $0.95$ (like ...
2
votes
2answers
331 views
Compare scores of models
We got several models with predictions. How can we compare scores of different models with each other?
We assume that we got xgboost models and scores distribution can be different for each model, so ...
1
vote
1answer
1k views
Probability calibration is worsening my model performance
I'm using RandomForest and XGBoost for binary classification, and my task is to predict probabilities for each class. Since tree-based models are bad with outputting usable probabilities, i imported ...
2
votes
1answer
594 views
Predict_proba on a binary classification problem
I have a binary classification task on my hands, i have a bunch of people that i need to classify as being ones or zeros and then use predict_proba to estimate how confident my prediction was on the ...
1
vote
0answers
19 views
How to evaluate a model based on the effect of one important variable?
We want a well-calibrated classifier that tells us the probability of an event. The model has multiple inputs, but we are interested in how the probability of an event changes as we vary one input. ...
5
votes
2answers
3k views
convert predict_proba results using class_weight in training
As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I penalize the misclassification of a ...
1
vote
2answers
49 views
Determining threshold in an area with very few samples of positive label
I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...
1
vote
0answers
119 views
Probability Calibration : For 2D image data, how to use the calibration?
I have a model which takes 2D input data and does multi class classification in keras. I would like to plot the probability calibration curve. However, using the scikit function, it returns an error ...
3
votes
0answers
379 views
How to explain a Calibration Plot for many models?
I have a heavy imbalanced dataset with a classification problem. I try to plot the Calibration Curve from the sklearn.calibration package. In specific, I try the ...
2
votes
1answer
103 views
Does PMML support probability calibration?
As I need to port a decision tree model from Python to Java, I would like to know whether PMML (Predictive Model Markup Language) supports probability calibration.
1
vote
1answer
825 views
Calibrate the predicted class probability to make it represent a true probability?
Let's say that we have a simple binary classification model (a neural network -- NN) for classifying input images as "dog" ($y=1$) or "not dog" ($y=0$). Let's assume that the NN has one "sigmoid ...
1
vote
2answers
764 views
Do I need to correct predict_proba by training fraction?
Many algorithms provide a predict_proba function indicating probability of a case to belong to that class (e.g. https://scikit-learn.org/stable/modules/generated/...
6
votes
2answers
3k views
Probability Calibration : role of hidden layer in Neural Network
I try a simple Neural Network (Logistic Regression) to play with Keras.
In input I have 5,000 features (output of a simple tf-idf vectorizer) and in the output layer I just use a random uniform ...
9
votes
1answer
7k views
Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?
1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event ...
18
votes
4answers
1k views
XGBoost outputs tend towards the extremes
I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature ...
4
votes
1answer
159 views
which loss function (if any) optimizes the calibration graph
The calibration graph is the predicted versus actual probability(see http://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html). Is it possible to optimize the ...
