Skip to main content

Questions tagged [probability-calibration]

The tag has no usage guidance.

22 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
0 answers
477 views

How to explain a Calibration Plot for many models?

I have a heavy imbalanced dataset with a classification problem. I try to plot the Calibration Curve from the sklearn.calibration package. In specific, I try the ...
Tasos's user avatar
  • 3,930
2 votes
0 answers
20 views

Why shouldn't post modeling calibration be done on the training set

I've read in multiple places that calibration on model results shouldn't be done on the training set (the set that the model is build on), but rather, on a set that the model have not seen. I failed ...
Yue Y's user avatar
  • 121
2 votes
0 answers
146 views

Imbalanced text classification by oversampling: correction of class predicted probability by prior probability

My dataset has 3 class and 900 examples for training. Class distribution is 255, 185, and 460. I found that if I oversample (random) the training data then I have to correct/calibrate the predicted ...
user3363813's user avatar
2 votes
0 answers
296 views

Platt Scaling vs Isotonic Regression for reliability curve

I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
yl637's user avatar
  • 21
1 vote
0 answers
28 views

Best metric to evaluate model probabilities

i'm trying to create ML model for binary classification problem with balanced dataset and i care mostly about probabilities. I was trying to search web and i find only advices to use AUC or logloss ...
Robert Chasnouski's user avatar
1 vote
0 answers
25 views

Predictions using calibrated classifer

I find myself asking alot of calibration related questions recently - but i cannot find adequate material on it! I am training a binary classifier to predict default. This probability will be used in ...
Maths12's user avatar
  • 526
1 vote
0 answers
16 views

Should I use "sample_weights" on a calibrator if I already used them while training the model (imbalanced dataset)?

I was wondering what is the right way to proceed when you are dealing with an imbalanced dataset and you want to use a calibrator. When I work with a single model and imbalanced datasets I usually ...
Jacobo O's user avatar
1 vote
0 answers
181 views

xgboost calibration kde plots (isotonic) not smooth

i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
Maths12's user avatar
  • 526
1 vote
0 answers
52 views

calibrating classifier probabilities for unbalanced data when class ratios are unknown

I've built a binary classification convolutional neutral network, trained on simulated data with equal numbers of simulations for each class. I've obtained good results for a validation set with equal ...
Graham501617's user avatar
1 vote
0 answers
19 views

How to evaluate a model based on the effect of one important variable?

We want a well-calibrated classifier that tells us the probability of an event. The model has multiple inputs, but we are interested in how the probability of an event changes as we vary one input. ...
Maia's user avatar
  • 51
1 vote
0 answers
186 views

Probability Calibration : For 2D image data, how to use the calibration?

I have a model which takes 2D input data and does multi class classification in keras. I would like to plot the probability calibration curve. However, using the scikit function, it returns an error ...
Sandeep Pandey's user avatar
0 votes
0 answers
6 views

Resampling calibration data

Is it appropriate to resample a calibration dataset, or should the class distribution be consistent with the observed data?
hmm's user avatar
  • 1
0 votes
0 answers
54 views

Is this the appropriate way to calculate a multiclass reliability diagram for model calibration?

I'm trying to generalize reliability diagrams [1] to a multiclass classifier and implement that using pytorch and pytorch-metrics. So far so good but I'm somewhat confused about the definition of ...
Nirro's user avatar
  • 101
0 votes
0 answers
86 views

XGBoost Classifier + Isotonic Regression leading to worse probability accuracy

I'm testing out an XGBoost Classifier with the goal of using the probabilities it predicts in production. I know that tree based model probabilities are often not calibrated well so I decided to test ...
Ted's user avatar
  • 1
0 votes
0 answers
75 views

Is it possible to perform probability calibration with a model with the best hyperparams?

If I use RandomizedSearchCV to find the optimal hyperparams of a model, can I create another model, with those parameters, to calibrate probabilities using CalibratedClassifierCV? The new model is not ...
Flavio Brienza's user avatar
0 votes
0 answers
197 views

Classifier calibration leading to worse outcome

I am trying to calibrate some classifiers to output more accurate probabilities. For this, I am using a sigmoid regression as implemented in ...
C.S.'s user avatar
  • 23
0 votes
1 answer
24 views

Help me name my problem – Online Ensemble Realibration

I have the following problem: k predictors (let's say A, B) . Each predicts a value and ...
MSKL's user avatar
  • 303
0 votes
0 answers
39 views

Model with no historical data

I need to develop a new credit default classification model for which there are a lot of features available but very few historical data (because it's a new activity launched by the company I work for)...
Anatole's user avatar
  • 181
0 votes
0 answers
29 views

Model recalibration on different dataset

I have a large dataset approximately 150k rows and 1500 of positive labels on which I can train my model for binary classification. And also I have the other dataset which is smaller and is comprised ...
James Flash's user avatar
0 votes
0 answers
98 views

Account for imbalanced data in a Neural Network using prior distribution

I have a dataset with 4 classes, say their distribution in the training-set is $P_{prior}(C1) = 60\% $ $P_{prior}(C2) = 25\% $ $P_{prior}(C3) = 10\% $ $P_{prior}(C4) = 5\% $ After training a Neural ...
CutePoison's user avatar
0 votes
0 answers
53 views

Calibration curve motivation

I struggle to understand the mathematical motivation for the binary classification model calibration curve. Why do we assume that the predicted probabilities should be consistent with the proportion ...
James Flash's user avatar
0 votes
0 answers
106 views

How to ouput buckets of probabilities?

I am dealing with an unbalanced binary classification problem. The problem is so unbalanced (2:98) and hard to predict that I am interested in probability of the positive outcome instead of trying to ...
Lucas Morin's user avatar
  • 2,289