Questions tagged [probability-calibration]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
2 answers
82 views

Xgboost model predicting extreme values for events and non-events | Overfitting

Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that ...
Scott Grammilo's user avatar
0 votes
0 answers
33 views

XGBoost Classifier + Isotonic Regression leading to worse probability accuracy

I'm testing out an XGBoost Classifier with the goal of using the probabilities it predicts in production. I know that tree based model probabilities are often not calibrated well so I decided to test ...
Ted's user avatar
  • 1
0 votes
1 answer
39 views

Improve image classification model with trained generator

Is it possible to improve an image classification model with a generator (trained class conditionally). (so this is same source/target distribution and same source/target task, so not domain ...
InKodeWeTrust's user avatar
0 votes
0 answers
11 views

Comparing probability models for alignment

I have a probability model which predicts a probability for a binary classification problem. I am interested in how well the predicted probability aligns with the true probability. For instance, you ...
Moss Richardson's user avatar
0 votes
0 answers
38 views

Is it possible to perform probability calibration with a model with the best hyperparams?

If I use RandomizedSearchCV to find the optimal hyperparams of a model, can I create another model, with those parameters, to calibrate probabilities using CalibratedClassifierCV? The new model is not ...
Flavio Brienza's user avatar
0 votes
0 answers
13 views

find the parameter that minimize a multivariate distribution

i was trying to use scipy's minimize scalar to find the value of the parameter T that minimize the negative log-likelihood of a multivariate distribution with covariance matrix C. if i understood ...
Alucard's user avatar
  • 111
2 votes
0 answers
16 views

Why shouldn't post modeling calibration be done on the training set

I've read in multiple places that calibration on model results shouldn't be done on the training set (the set that the model is build on), but rather, on a set that the model have not seen. I failed ...
Yue Y's user avatar
  • 121
0 votes
0 answers
75 views

How to calibrate autoregressive model?

TL;DR What metric, and how to calibrate autoregressive language (deep learning) model? Background Usually there are several popular approach to calibrate a non-autoregressive model such as isotonic ...
Wakeme UpNow's user avatar
0 votes
0 answers
103 views

Classifier calibration leading to worse outcome

I am trying to calibrate some classifiers to output more accurate probabilities. For this, I am using a sigmoid regression as implemented in ...
C.S.'s user avatar
  • 23
0 votes
1 answer
182 views

Precision vs probability

Say I have a model which predicts a class $C_i$ from an input $X$, with a probability of 0.95 i.e $P(C_i| X)=0.95$. That would mean that if we do this over and over, then 95/100 times we would be ...
CutePoison's user avatar
0 votes
1 answer
24 views

Help me name my problem – Online Ensemble Realibration

I have the following problem: k predictors (let's say A, B) . Each predicts a value and ...
MSKL's user avatar
  • 303
0 votes
0 answers
35 views

Model with no historical data

I need to develop a new credit default classification model for which there are a lot of features available but very few historical data (because it's a new activity launched by the company I work for)...
Anatole's user avatar
  • 181
0 votes
1 answer
94 views

Is there any way to artificially create a probability calibration for data coming from another model?

I have predictions, which come from a survival model, this model gives me very low probabilities, and I am not sure if they fulfill the real probability of the phenomenon. For example, I calculate $P\...
Juan Esteban de la Calle's user avatar
1 vote
0 answers
27 views

Best metric to evaluate model probabilities

i'm trying to create ML model for binary classification problem with balanced dataset and i care mostly about probabilities. I was trying to search web and i find only advices to use AUC or logloss ...
Robert Chasnouski's user avatar
0 votes
0 answers
26 views

Model recalibration on different dataset

I have a large dataset approximately 150k rows and 1500 of positive labels on which I can train my model for binary classification. And also I have the other dataset which is smaller and is comprised ...
James Flash's user avatar
1 vote
0 answers
24 views

Predictions using calibrated classifer

I find myself asking alot of calibration related questions recently - but i cannot find adequate material on it! I am training a binary classifier to predict default. This probability will be used in ...
Maths12's user avatar
  • 506
1 vote
0 answers
14 views

Should I use "sample_weights" on a calibrator if I already used them while training the model (imbalanced dataset)?

I was wondering what is the right way to proceed when you are dealing with an imbalanced dataset and you want to use a calibrator. When I work with a single model and imbalanced datasets I usually ...
Jacobo O's user avatar
0 votes
0 answers
88 views

Account for imbalanced data in a Neural Network using prior distribution

I have a dataset with 4 classes, say their distribution in the training-set is $P_{prior}(C1) = 60\% $ $P_{prior}(C2) = 25\% $ $P_{prior}(C3) = 10\% $ $P_{prior}(C4) = 5\% $ After training a Neural ...
CutePoison's user avatar
0 votes
0 answers
53 views

Calibration curve motivation

I struggle to understand the mathematical motivation for the binary classification model calibration curve. Why do we assume that the predicted probabilities should be consistent with the proportion ...
James Flash's user avatar
0 votes
1 answer
356 views

Calibration Curve Error

I want to calibrate probability outputs of a model. I'm using Isotonic Regression. After calibration, when I called calibration_curve function of sklearn calibration module I got this error: ...
tkarahan's user avatar
  • 422
2 votes
1 answer
1k views

Calibrating probability thresholds for multiclass classification

I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch ...
machinery's user avatar
  • 236
0 votes
0 answers
85 views

How to ouput buckets of probabilities?

I am dealing with an unbalanced binary classification problem. The problem is so unbalanced (2:98) and hard to predict that I am interested in probability of the positive outcome instead of trying to ...
Lucas Morin's user avatar
  • 2,567
1 vote
0 answers
162 views

xgboost calibration kde plots (isotonic) not smooth

i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the ...
Maths12's user avatar
  • 506
2 votes
1 answer
106 views

How can i tell if my model is overfitting from the distribution of predicted probabilities?

all, i am training light gradient boosting and have used all of the necessary parameters to help in over fitting.i plot the predicted probabilities (i..e probabililty has cancer) distribution from the ...
Maths12's user avatar
  • 506
4 votes
1 answer
624 views

why does my calibration curve for platts and isotonic have less points than my uncalibrated model?

i train a model using grid search then i use the best parameters from this to define my chosen model. ...
Maths12's user avatar
  • 506
1 vote
0 answers
51 views

calibrating classifier probabilities for unbalanced data when class ratios are unknown

I've built a binary classification convolutional neutral network, trained on simulated data with equal numbers of simulations for each class. I've obtained good results for a validation set with equal ...
Graham501617's user avatar
3 votes
1 answer
144 views

How to determine the correct target for classification probability when the observed samples are probabilities of each class?

I have data in which each event's outcome can be described by a probability of a categorical occurrence. For example, if all of the possible class outcomes are A, B, C, or D suppose in one event 7/10 ...
user3327134's user avatar
2 votes
0 answers
132 views

Imbalanced text classification by oversampling: correction of class predicted probability by prior probability

My dataset has 3 class and 900 examples for training. Class distribution is 255, 185, and 460. I found that if I oversample (random) the training data then I have to correct/calibrate the predicted ...
user3363813's user avatar
2 votes
0 answers
266 views

Platt Scaling vs Isotonic Regression for reliability curve

I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
yl637's user avatar
  • 21
3 votes
1 answer
45 views

I have 3 graphs of a binary Logistic Regression that I want to understand better what is happening and learn of a strategy to make the model better

My problem is the following: I have a binary Logistic Regression model with a very imbalanced dataset that outputs the percentage of the prediction. As can be seen in the images, as the threshold is ...
Gabriel Almeida's user avatar
5 votes
1 answer
2k views

XGBoost: how to adjust the probabilities of a binary classifier to match training data?

Training and testing data have around 1% positives, but the model predicts only around 0.1% as positives. The model is an xgboost classifier. I’ve tried calibration but it didn’t improve much. I ...
Henrique Nader's user avatar
7 votes
1 answer
460 views

Are calibrated probabilities always more reliable?

EDIT: Based on the answer below, I have updated the question and added more detail. I have applied Dirichlet calibration to my fast-bert sentiment classification model, and I am struggling to really ...
Danyal Andriano's user avatar
1 vote
1 answer
751 views

Adjust predicted probability after smote

i have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model. I have two ...
anat's user avatar
  • 155
1 vote
1 answer
2k views

XGBoost probability distribution tending towards the extreme

I am using an XGBoost classifier to make risk predictions, and I see that even if it has very good binary classification results, the probability outputs are mainly under $0.05$ or over $0.95$ (like ...
Ismalyt's user avatar
  • 71
2 votes
2 answers
856 views

Compare scores of models

We got several models with predictions. How can we compare scores of different models with each other? We assume that we got xgboost models and scores distribution can be different for each model, so ...
Vladimir Ershov's user avatar
2 votes
1 answer
2k views

Probability calibration is worsening my model performance

I'm using RandomForest and XGBoost for binary classification, and my task is to predict probabilities for each class. Since tree-based models are bad with outputting usable probabilities, i imported ...
Blenz's user avatar
  • 2,044
2 votes
1 answer
1k views

Predict_proba on a binary classification problem

I have a binary classification task on my hands, i have a bunch of people that i need to classify as being ones or zeros and then use predict_proba to estimate how confident my prediction was on the ...
Blenz's user avatar
  • 2,044
1 vote
0 answers
19 views

How to evaluate a model based on the effect of one important variable?

We want a well-calibrated classifier that tells us the probability of an event. The model has multiple inputs, but we are interested in how the probability of an event changes as we vary one input. ...
Maia's user avatar
  • 51
5 votes
2 answers
4k views

convert predict_proba results using class_weight in training

As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I penalize the misclassification of a ...
srl's user avatar
  • 51
1 vote
2 answers
86 views

Determining threshold in an area with very few samples of positive label

I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...
Gnoevoet's user avatar
1 vote
0 answers
181 views

Probability Calibration : For 2D image data, how to use the calibration?

I have a model which takes 2D input data and does multi class classification in keras. I would like to plot the probability calibration curve. However, using the scikit function, it returns an error ...
Sandeep Pandey's user avatar
3 votes
0 answers
463 views

How to explain a Calibration Plot for many models?

I have a heavy imbalanced dataset with a classification problem. I try to plot the Calibration Curve from the sklearn.calibration package. In specific, I try the ...
Tasos's user avatar
  • 3,890
2 votes
1 answer
148 views

Does PMML support probability calibration?

As I need to port a decision tree model from Python to Java, I would like to know whether PMML (Predictive Model Markup Language) supports probability calibration.
Javide's user avatar
  • 177
1 vote
1 answer
917 views

Calibrate the predicted class probability to make it represent a true probability?

Let's say that we have a simple binary classification model (a neural network -- NN) for classifying input images as "dog" ($y=1$) or "not dog" ($y=0$). Let's assume that the NN has one "sigmoid ...
Andreas K.'s user avatar
1 vote
2 answers
1k views

Do I need to correct predict_proba by training fraction?

Many algorithms provide a predict_proba function indicating probability of a case to belong to that class (e.g. https://scikit-learn.org/stable/modules/generated/...
rnso's user avatar
  • 1,558
6 votes
2 answers
4k views

Probability Calibration : role of hidden layer in Neural Network

I try a simple Neural Network (Logistic Regression) to play with Keras. In input I have 5,000 features (output of a simple tf-idf vectorizer) and in the output layer I just use a random uniform ...
BimBimBap's user avatar
12 votes
1 answer
8k views

Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event ...
Gale's user avatar
  • 403
18 votes
4 answers
2k views

XGBoost outputs tend towards the extremes

I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature ...
alwayslearning's user avatar
3 votes
1 answer
208 views

which loss function (if any) optimizes the calibration graph

The calibration graph is the predicted versus actual probability(see http://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html). Is it possible to optimize the ...
Hanan Shteingart's user avatar