9
$\begingroup$

I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities from the models do not match the actual probabilities in the modeling data.

Is there a way to adjust the predicted probabilities from the class weighted models to match the actual probabilities in the data? I have seen equations for under-sampling (https://www3.nd.edu/~rjohns15/content/papers/ssci2015_calibrating.pdf) but they don't seem to work for class weights. I have searched online for an answer but maybe I am not using the right language?

Thank you!

$\endgroup$
3
  • 2
    $\begingroup$ This reference is exactly what you're looking for: gking.harvard.edu/files/0s.pdf Logistic Regression in Rare Events Data (Gary King) $\endgroup$ Commented Jul 12, 2020 at 18:13
  • 1
    $\begingroup$ Link to ssci2015_calibrating.pdf is dead today. Can you find online? I cannot because you did not give an author's name or a title of the article. $\endgroup$
    – pauljohn32
    Commented Mar 9, 2021 at 23:57
  • $\begingroup$ @pauljohn32, I believe it was "Calibrating Probability with Undersampling for Unbalanced Classification" by Dal Pozzolo et al. $\endgroup$
    – Ben Reiniger
    Commented Mar 10, 2021 at 1:51

3 Answers 3

10
+50
$\begingroup$

A more general adjustment for resampling (not just the simple undersampling in your linked paper) exists:

Add $\ln\left(\frac{p_1(1-r_1)}{(1-p_1)r_1}\right)$ to the log-odds of each prediction, where $p_1$ is the proportion of the positive class in the original dataset, and $r_1$ is the proportion of the positive class in the resampled dataset.

Equivalently, multiply the odds by the quantity inside the logarithm. (Unfortunately, this doesn't lead to a clean adjustment directly to the probabilities.)


Let's do a little rewriting to see the connection to your linked paper. $1-r_1$ is the proportion of negative classes call it $r_0$, and similarly with $p_1$. Use capitals $R_1, \dotsc$ to denote the number (or total weight) of samples rather than the proportions, and without subscripts $P,R$ to denote total numbers (or weights) of samples before and after resampling. So the multiplier becomes $$\frac{p_1(1-r_1)}{(1-p_1)r_1} = \frac{p_1 r_0}{p_0 r_1} = \frac{(P_1/P) (R_0/R)}{(P_0/P) (R_1/R)} = \frac{P_1 R_0}{P_0 R_1}.$$ In the context of the linked paper, positive class samples are not resampled, so $P_1=R_1$ and the adjustment simplifies to $R_0/P_0$, which is the parameter $\beta$ used in the paper.

Finally, using their equation (4), we check the change in odds: $$\text{new odds} = \frac{p}{1-p} = \frac{1}{\frac1p - 1} = \frac{1}{\frac{\beta p_s−p_s+ 1}{\beta p_s} - 1} = \frac{\beta p_s}{1-p_s} = \beta\cdot\text{old odds}. $$


So, what about weightings instead of resampling? Well, class_weights might have different effects in different algorithms, but generally the idea is that (positive) integer values of class_weights should correspond to duplicating the samples that many times, and fractional values interpolate that. So, it should be about the same to use the multiplicative factor above. Using the size version rather than the proportion version, we should interpret $R_0$ and $R_1$ as the total weights of the relevant classes.

I've not been able to find a reference for this version, so I put together a short experiment; it seems to verify that this shift works.
GitHub/Colab notebook


Finally, this shift in log-odds will fail to produce properly calibrated probabilities if the classifier is poorly calibrated on the weighted data. You could look into calibration techniques, from Platt to Beta to Isotonic. In this case, the shift above is probably superfluous.

$\endgroup$
2
  • $\begingroup$ Sorry I took so long to accept the question as answered. I had a close look at your code, which took some time, and it seems to work. The only thing is that you left r1 out of your adjustment, which assumes that r1 = 0.5. This works for the class weight scheme you used but if someone uses a different weighting scheme, r1 should be included. Thanks again! $\endgroup$ Commented Sep 18, 2019 at 17:27
  • 1
    $\begingroup$ re: "weightings are asymptotically equivalent to resamplings", I've just found the 2021 paper "Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients" by An, Ying, and Zhu. I haven't read into it much yet. web.stanford.edu/~lexing/resw.pdf $\endgroup$
    – Ben Reiniger
    Commented Feb 5 at 15:45
0
$\begingroup$

I had the same problem as you. However, when you apply the technique in the accepted answer, you cancel out the predictive power gained by class weights. I also stumbled on CalibratedClassifierCV method but it had a similar impact as the accepted answer so I did not use it at the end. As an alternative, I tried to collect more data (which I know not always possible) and try other models such as Decision Tree based models.

$\endgroup$
3
  • $\begingroup$ What exactly do you mean by predictive power? If you're relying on hard class predictions (so accuracy, recall, f1, etc.) then probably the problem is just the threshold; you can recover the same hard predictions by shifting the threshold by the same amount. $\endgroup$
    – Ben Reiniger
    Commented Nov 22, 2023 at 19:21
  • $\begingroup$ But what else can you do without relying on hard class predictions to assess your model? Without any class weights, the model will have a hard time to correctly detect and label positive class observations. If you lower the threshold, again this difficulty will persist but you will get more true positives but also more false positives. With class weights you tell the model that misclassifying an observation from the positive class becomes more costly. This improves positive class detection. I think OP has to decide between predictive performance or using predicted probs. $\endgroup$ Commented Nov 23, 2023 at 10:30
  • $\begingroup$ The probability shift in my answer doesn't change the rank-ordering of instances, so by moving the threshold by the same shift you recover exactly the same hard classifier, but with closer to true probabilities. $\endgroup$
    – Ben Reiniger
    Commented Feb 5 at 16:07
-1
$\begingroup$

The difference between predicted probabilities and actual probabilities is called training error.

There are many ways to reduce training error. Engineering better features and choosing a different machine learning algorithm are the most common.

$\endgroup$
1
  • 2
    $\begingroup$ Sorry, that is not quite what I am after. Without using class weights, the model will predict the actual probabilities. But when using class weights, the model will not predict the actual probabilities. I am hoping to use class weights, and adjust/correct the predicted probabilities to match the actual probabilities. $\endgroup$ Commented Sep 9, 2019 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.