Does class weights guarantee calibration?

Question

There is possibility to assign class weights while training classifiers, e.g. CatBoost. To the best of my knowledge it adds weight to objects in computation of loss function, therefore penalizing errors on them more or less. Now, if I downsample negative samples with subsample ratio w, train classifier without weights and then recalibrate predictions with formulae from here

p = p' / ( p' +  (1-p') / w)

where p is probability of positive prediction on original dataset and p' is from subsampled dataset, will I obtain the same result with just training classifier on sampled dataset with weight of positive class set to w?

Answer 1

TLDR: No for large w

In theory, using class weights or downsampling with recalibration could yield the same results if the machine learning model learns all classes equally well. In practice, you’ll likely achieve similar outcomes if the subsampling ratio w isn’t too large (i.e., when the difference in the number of samples per class isn’t too extreme). However, if w is too large, the model may struggle to effectively learn the minority class. If the loss from a particular sample/group of samples is minimal, the model might ignore it.

Class weights, upsampling, or downsampling are generally used to ensure balanced class representation, helping the model consider all classes equally important and allowing it to rely on features rather than frequency statistics for discrimination. However, these methods are not explicitly used for calibration. Calibration is usually performed later using techniques specifically designed for that purpose.