convert predict_proba results using class_weight in training

Question

As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I penalize the misclassification of a rare positive cases.

rf = RandomForestClassifier(max_depth=m, n_estimators=n_estimator,class_weight = "balanced")
rf.fit(X_train, y_train)

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

In my case the classes frequencies are:

fc = len(y_train)/(len(np.unique(y_train))*np.bincount(y_train))

10000/(2*np.array([9500,500])) array([ 0.52631579, 10. ])

I apply my model over a test dataset using predict proba function:

y_predicted_proba = rf.predict_proba(X_test)

The second column presents the probabilities of being 1 to the input samples. However I understand this probability must be corrected to be real.

If I divide them by class_weight values these new probabilities don't sum one...

How can this correction be achieved?

Answer 1

The correction that you are talking about is called probability calibration -- you want the "real" probability of an observation being in each class, right?

The two most common approaches to probability calibration are Platt Scaling and Isotonic regression. Since you are training on a balanced training set (which is the right thing to do in your case because the original dataset is unbalanced), you can apply these techniques afterwards on your test set.

Sorry I can't explain the techniques fully here, but hopefully knowing the names of these terms gives you a starting point.

Answer 2

First, you should consider not balancing the dataset. It may well be unnecessary for this problem.

---

Now to your actual question. In sklearn, each decision tree reports the probability and these are averaged across the trees (as opposed to trees reporting their decisions and voting). So we can just understand how weighting affects these probabilities. In each leaf during training, the score given is $$\#\text{ positives in leaf}\ /\ \#\text{ total in leaf.}$$ With the shorthand $n_1$ and $n_0$ as the number of positives and negatives in the leaf, we can rewrite as $$p=\frac{n_1}{n_0+n_1} = \frac{1}{1+\frac{n_0}{n_1}}.$$ And weighting has the effect of transforming this quantity to instead $$p'=\frac{1}{1+\frac{0.526n_0}{10n_1}}.$$

(See e.g. [1], [2].) Just a bit of algebra then gives $$p = \frac{1}{1+\frac{10}{0.526}(\frac{1}{p'}-1)}.$$

It's worth noting that this implies $$\textrm{Adjusted odds} = \frac{p}{1-p} = \frac{0.526}{10}\left(\frac{p'}{1-p'}\right) = \frac{0.526}{10}\cdot\textrm{Weighted model's odds}.$$ This lines up with a resampling adjustment that is well-known for logistic regression models, but less so for other models. As for class weights in general: Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?

---

Finally, random forests are not generally well-calibrated, i.e. the probability scores you get out won't necessarily align well with the true proportions. (In particular, predictions tend to shy away from 0 and 1.) For this, you can apply calibration methods as @Maia mentions (which obviates the above adjustment).