# convert predict_proba results using class_weight in training

As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I penalize the misclassification of a rare positive cases.

rf = RandomForestClassifier(max_depth=m, n_estimators=n_estimator,class_weight = "balanced")
rf.fit(X_train, y_train)


The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

In my case the classes frequencies are:

fc = len(y_train)/(len(np.unique(y_train))*np.bincount(y_train))


10000/(2*np.array([9500,500])) array([ 0.52631579, 10. ])

I apply my model over a test dataset using predict proba function:

y_predicted_proba = rf.predict_proba(X_test)


The second column presents the probabilities of being 1 to the input samples. However I understand this probability must be corrected to be real.

If I divide them by class_weight values these new probabilities don't sum one...

How can this correction be achieved?

The correction that you are talking about is called probability calibration -- you want the "real" probability of an observation being in each class, right?

The two most common approaches to probability calibration are Platt Scaling and Isotonic regression. Since you are training on a balanced training set (which is the right thing to do in your case because the original dataset is unbalanced), you can apply these techniques afterwards on your test set.

Sorry I can't explain the techniques fully here, but hopefully knowing the names of these terms gives you a starting point.

First, you should consider not balancing the dataset. It may well be unnecessary for this problem.

Now to your actual question. In sklearn, each decision tree reports the probability and these are averaged across the trees (as opposed to trees reporting their decisions and voting). So we can just understand how weighting affects these probabilities. In each leaf during training, the score given is $$\#\text{ positives in leaf}\ /\ \#\text{ total in leaf.}$$ With the shorthand $$n_1$$ and $$n_0$$ as the number of positives and negatives in the leaf, we can rewrite as $$p=\frac{n_1}{n_0+n_1} = \frac{1}{1+\frac{n_0}{n_1}.}$$ And weighting has the effect of transforming this quantity to instead $$p'=\frac{1}{1+\frac{0.526n_0}{10n_1}.}$$ Just a bit of algebra then gives $$p = \frac{1}{1+\frac{10}{0.526}(\frac{1}{p'}-1)}.$$

Finally, random forests are not generally well-calibrated, i.e. the probability scores you get out won't necessarily align well with the true proportions. For this, you can apply calibration methods as @Maia mentions (which probably obviates the above adjustment, but I thought it was worth working out).