5
$\begingroup$

As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I penalize the misclassification of a rare positive cases.

rf = RandomForestClassifier(max_depth=m, n_estimators=n_estimator,class_weight = "balanced")
rf.fit(X_train, y_train)

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

In my case the classes frequencies are:

fc = len(y_train)/(len(np.unique(y_train))*np.bincount(y_train))

10000/(2*np.array([9500,500])) array([ 0.52631579, 10. ])

I apply my model over a test dataset using predict proba function:

y_predicted_proba = rf.predict_proba(X_test)

screencap of output

The second column presents the probabilities of being 1 to the input samples. However I understand this probability must be corrected to be real.

If I divide them by class_weight values these new probabilities don't sum one...

screencap of new output

How can this correction be achieved?

$\endgroup$
  • $\begingroup$ I've added some more details and references to my answer. Let me know if there's something specific that's still missing. $\endgroup$ – Ben Reiniger Feb 10 at 1:28
3
$\begingroup$

The correction that you are talking about is called probability calibration -- you want the "real" probability of an observation being in each class, right?

The two most common approaches to probability calibration are Platt Scaling and Isotonic regression. Since you are training on a balanced training set (which is the right thing to do in your case because the original dataset is unbalanced), you can apply these techniques afterwards on your test set.

Sorry I can't explain the techniques fully here, but hopefully knowing the names of these terms gives you a starting point.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The main drawback I see in these calibration methods (compared to the shift correction) is that they require an additional training dataset. The main benefit is that they aren't simple shifts, so they will correct for poor calibration of the model itself, beyond just the class weight (or resampling) bias. $\endgroup$ – Ben Reiniger Feb 8 at 18:59
1
$\begingroup$

First, you should consider not balancing the dataset. It may well be unnecessary for this problem.


Now to your actual question. In sklearn, each decision tree reports the probability and these are averaged across the trees (as opposed to trees reporting their decisions and voting). So we can just understand how weighting affects these probabilities. In each leaf during training, the score given is $$\#\text{ positives in leaf}\ /\ \#\text{ total in leaf.}$$ With the shorthand $n_1$ and $n_0$ as the number of positives and negatives in the leaf, we can rewrite as $$p=\frac{n_1}{n_0+n_1} = \frac{1}{1+\frac{n_0}{n_1}}.$$ And weighting has the effect of transforming this quantity to instead $$p'=\frac{1}{1+\frac{0.526n_0}{10n_1}}.$$

(See e.g. [1], [2].) Just a bit of algebra then gives $$p = \frac{1}{1+\frac{10}{0.526}(\frac{1}{p'}-1)}.$$

It's worth noting that this implies $$\textrm{Adjusted odds} = \frac{p}{1-p} = \frac{0.526}{10}\left(\frac{p'}{1-p'}\right) = \frac{0.526}{10}\cdot\textrm{Weighted model's odds}.$$ This lines up with a resampling adjustment that is well-known for logistic regression models, but less so for other models. As for class weights in general: Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?


Finally, random forests are not generally well-calibrated, i.e. the probability scores you get out won't necessarily align well with the true proportions. (In particular, predictions tend to shy away from 0 and 1.) For this, you can apply calibration methods as @Maia mentions (which obviates the above adjustment).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.