Calibration using predict_proba vs class_weight

I am making a Random Forest Classifier to determine whether a sentence is "positive" (1), "negative"(-1) or "neutral"(0).

However, I prefer having false negative than false positive, that is, I prefer saying that a sentence is neutral even if it's not than to say that a sentence is positive when it's neutral, a fortiori if it's negative.

So I use predict_proba, with something like:

def my_pred(rfc, X, weight=0.5):
res = rfc.predict_proba(X)
if res[0]>weight: return -1
elif res[2]>weight: return 1
return 0


But I wonder if I can make such things (give more importance to the neutral class) using class_weight? Would it be better?

There are several options for you:

sklearn.svm.SVC, sklearn.ensemble.RandomForestClassifier, and others. Note there's no theoretical limit to the weight ratio, so even if 1 to 100 isn't strong enough for you, you can go on with 1 to 500, etc.

• You can also select the decision threshold very low during the cross-validation to pick the model that gives highest recall (though possibly low precision). The recall close to 1.0 effectively means false_negatives close to 0.0, which is what to want. For that, use sklearn.model_selection.cross_val_predict and sklearn.metrics.precision_recall_curve functions:

  y_scores = cross_val_predict(classifier, x_train, y_train, cv=3,

method="decision_function")

precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)


If you plot the precisions and recalls against the thresholds, you should see the picture like this:

After picking the best threshold, you can use the raw scores from classifier.decision_function() method for your final classification.

Finally, try not to over-optimize your classifier, because you can easily end up with a trivial const classifier (which is obviously never wrong, but is useless).

As said there are 2 stages to make this Tuning: in the model training stage (like custom weights) and the prediction stage (like lowering the decision threshold).

Another tuning for the model-training stage is using a recall scorer. you can use it in your grid-search cross-validation (GridSearchCV) for tuning your classifier with the best hyper-param towards high recall.

GridSearchCV scoring parameter can either accepts the 'recall' string or the function recall_score.

Since you're using a binary classification, both options should work out of the box, and call recall_score with its default values that suits a binary classification:

• average: 'binary' (i.e. one simple recall value)

• pos_label: 1 (like numpy's True value)

Should you need to custom it, you can wrap an existing scorer, or a custom one, with make_scorer, and pass it to the scoring parameter.