3
$\begingroup$

I am making a Random Forest Classifier to determine whether a sentence is "positive" (1), "negative"(-1) or "neutral"(0).

However, I prefer having false negative than false positive, that is, I prefer saying that a sentence is neutral even if it's not than to say that a sentence is positive when it's neutral, a fortiori if it's negative.

So I use predict_proba, with something like:

def my_pred(rfc, X, weight=0.5):
  res = rfc.predict_proba(X) 
  if res[0]>weight: return -1
  elif res[2]>weight: return 1
  return 0

But I wonder if I can make such things (give more importance to the neutral class) using class_weight? Would it be better?

$\endgroup$
4
$\begingroup$

There are several options for you:

sklearn.svm.SVC, sklearn.ensemble.RandomForestClassifier, and others. Note there's no theoretical limit to the weight ratio, so even if 1 to 100 isn't strong enough for you, you can go on with 1 to 500, etc.

  • You can also select the decision threshold very low during the cross-validation to pick the model that gives highest recall (though possibly low precision). The recall close to 1.0 effectively means false_negatives close to 0.0, which is what to want. For that, use sklearn.model_selection.cross_val_predict and sklearn.metrics.precision_recall_curve functions:

      y_scores = cross_val_predict(classifier, x_train, y_train, cv=3,
    
                                   method="decision_function")
    
    
    
      precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)
    

    If you plot the precisions and recalls against the thresholds, you should see the picture like this:

    precision-recall-tradeoff

    After picking the best threshold, you can use the raw scores from classifier.decision_function() method for your final classification.

Finally, try not to over-optimize your classifier, because you can easily end up with a trivial const classifier (which is obviously never wrong, but is useless).

As said there are 2 stages to make this Tuning: in the model training stage (like custom weights) and the prediction stage (like lowering the decision threshold).

Another tuning for the model-training stage is using a recall scorer. you can use it in your grid-search cross-validation (GridSearchCV) for tuning your classifier with the best hyper-param towards high recall.

GridSearchCV scoring parameter can either accepts the 'recall' string or the function recall_score.

Since you're using a binary classification, both options should work out of the box, and call recall_score with its default values that suits a binary classification:

  • average: 'binary' (i.e. one simple recall value)

  • pos_label: 1 (like numpy's True value)

Should you need to custom it, you can wrap an existing scorer, or a custom one, with make_scorer, and pass it to the scoring parameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.