5
$\begingroup$

When we care more that there should be no false negatives, as far as possible… ie. higher recall (video is suitable for kid or not), we should use (receiver operating characteristic) ROC (area under the curve) AUC and try to maximize it.

Scikit-Learn provides a function to compute this directly:

from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)

Similarly, when we care more about the false positives than the false negatives, example shop lifting case then what should we do?

Should we just try to maximize the recall ignoring the precision or is there a better metric?

I had read somewhere "Use the precision vs recall curve and get that max" but not very sure what that means... Can you please explain the same?

If there a direct function for this metric in sci-kit learn, like above for the first case, please do let me know.

$\endgroup$
1
$\begingroup$

As far as a fuction in scikit to implement a certain threshold for a higher recall, I don't think there is one.

But, depending on what model you're using, you can vary the threshold for probability outputs of the model to obtain a higher Recall.

For example, say you are using a Random Forest or Logistic Regression classifier model, then you can use the predict_proba function on your data set; this will return probabilities for each sample belonging to a particular class. If you then create a new array from the output of predict_proba using your threshold, then the model will have a Recall with your desired accuracy when it classifies new unseen data.

e.g pseudo code for binary classification

clf = sklearn.ensemble.RandomForestClassifier()
model = fit(X,y) # fit model to training datset
probs = model.predict_proba(X_new) # prediction on a new dataset X_new

threshold = 0.7 # threshold we set where the probability prediction must be above this to be classified as a '1'
classes = probs[:,1] # say it is the class in the second column you care about predictint
classes[classes>=threshold] = 1
classes[classes<threshold] = 0
$\endgroup$
0
$\begingroup$

The easiest way is to replace your labels. The other way is to set importance of the more important class to a higher value so the cost function moves toward direction to take much care for your desired label. You can set the class_weight. Take a look at here and here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.