1
$\begingroup$

I learned that Keras doesn't have a built-in way to set a threshold for precision and accuracy when building a classifier.

Courtesy of a solution here, I wanted to see what would happen when I fit a simple 3 layer Multilayer Perceptron with different classification thresholds for these metrics.

So I went ahead and fit a model with as such:

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics = [precision_threshold(0.5), recall_threshold(0.5)])

and got out these results:

val_precision: 0.7048 - val_recall: 0.5795

Further, I plotted a normalized confusion matrix. And using this:

def precision(cm = confusion_matrix):
    return round(cm[1,1] / (cm[0,1] + cm[1,1]), 4)

def recall(cm = confusion_matrix):
    return round(cm[1,1] / (cm[1,0] + cm[1,1]), 4)

I calculated the resulting precision and recall:

Normalized confusion matrix
[[ 0.6026  0.3974]
 [ 0.2204  0.7796]]

Precision: 0.71
Recall:    0.78

This seems pretty reasonable so far.

Then I switched up the thresholds and looked at what outcomes resulted. The table below shows what I got when I changed the precision threshold to 0.1 and the recall threshold to 0.9:

|-----------------------------------------------------|
| paramter settings  for model fitting:               |
| [precision_threshold(0.1)    recall_threshold(0.9)] |
|                                                     |
| validation results                                  |
|- val_precision  0.4089     - val_recall 0.0457      |
|                                                     |
| Normalized confusion matrix                         |
| [[ 0.6148  0.3852]                                  |
| [ 0.2303  0.7697]]                                  |
|                                                     |
| Precision   0.70                                    |
| Recall      0.79                                    |
|-----------------------------------------------------|

Interestingly, if you look at the final Precision and Recall scores, I notice that they didn't change, even with the extreme parameters I set. Basically the same confusion matrix as well.

I repeated this for a large set of different settings and got generally the same final Precision and Recall scores. I could past them in here but it would take up a lot of space.

One more detail then I'll switch to my actual question.

My model is based on the business case where a large precision value is very desirable and recall is not relevant.

My question is this -- given that I desire large precision -- is tuning the specific precision and recall thresholds for the model valuable at all, if the final Precision and Recall scores are left fundamentally unchanged across all configurations of the thresholds?

I hope this makes sense. Thanks for reading this.

$\endgroup$
  • $\begingroup$ Your validation recall went from 0.58 to 0.05. Are you including the training data when calculating the confusion matrix, precision and recall? This could be why it is not changing much. $\endgroup$ – geometrikal Aug 28 '17 at 7:46
  • $\begingroup$ No, using the test data. But that's part of the question -- val_precision and val_recall changed, but not the Precision and Recall. So is this actually valuable? I'm not sure how to reconcile these scores. $\endgroup$ – Monica Heddneck Aug 28 '17 at 7:50
  • $\begingroup$ I think your precision and recall formulas are wrong. Have a look at sklearn.metrics.precision_recall_fscore_support, it will calculate it properly. $\endgroup$ – geometrikal Aug 28 '17 at 7:58
  • $\begingroup$ Hummm...I'm using the same formulas on the sklearn page...but I'll check it out $\endgroup$ – Monica Heddneck Aug 28 '17 at 7:59
  • $\begingroup$ Yea I think you have to average each class. Still, doesn't make sense to have val_recall so low given that confusion matrix. Btw have a look here: scikit-learn.org/stable/auto_examples/model_selection/… it has an example to plot a precision / recall curve that may be useful to visualise the optimum values $\endgroup$ – geometrikal Aug 28 '17 at 8:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.