# Loss function for optimising precision & recall / sensitivity & specificity?

I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files

Sensitivity & specificity is what I want to optimise for. Every epoch I output it:

class SensitivitySpecificityCallback(Callback):
def on_epoch_end(self, epoch, logs=None):
if epoch:
x_test = self.validation_data[0]
y_test = self.validation_data[1]
predictions = self.model.predict(x_test)
output_sensitivity_specificity(epoch, predictions, y_test)

def output_sensitivity_specificity(epoch, predictions, y_test):
y_test = np.argmax(y_test, axis=-1)
predictions = np.argmax(predictions, axis=-1)
c = confusion_matrix(y_test, predictions)
print('Confusion matrix:\n', c)
print('[{:03d}] sensitivity'.format(epoch), c[0, 0] / (c[0, 1] + c[0, 0]))
print('[{:03d}] specificity'.format(epoch), c[1, 1] / (c[1, 1] + c[1, 0]))


On my 203rd epoch I get a great result, then it's all downhill—in the wrong direction!—from there.

How do I optimise for sensitivity and specificity? - Thinking to update the weights, and/or develop a custom loss function…

• Sensitivity and specificity are not differentiable so you can't optimize for them. You can use a custom Callback that saves your metrics along with a ModelCheckpoint that saves every model. When training is over, you can lookup the best metric from your Callback and then load that model file. – Bert Kellerman Jun 13 '18 at 18:38
• @BertKellerman But that will only help identify & store the best epoch. It won't improve the performance of the model. Say I want both the sensitivity and specificity to be above a threshold. Then it can try to improve by a rate of threshold + delta every epoch. Could something like this be defined as a loss function? – A T Jun 14 '18 at 10:36
• AFAIK, there isn't a direct way to do that. I am using save_best_only=False with ModelCheckpoint to save the model after every epoch. Afterwards, I check the metric callback for the best val metric epoch and load that model. – Bert Kellerman Jun 14 '18 at 13:10