5
$\begingroup$

I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files

Sensitivity & specificity is what I want to optimise for. Every epoch I output it:

class SensitivitySpecificityCallback(Callback):
    def on_epoch_end(self, epoch, logs=None):
        if epoch:
            x_test = self.validation_data[0]
            y_test = self.validation_data[1]
            predictions = self.model.predict(x_test)
            output_sensitivity_specificity(epoch, predictions, y_test)


def output_sensitivity_specificity(epoch, predictions, y_test):
    y_test = np.argmax(y_test, axis=-1)
    predictions = np.argmax(predictions, axis=-1)
    c = confusion_matrix(y_test, predictions)
    print('Confusion matrix:\n', c)
    print('[{:03d}] sensitivity'.format(epoch), c[0, 0] / (c[0, 1] + c[0, 0]))
    print('[{:03d}] specificity'.format(epoch), c[1, 1] / (c[1, 1] + c[1, 0]))

On my 203rd epoch I get a great result, then it's all downhill—in the wrong direction!—from there.

How do I optimise for sensitivity and specificity? - Thinking to update the weights, and/or develop a custom loss function…

$\endgroup$
  • $\begingroup$ Sensitivity and specificity are not differentiable so you can't optimize for them. You can use a custom Callback that saves your metrics along with a ModelCheckpoint that saves every model. When training is over, you can lookup the best metric from your Callback and then load that model file. $\endgroup$ – Bert Kellerman Jun 13 '18 at 18:38
  • $\begingroup$ @BertKellerman But that will only help identify & store the best epoch. It won't improve the performance of the model. Say I want both the sensitivity and specificity to be above a threshold. Then it can try to improve by a rate of threshold + delta every epoch. Could something like this be defined as a loss function? $\endgroup$ – A T Jun 14 '18 at 10:36
  • $\begingroup$ AFAIK, there isn't a direct way to do that. I am using save_best_only=False with ModelCheckpoint to save the model after every epoch. Afterwards, I check the metric callback for the best val metric epoch and load that model. $\endgroup$ – Bert Kellerman Jun 14 '18 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.