5
$\begingroup$

I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files

Sensitivity & specificity is what I want to optimise for. Every epoch I output it:

class SensitivitySpecificityCallback(Callback):
    def on_epoch_end(self, epoch, logs=None):
        if epoch:
            x_test = self.validation_data[0]
            y_test = self.validation_data[1]
            predictions = self.model.predict(x_test)
            output_sensitivity_specificity(epoch, predictions, y_test)


def output_sensitivity_specificity(epoch, predictions, y_test):
    y_test = np.argmax(y_test, axis=-1)
    predictions = np.argmax(predictions, axis=-1)
    c = confusion_matrix(y_test, predictions)
    print('Confusion matrix:\n', c)
    print('[{:03d}] sensitivity'.format(epoch), c[0, 0] / (c[0, 1] + c[0, 0]))
    print('[{:03d}] specificity'.format(epoch), c[1, 1] / (c[1, 1] + c[1, 0]))

On my 203rd epoch I get a great result, then it's all downhill—in the wrong direction!—from there.

How do I optimise for sensitivity and specificity? - Thinking to update the weights, and/or develop a custom loss function…

$\endgroup$
3
  • $\begingroup$ Sensitivity and specificity are not differentiable so you can't optimize for them. You can use a custom Callback that saves your metrics along with a ModelCheckpoint that saves every model. When training is over, you can lookup the best metric from your Callback and then load that model file. $\endgroup$ – Bert Kellerman Jun 13 '18 at 18:38
  • $\begingroup$ @BertKellerman But that will only help identify & store the best epoch. It won't improve the performance of the model. Say I want both the sensitivity and specificity to be above a threshold. Then it can try to improve by a rate of threshold + delta every epoch. Could something like this be defined as a loss function? $\endgroup$ – A T Jun 14 '18 at 10:36
  • $\begingroup$ AFAIK, there isn't a direct way to do that. I am using save_best_only=False with ModelCheckpoint to save the model after every epoch. Afterwards, I check the metric callback for the best val metric epoch and load that model. $\endgroup$ – Bert Kellerman Jun 14 '18 at 13:10
0
$\begingroup$

The comments raise two important points:

  • you can't directly optimize for sensitivity or specificity
  • you should save the model at the best epoch and use that one

Below I explain something that is not directly related, but might be interesting for you:

You can add the weights to the loss of some examples. For example, you can increase the weight of positive examples which means that the model will focus more on correctly classifying positive examples at the potential expense of negative examples. This is especially useful if your data is suffering from class imbalance.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.