I have some model for which I can construct the confusion matrix, although I need a custom loss function which will be as:

confusion matrix

true negatives (TN): We predicted no, and it is no.

false positives (FP): We predicted yes, but it is no.

My loss function should be like:

  1 -  _________
        TN + FP 

How do I implement the same in Keras: There is an explanation for the custom loss function here: StackOverFlow

Although, I am not sure how the differentiation part for the same would be like.

Image source: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/


The Above Solution does not work when you use it during training it throws an error because of this line:

TN = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 0)

Maybe this could be a better solution to this problem you can calculate specificity after every epoch and also plot!

def specificity(y_pred, y_true):
    y_pred - Predicted labels
    y_true - True labels 
    Specificity score
    neg_y_true = 1 - y_true
    neg_y_pred = 1 - y_pred
    fp = K.sum(neg_y_true * y_pred)
    tn = K.sum(neg_y_true * neg_y_pred)
    specificity = tn / (tn + fp + K.epsilon())
    return specificity

So it seems that you would like to compute the specificity of your model:

$${\displaystyle \mathrm {TNR} ={\frac {\mathrm {TN} }{N}}={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FP} }}}$$

So you would require a function that can take your predictions, compute the number of true negative and false positive, then compute the specifictiy using the equation above. The body of this function is borrowed from here and simply modified for two classes.

import numpy as np
import keras.backend as K

def compute_binary_specificity(y_pred, y_true):
    """Compute the confusion matrix for a set of predictions.

    y_pred   : predicted values for a batch if samples (must be binary: 0 or 1)
    y_true   : correct values for the set of samples used (must be binary: 0 or 1)

    out : the specificity

    check_binary(K.eval(y_true))    # must check that input values are 0 or 1
    check_binary(K.eval(y_pred))    # 

    TN = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 0)
    FP = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 1)

    # as Keras Tensors
    TN = K.sum(K.variable(TN))
    FP = K.sum(K.variable(FP))

    specificity = TN / (TN + FP + K.epsilon())
    return specificity

Edit: this function gives results equivalent to a numpy version of the function and is tested to work for 2d, 3d, 4d and 5d arrays.

As the link you added suggests, you must also create a wrapper function to use this custom function as a loss function in Keras:

def specificity_loss_wrapper():
    """A wrapper to create and return a function which computes the specificity loss, as (1 - specificity)

    # Define the function for your loss
    def specificity_loss(y_true, y_pred):
        return 1.0 - compute_binary_specificity(y_true, y_pred)

    return specificity_loss    # we return this function object

Note the the specificity loss is returned from the wrapper function as $1 - specificity$. This could have been performed in the first function too - it should matter, I just separated the computation of specificity from that off the loss.

This can then be used like this:

# Create a Keras model object as usual
model = my_model()

# ... (add layers etc)

# Create the loss function object using the wrapper function above
spec_loss = specificity_loss_wrapper()

# compile model using the return los function object

# ... train model as usual

Additionally, you could try importing Tensorflow itself and use its built-in tf.confusion_matrix operation.

  • $\begingroup$ 1'st function should be 'for i in range(len(y_true))'? And the operations in the functions should be coded using Keras.backend, so it would be auto differentiable ?(got 'object of type 'Tensor' has no len()' error on the mentioned line), I'll try to replicate this, Thanks! $\endgroup$ – Nikhil Verma Jun 25 '18 at 10:03
  • $\begingroup$ @NikhilVerma - I added how you could fix the error of iterating over the values and made conf_mat a Keras variable tensor. $\endgroup$ – n1k31t4 Jun 25 '18 at 10:47
  • $\begingroup$ Hey, getting 'TypeError: index returned non-int (type NoneType)' at for line $\endgroup$ – Nikhil Verma Jun 25 '18 at 11:00
  • $\begingroup$ pastebin.com/Q94Dyc96, link has the sample model $\endgroup$ – Nikhil Verma Jun 25 '18 at 11:00
  • $\begingroup$ For me it works, but I've added the .value attibute which should return the actual number (perhaps you can't iterate on the returned Dimension object in all python/keras versions). I tested on empty arrays as well, but still can't reproduce your error. I spotted a typo in my code, so have fixed it by simplifying the compute_specificity function. $\endgroup$ – n1k31t4 Jun 25 '18 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.