How do I create a Keras custom loss function for a one-hot-encoded binary classifier?

Question

I have a CNN binary classifier with one-hot-encoded labels that I've written using Keras and it's just not training to the metric I want to encourage. My data is very imbalanced (91% class 0, 9% class 1) and, no matter what I do, it always favors accuracy in the majority class. I've played with class weights. I've tried creating balanced test and train files. It just seems that it's harder for my CNN to find patterns in the minority class, so it always returns improvements in the majority class. What I'd like to do is create a custom loss that allows me to define a score that rewards the minority class's true positives (TP) and penalizes its false positives (FP). Something like the following:

def minority_score(y_true, y_pred):

    max_value_true = K.argmax(y_true, -1)
    max_value_pred = K.argmax(y_pred, -1)

    FP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 0)
    TP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 1)
    score = (TP *3) - FP # punish each FP with a -1 and reward each TP with a +3
    return score # invert if using as loss function

model = build_model()

model.compile(loss=minority_score,
    optimizer=keras.optimizers.Adam(lr=0.0001),
    metrics=[minority_score, metrics.categorical_accuracy])

I've tried various things but, as my labels are one-hot-encoded, I always run into problems decoding them. y_pred and y_true are tensors in the above example. In that example, I get this error:

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'conv2d_1_input' with dtype float and shape [?,28,28,1]

Here's the relevant code all put together. Thank you.

def minority_score(y_true, y_pred):

    max_value_true = K.argmax(y_true, -1)
    max_value_pred = K.argmax(y_pred, -1)

    # below line errors out
    FP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 0) 
    TP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 1)
    score = (TP *3) - FP # punish each FP with a -1 and reward each TP with a +3
    return score 

class Modeler:
    def build_model(self):
        """
        this method only builds the scaffolding
        weights should be set and/or loaded
        outside of it

        """

        model = Sequential()

        # layerset 1
        model.add(Conv2D(
            filters=32,
            kernel_size=[3, 3],
            strides=(1, 1),
            input_shape=(28, 28, 1),
            padding='same'
        ))
        model.add(BatchNormalization())
        model.add(Activation("relu"))
        model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
        model.add(Dropout(0.2))
        # layerset 2
        model.add(Conv2D(
            filters=64,
            kernel_size=[3, 3],
            padding='same'
        ))
        model.add(BatchNormalization())
        model.add(Activation("relu"))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.2))
        # layerset 3
        model.add(Conv2D(
            filters=128,
            kernel_size=[3, 3],
            padding='same'
        ))
        model.add(BatchNormalization())
        model.add(Activation("relu"))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))

        model.add(Flatten())
        model.add(Dense(2048, activation='relu'))
        model.add(Dense(self.num_classes, activation='softmax'))

        return model

    def compile_model(self, model, learning_rate):
        model.compile(loss=minority_score,
                      optimizer=keras.optimizers.Adam(lr=learning_rate),
                      metrics=[minority_score, metrics.categorical_accuracy)
        return model

    def one_hot_encode_labels(self, data):

        return keras.utils.to_categorical(data, self.num_classes)

modeler = modeler.build_model()
model = modeler.compile_model()

# load model weights from checkpoint here
# load and shape data here
# create class weight dictionary here

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=num_epochs,
          verbose=1,
          validation_data=(x_test, y_test),
          callbacks=callbacks_list,
          class_weight=weight_dict)


```

Answer 1

If your problem is unbalanced classification, I don't think the problem can be solved through a custom loss function.

Building custom, balanced mini-batches is usually the thing to do, if it doesn't work it could be that your dataset is so much inbalanced that even this trick doesn't work. Can I ask you how many observations do you have for the "rare" class?

If they are too little, image augmentation could be the way to go: applying random distortions to original images before feeding them into the Network at each training iteration is a way to artificially increase the size of your dataset (while fighting overfitting at the same time).

An alternative could be to crate an Autoencoder, and treat the problem as an anomaly detection task. Anomaly detection has to deal with anomalies, that, by definition, are very rare events. You could exploit the fact that your model learns only one class properly, and treat the occurrence of the other class as an anomaly. Its appearance should be detected from an unusually compression Loss by the Autoencoder.

Answer 2

Assuming CLASS_WEIGHTS contains the weights you want to apply per class, you can use the following function to weight the outcome of a predefined loss.

from tensorflow.keras import backend as K

def class_weighted_loss(y_true, y_pred, **kwargs):
  weights = tf.constant(np.array([CLASS_WEIGHTS]), dtype=tf.float32)
  y_class = K.argmax(y_true, axis=1)
  w = tf.gather(weights, y_class)
  loss = keras.losses.categorical_crossentropy(y_true, y_pred)
  result = loss * w / K.sum(w)
  return result

Notebook with a complete example: https://colab.research.google.com/drive/1FB-dnyNeicdUYAvGvll1XykhcFadjUfd

In my test case, it isn't a clear win to use a weighted loss. I think that the results I've are rather similar in both cases.