# How do I create a Keras custom loss function for a one-hot-encoded binary classifier?

I have a CNN binary classifier with one-hot-encoded labels that I've written using Keras and it's just not training to the metric I want to encourage. My data is very imbalanced (91% class 0, 9% class 1) and, no matter what I do, it always favors accuracy in the majority class. I've played with class weights. I've tried creating balanced test and train files. It just seems that it's harder for my CNN to find patterns in the minority class, so it always returns improvements in the majority class. What I'd like to do is create a custom loss that allows me to define a score that rewards the minority class's true positives (TP) and penalizes its false positives (FP). Something like the following:


def minority_score(y_true, y_pred):

max_value_true = K.argmax(y_true, -1)
max_value_pred = K.argmax(y_pred, -1)

FP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 0)
TP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 1)
score = (TP *3) - FP # punish each FP with a -1 and reward each TP with a +3
return score # invert if using as loss function

model = build_model()

model.compile(loss=minority_score,
metrics=[minority_score, metrics.categorical_accuracy])


I've tried various things but, as my labels are one-hot-encoded, I always run into problems decoding them. y_pred and y_true are tensors in the above example. In that example, I get this error:

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'conv2d_1_input' with dtype float and shape [?,28,28,1]


Here's the relevant code all put together. Thank you.


def minority_score(y_true, y_pred):

max_value_true = K.argmax(y_true, -1)
max_value_pred = K.argmax(y_pred, -1)

# below line errors out
FP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 0)
TP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 1)
score = (TP *3) - FP # punish each FP with a -1 and reward each TP with a +3
return score

class Modeler:
def build_model(self):
"""
this method only builds the scaffolding
weights should be set and/or loaded
outside of it

"""

model = Sequential()

# layerset 1
filters=32,
kernel_size=[3, 3],
strides=(1, 1),
input_shape=(28, 28, 1),
))
# layerset 2
filters=64,
kernel_size=[3, 3],
))
# layerset 3
filters=128,
kernel_size=[3, 3],
))

return model

def compile_model(self, model, learning_rate):
model.compile(loss=minority_score,
metrics=[minority_score, metrics.categorical_accuracy)
return model

def one_hot_encode_labels(self, data):

return keras.utils.to_categorical(data, self.num_classes)

modeler = modeler.build_model()
model = modeler.compile_model()

# load model weights from checkpoint here
# load and shape data here
# create class weight dictionary here

model.fit(x_train, y_train,
batch_size=batch_size,
epochs=num_epochs,
verbose=1,
validation_data=(x_test, y_test),
callbacks=callbacks_list,
class_weight=weight_dict)

$$$$

• Keras loss and metrics functions operate based on tensors, not on bumpy arrays. Usually one can find a Keras backend function or a tf function that does implement the similar functionality. When that is not at all possible, one can use tf.py_function to allow one to use numpy operations. Please keep in mind that tensor operations include automatic auto-differentiation support. python numpy operations do not. So if the function affects gradient computations you would need to tell tensorflow how to compute the gradients. – Pedro Marques Jul 7 '19 at 19:27
• @PedroMarques . Thank you for the response. I think the problem with my attempt is with the K.argmax more than the numpy.logical_and. Though, to be honest, I'm not sure I understand gradient differentiation and back propagation well enough to really understand my problem. Regardless, it seems that a full weekend of searching hasn't shown me an example of anyone using class-specific metrics in a loss function so I wonder if it's not possible. – Dan McConkey Jul 7 '19 at 22:58

If your problem is unbalanced classification, I don't think the problem can be solved through a custom loss function.

Building custom, balanced mini-batches is usually the thing to do, if it doesn't work it could be that your dataset is so much inbalanced that even this trick doesn't work. Can I ask you how many observations do you have for the "rare" class?

If they are too little, image augmentation could be the way to go: applying random distortions to original images before feeding them into the Network at each training iteration is a way to artificially increase the size of your dataset (while fighting overfitting at the same time).

An alternative could be to crate an Autoencoder, and treat the problem as an anomaly detection task. Anomaly detection has to deal with anomalies, that, by definition, are very rare events. You could exploit the fact that your model learns only one class properly, and treat the occurrence of the other class as an anomaly. Its appearance should be detected from an unusually compression Loss by the Autoencoder.

Assuming CLASS_WEIGHTS contains the weights you want to apply per class, you can use the following function to weight the outcome of a predefined loss.

from tensorflow.keras import backend as K

def class_weighted_loss(y_true, y_pred, **kwargs):
weights = tf.constant(np.array([CLASS_WEIGHTS]), dtype=tf.float32)
y_class = K.argmax(y_true, axis=1)
w = tf.gather(weights, y_class)
loss = keras.losses.categorical_crossentropy(y_true, y_pred)
result = loss * w / K.sum(w)
return result
`