4
$\begingroup$

I have a CNN binary classifier with one-hot-encoded labels that I've written using Keras and it's just not training to the metric I want to encourage. My data is very imbalanced (91% class 0, 9% class 1) and, no matter what I do, it always favors accuracy in the majority class. I've played with class weights. I've tried creating balanced test and train files. It just seems that it's harder for my CNN to find patterns in the minority class, so it always returns improvements in the majority class. What I'd like to do is create a custom loss that allows me to define a score that rewards the minority class's true positives (TP) and penalizes its false positives (FP). Something like the following:


def minority_score(y_true, y_pred):

    max_value_true = K.argmax(y_true, -1)
    max_value_pred = K.argmax(y_pred, -1)

    FP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 0)
    TP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 1)
    score = (TP *3) - FP # punish each FP with a -1 and reward each TP with a +3
    return score # invert if using as loss function

model = build_model()

model.compile(loss=minority_score,
    optimizer=keras.optimizers.Adam(lr=0.0001),
    metrics=[minority_score, metrics.categorical_accuracy])

I've tried various things but, as my labels are one-hot-encoded, I always run into problems decoding them. y_pred and y_true are tensors in the above example. In that example, I get this error:

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'conv2d_1_input' with dtype float and shape [?,28,28,1]

Here's the relevant code all put together. Thank you.


def minority_score(y_true, y_pred):

    max_value_true = K.argmax(y_true, -1)
    max_value_pred = K.argmax(y_pred, -1)

    # below line errors out
    FP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 0) 
    TP = np.logical_and(K.eval(max_value_pred) == 1, K.eval(max_value_true) == 1)
    score = (TP *3) - FP # punish each FP with a -1 and reward each TP with a +3
    return score 

class Modeler:
    def build_model(self):
        """
        this method only builds the scaffolding
        weights should be set and/or loaded
        outside of it

        """

        model = Sequential()

        # layerset 1
        model.add(Conv2D(
            filters=32,
            kernel_size=[3, 3],
            strides=(1, 1),
            input_shape=(28, 28, 1),
            padding='same'
        ))
        model.add(BatchNormalization())
        model.add(Activation("relu"))
        model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
        model.add(Dropout(0.2))
        # layerset 2
        model.add(Conv2D(
            filters=64,
            kernel_size=[3, 3],
            padding='same'
        ))
        model.add(BatchNormalization())
        model.add(Activation("relu"))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.2))
        # layerset 3
        model.add(Conv2D(
            filters=128,
            kernel_size=[3, 3],
            padding='same'
        ))
        model.add(BatchNormalization())
        model.add(Activation("relu"))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(Dropout(0.25))

        model.add(Flatten())
        model.add(Dense(2048, activation='relu'))
        model.add(Dense(self.num_classes, activation='softmax'))

        return model

    def compile_model(self, model, learning_rate):
        model.compile(loss=minority_score,
                      optimizer=keras.optimizers.Adam(lr=learning_rate),
                      metrics=[minority_score, metrics.categorical_accuracy)
        return model

    def one_hot_encode_labels(self, data):

        return keras.utils.to_categorical(data, self.num_classes)

modeler = modeler.build_model()
model = modeler.compile_model()

# load model weights from checkpoint here
# load and shape data here
# create class weight dictionary here

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=num_epochs,
          verbose=1,
          validation_data=(x_test, y_test),
          callbacks=callbacks_list,
          class_weight=weight_dict)


```
$\endgroup$
  • $\begingroup$ Keras loss and metrics functions operate based on tensors, not on bumpy arrays. Usually one can find a Keras backend function or a tf function that does implement the similar functionality. When that is not at all possible, one can use tf.py_function to allow one to use numpy operations. Please keep in mind that tensor operations include automatic auto-differentiation support. python numpy operations do not. So if the function affects gradient computations you would need to tell tensorflow how to compute the gradients. $\endgroup$ – Pedro Marques Jul 7 '19 at 19:27
  • $\begingroup$ @PedroMarques . Thank you for the response. I think the problem with my attempt is with the K.argmax more than the numpy.logical_and. Though, to be honest, I'm not sure I understand gradient differentiation and back propagation well enough to really understand my problem. Regardless, it seems that a full weekend of searching hasn't shown me an example of anyone using class-specific metrics in a loss function so I wonder if it's not possible. $\endgroup$ – Dan McConkey Jul 7 '19 at 22:58
1
$\begingroup$

If your problem is unbalanced classification, I don't think the problem can be solved through a custom loss function.

Building custom, balanced mini-batches is usually the thing to do, if it doesn't work it could be that your dataset is so much inbalanced that even this trick doesn't work. Can I ask you how many observations do you have for the "rare" class?

If they are too little, image augmentation could be the way to go: applying random distortions to original images before feeding them into the Network at each training iteration is a way to artificially increase the size of your dataset (while fighting overfitting at the same time).

An alternative could be to crate an Autoencoder, and treat the problem as an anomaly detection task. Anomaly detection has to deal with anomalies, that, by definition, are very rare events. You could exploit the fact that your model learns only one class properly, and treat the occurrence of the other class as an anomaly. Its appearance should be detected from an unusually compression Loss by the Autoencoder.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Assuming CLASS_WEIGHTS contains the weights you want to apply per class, you can use the following function to weight the outcome of a predefined loss.

from tensorflow.keras import backend as K

def class_weighted_loss(y_true, y_pred, **kwargs):
  weights = tf.constant(np.array([CLASS_WEIGHTS]), dtype=tf.float32)
  y_class = K.argmax(y_true, axis=1)
  w = tf.gather(weights, y_class)
  loss = keras.losses.categorical_crossentropy(y_true, y_pred)
  result = loss * w / K.sum(w)
  return result

Notebook with a complete example: https://colab.research.google.com/drive/1FB-dnyNeicdUYAvGvll1XykhcFadjUfd

In my test case, it isn't a clear win to use a weighted loss. I think that the results I've are rather similar in both cases.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the response. I'm not sure it answers my question, though. I've implemented class weights already but that doesn't seem to be strong enough. I think my real problem is that, no matter what, it's still evaluating on accuracy and I need it to evaluate on something closer to the F1-score $\endgroup$ – Dan McConkey Jul 9 '19 at 18:03
  • $\begingroup$ You formulated your question as "I'd like to do is create a custom loss that allows me to define a score that rewards the minority class true positives...". The example above computes weights for classes and allows you to identify each class and thus weight the true positives. You will have to extend it to detect the false positives; if you are convinced that that is the metric you need. $\endgroup$ – Pedro Marques Jul 9 '19 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.