I am working on a DNN that is training on ecg data with a shape of [None,1,2500] and output shape of [None,12,19] where 19 is a vector of 1 second bins and each of the 12 rows is a distinct binary class, 0,1. Classes are NOT exclusive. I am currently using sigmoid activation and binary crossentropy. The goal is to use weighted binary crossentropy as many of the classes are imbalanced in favor of the negative class i.e. 0. Using the off the shelf Tensorflow binarycrossentropy function breaks training with error,

    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1051, in train_function  *
        return step_function(self, iterator)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1040, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/martin/.local/lib/python3.8/site-packages/six.py", line 719, in reraise
        raise value
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1030, in run_step  **
        outputs = model.train_step(data)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 890, in train_step
        loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 948, in compute_loss
        return self.compiled_loss(
    File "/usr/lib/python3/dist-packages/keras/engine/compile_utils.py", line 201, in __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "/usr/lib/python3/dist-packages/keras/losses.py", line 140, in __call__
        return losses_utils.compute_weighted_loss(
    File "/usr/lib/python3/dist-packages/keras/utils/losses_utils.py", line 326, in compute_weighted_loss
        losses, _, sample_weight = squeeze_or_expand_dimensions(  # pylint: disable=unbalanced-tuple-unpacking
    File "/usr/lib/python3/dist-packages/keras/utils/losses_utils.py", line 212, in squeeze_or_expand_dimensions
        sample_weight = tf.squeeze(sample_weight, [-1])
    ValueError: Can not squeeze dim[1], expected a dimension of 1, got 19 for '{{node binary_crossentropy/weighted_loss/Squeeze}} = Squeeze[T=DT_FLOAT, squeeze_dims=[-1]](cond/Identity_4)' with input shapes: [?,19].

The failure of applying weights to the output with Tensorflows off-the-shelf binarycrossentropy spurred the use of a custom loss function as seen below, however, performance of the model with this custom loss is weak.

class MultiLabelLoss(tf.keras.losses.Loss):
    def __init__(self, pos_weights, neg_weights, **kwargs):
        self.pos_weights = pos_weights
        self.neg_weights = neg_weights
    def call(self, y_true, y_logit):
        Multi-label cross-entropy
        * Required "Wp", "Wn" as positive & negative class-weights
        y_true: true value
        y_logit: predicted value
        arrh_classes = ArrhClassType
        class_count = ArrhClassType.get_active_class_count()
        loss = float(0)
        # wp = self.class_weights['positive_weights']
        # wn = self.class_weights['negative_weights']
        wp = self.pos_weights
        wn = self.neg_weights
        #print("LOGITS: ", y_logit.numpy().shape)
        for i in range(class_count):
            class_key = arrh_classes.find_from_annot_id(i).value['short']
            if class_key not in wp:
                wp[class_key] = 0.5
                wn[class_key] = 0.5
            log_logit = K.log(y_logit[:,:,i] + K.epsilon())
            true_val = y_true[:,:,i]
            ann = wp[class_key]
            first_term = ann * true_val * log_logit
            second_term = wn[class_key] * (1 - y_true[:,:,i]) * K.log(1 - y_logit[:,:,i] + K.epsilon())
            loss -= (first_term + second_term)
        return loss
    def get_config(self):
        base_config = super().get_config()
        self.pos_weights = {key:value.numpy() if type(value)==tf_python.framework.ops.EagerTensor else value for key,value in self.pos_weights.items()}
        self.neg_weights = {key:value.numpy() if type(value)==tf_python.framework.ops.EagerTensor else value for key,value in self.neg_weights.items()}

        return {**base_config, "pos_weights": self.pos_weights, "neg_weights": self.neg_weights}

So the question is, is a custom loss function necessary to train my model with weighting with the output structure I have defined? If not, how can I apply weighting to Tensorflows binarycrossentropy? If yes, how can I define an effective custom binarycrossentropy loss function for my classification problem with weighting for my imbalanced classes.

Finally, is my approach completely incorrect? Should I be structuring my outputs in a completely different way? Any and all insight is welcome and appreciated.

Thank you

  • $\begingroup$ I have multiple questions, you have 2 classes or 12?. You say each row is a distinct binary class. Secondly from first look at your error I think the shape of your class weights tensor is causing the problems. Please check/post the shape of this tensor it should have one weight for each class i.e. one element per neuron in output layer. $\endgroup$ Mar 16, 2023 at 9:31
  • $\begingroup$ Thank you for your response, each of the 12 rows is a distinct class that has 19 bins. Thus in each row, each bin can either be a 1 or 0 representing that rows class. $\endgroup$ Mar 16, 2023 at 13:38
  • $\begingroup$ No its confusing. Each row is a class and then each row has 19 bins, and then each bin represents row's class. $\endgroup$ Mar 16, 2023 at 15:45
  • $\begingroup$ This does not seem right. Again what are you trying to classify using the ECG Signal? Are there 2 classses or 12? $\endgroup$ Mar 16, 2023 at 15:47
  • $\begingroup$ I am trying to classify arrhythmias, there are 12 possible classes and some classes have possible overlap in a given bin. $\endgroup$ Mar 16, 2023 at 16:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.