7
$\begingroup$

I have a binary segmentation problem with highly imbalanced data such that there are almost 60 class zero samples for every class one sample. To address this issue, I coded a simple weighted binary cross entropy loss function in Keras with Tensorflow as the backend.

def weighted_bce(y_true, y_pred):
  weights = (y_true * 59.) + 1.
  bce = K.binary_crossentropy(y_true, y_pred)
  weighted_bce = K.mean(bce * weights)
  return weighted_bce

I wanted to ask if this implementation is correct because I am new to Keras/Tensorflow and the optimizer is having a hard time optimizing this. The loss goes from something like 1.5 to 0.4 and doesn't go down further. Normal binary cross entropy performs better if I train it for a long time to the point of over-fitting. Before anyone asks, I cannot use class_weight because I am training a fully convolutional network.

$\endgroup$
2
  • $\begingroup$ As mentioned in the Tensorflow documentation for imbalanced class classification some optimizers might fail to converge when weighting the Binary Cross Entropy loss, since they are highly sensitive to the scale of the gradient: $\endgroup$
    – Danfoa
    Mar 15 '20 at 22:57
  • 1
    $\begingroup$ "Note: Using class_weights changes the range of the loss. This may affect the stability of the training depending on the optimizer. Optimizers whose step size is dependent on the magnitude of the gradient, like optimizers.SGD, may fail. The optimizer used here, optimizers.Adam, is unaffected by the scaling change. Also note that because of the weighting, the total losses are not comparable between the two models. $\endgroup$
    – Danfoa
    Mar 15 '20 at 22:57
4
$\begingroup$

The code is correct. The reason, why normal binary cross entropy performs better, is that it doesn't penalize for mistakes on the smaller class so drastically as in weighted case. To be sure, that this approach is suitable for you, it's reasonable to evaluate f1 metrics both for the smaller and the larger classes on the validation data. It might show that performance on the smaller class becomes better. And training time can increase, because the model is forced to discriminate objects of different classes and to learn important patterns to do that.

$\endgroup$
0
$\begingroup$

accuracy is no longer a good enough metric for unbalanced data.

Simple example: for anomaly detection you usually find yourself with much more "normal" cases than anomalies. Let's say you have 98% of the time good cases and 2% of the time "anomalies", you would just need to predict all of the cases as "normal" and you would have a 98% accuracy (pretty good right?)

Now imagine that your model is to prevent credit card fraud do you think that 2% error is good enough? Not really, you would lose a lot of clients. In this case you need to use alternative metrics:

  • recall
  • F1 score

to evaluate your model performance. I would advise you to compare model performance (with your traditional and modified losses) on the alternative metrics and you will see an improvement.

$\endgroup$
1
  • 1
    $\begingroup$ The author hasn't said anything about accuracy, and according to the question just loss was considered. If I understand correctly, the question is why loss didn't decrease, but not why accuracy steal behave in a certain way. If the author considered just accuracy to evaluate the model, the question wouldn't probably appear at all. However, I absolutely agree with your explanation of f1 benefit in such case :) $\endgroup$
    – Lana
    Sep 6 '19 at 5:52
0
$\begingroup$

Here is my implementation, I hope it can help you. This is for image segmentation problem (binary classification). The ground truth image size is (512,512,1):

def weighted_BCE_loss(y_true, y_pred, positive_weight=5):
    # y_true: (None,None,None,None)     y_pred: (None,512,512,1)
    y_pred = K.clip(y_pred, min_value=1e-12, max_value=1 - 1e-12)
    weights = K.ones_like(y_pred)  # (None,512,512,1)
    weights = tf.where(y_pred < 0.5, positive_weight * weights, weights)
    # weights[y_pred<0.5]=positive_weight
    out = keras.losses.binary_crossentropy(y_true, y_pred)  # (None,512,512)
    out = K.expand_dims(out, axis=-1) * weights  # (None,512,512,1)* (None,512,512,1)
    return K.mean(out)```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.