# Intuitive explanation of Lovasz Softmax loss for Image Segmentation problems

Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.

The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.

The loss can be optimized on its own, but the optimal optimization hyperparameters (learning rates, momentum) might be different from the best ones for cross-entropy. As discussed in the paper, optimizing the dataset-mIoU (Pascal VOC measure) is dependent on the batch size and number of classes. Therefore you might have the best results by optimizing with cross-entropy first and finetuning with our loss, or by combining the two losses.

Here is the Implementation of Lovasz Softmax Loss in Pytorch & Tensorflow.

A step by step explanation of the important steps of the code:

Step 1: compute the errors of the predictions:

signs = 2. * labels.float() - 1.
errors = (1. - logits * Variable(signs))
errors_sorted, perm = torch.sort(errors, dim=0, descending=True)


The "errors" vector is positive when the predictions are false and negative when the predictions are correct. Nothing special here. Note that a margin of (1) is used here, which means those correct predictions that have a margin lower than 1 is considered as an error.

Step 2: Lovasz extension w.r.t sorted errors

gts = gt_sorted.sum()
ntersection = gts - gt_sorted.float().cumsum(0)
union = gts + (1 - gt_sorted).float().cumsum(0)
jaccard = 1. - intersection / union


At this point, the vector "jaccard" contains the evolution of the Jaccard index with respect to the sorted errors. It varies between 0 and the actual Jaccard index of the prediction.

Step 3: gradient of the Lovasz extension w.r.t sorted errors

The next step of the code consists of computing the gradient of this vector. It is computed by a backward difference:

jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]


This vector says what is the effect of an error on the evolution of the Jaccard index. The idea is to minimize the errors that penalize the Jaccard index the most.

Step 4: Compute the loss

In the last step of the code, the actual loss is computed as the dot product between the error vector and the gradient of the Lovasz extension. Relu is used because only a positive part of the error vector contains prediction errors.

loss = torch.dot(F.relu(errors_sorted), Variable(grad))