Is Loss not a good indication of performance?

Im trying to segment 3D volumes using a 3D uNet network. Ive reached a stage where I am getting very good validation loss using CrossEntropy and BCE

idx:  0 of  53 - Validation Loss:  0.029650183394551277
idx:  5 of  53 - Validation Loss:  0.009899887256324291
idx:  10 of  53 - Validation Loss:  0.05049080401659012
idx:  15 of  53 - Validation Loss:  0.02019292116165161
idx:  20 of  53 - Validation Loss:  0.04724293574690819
idx:  25 of  53 - Validation Loss:  0.02810296043753624
idx:  30 of  53 - Validation Loss:  0.02642594277858734
idx:  35 of  53 - Validation Loss:  0.029894422739744186
idx:  40 of  53 - Validation Loss:  0.04158024489879608
idx:  45 of  53 - Validation Loss:  0.04574814811348915
idx:  50 of  53 - Validation Loss:  0.05406259000301361

I assumed my network is performing very well so i wrote a script to visualize my network outputs against their respective targets. What I get is something very different, not something that justifies this loss. The samples are of depth 32 and I've outputted each z-plane as a single image. Here is the target:

And the predicted output:

All samples are like this, not one of them accurately represent the target with the reported loss.. so I ask is my loss wrong? What should I look into to fix this?

Thanks

Is Loss not a good indication of performance?

It is only a relative indicator of performance over the training session.

First, what is loss anyway? In general, the loss is some expression of the difference between the model's predicted output and the target output. Depending on the loss function used (e.g. if log is involved) or on the nominal values of the inputs themselves, the value of the loss can also be very small or very large.

People usually normalise data so the values are smaller, but the point is that you cannot always say that a validation loss of 0.0012345 is actually a good value. Or that 12345 is definitely bad!

Other possible loss functions

Other 3d segmentation models that I have seen it is common to use the Dice Coefficient for your cost/loss. Maybe give that a try. The Dice coefficient is essentially the same as the F1 score; you are really finding a trade-off in how to penalise a model for its mistakes in classification e.g. of pixels or voxels. Do you want to strongly punich bad cases or rather a more averging approach. As that link points out, it is similar to the difference between the $$L_1$$ and $$L_2$$ losses.

There is also the Jaccard index, which is essentially the same as the Dice Coefficient.

The Tversky index is a generalisation of the two - it is an asymmetric similarity measure. :

$$Tversky(A, B; α, β) = \frac{|TruePos|}{|TruePos| + \alpha |FalsePos| + \beta |FalseNeg|}$$

• The Dice coefficient is this with $$\alpha = \frac{1}{2}$$ and $$\beta = \frac{1}{2}$$
• The Jaccard index instead has $$\alpha = 1$$ and $$\beta = 1$$

This might be a nice approach for your problem to tweak how the loss is computed. There are plenty of source online to explain more about these well-defined measures and perhaps help you gain an intuition for their results.

With the losses you posted, it doesn't actually look as good as you explained. I am not sure exactly what your idx values mean, but the loss values are actually going up:

In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: import matplotlib.pyplot as plt
In [4]: val_loss = np.array([0.029650183394551277, 0.009899887256324291, 0.0504908040
...: 1659012, 0.02019292116165161, 0.04724293574690819, 0.02810296043753624, 0.026
...: 42594277858734, 0.029894422739744186, 0.04158024489879608, 0.0457481481134
...: 8913, 0.05406259000301361])
In [5]: pd.Series(val_loss, index=range(0, 51, 5)).plot()
Out[5]: <matplotlib.axes._subplots.AxesSubplot at 0x7f4405326390>
In [6]: plt.show()

This suggests you might be overfitting. However, the differences between the predictions and ground truth in the images you show suggest there is a more fundamental problem.

• This is very helpful, many thanks. I guess I will have to look into implementing DiceLoss. Nov 30 '18 at 17:16
• @StuckInPhD - you're welcome! Yes, you'll have to alter your data a little bit, but most libraries have an implementation of the actual Dice coefficient. Dec 1 '18 at 1:30