"Optimal range for loss function": Myth or Truth?

Question

I am currently working on a regression problem using a deep neural network that given a volume 32x32x256 in input need to generate a second volume of the same dimensions in output, this is not a segmentation problem and the volume generated by the net is formed by floats (to be more specific, im tryng to remove noise from a noisy x-ray beam).

As a loss function i am using the MSE beetwen the volume generated by the net and a ground volume already calculated without noise.

The volumes have been normalized on values between 0 and 1, before giving them to the net.

When we started the first trainings we were getting very low values in MSE (about 3*10^-4), but the deep-net was converging (on values near 5*10^6).

One of my co-worker stated that those values (of the loss) were far to low to get an optimal result with a framework as Keras adn Adam as optimizer, and proposed to change the loss function to get result within a range of 100 - 0.01, he proposed to do the sum of the square difference instead of the mean.

At first i was really confused by his statement, but a did a test:

Instead of normalizing the volumes on values between 0 and 1 i normalized them between values 0 and 1000, before giving them to the net.

In this test the MSE went from values of about 30'000 and coverged of values near 600, after denormalizing the data this new neural network gave result much more accurate of the first network, and just by changing the starting normalization!

Do you have any idea of the cause of this behaviour, was my coworker right?

Answer 1

Very interesting statement you got there, i was not aware of this 'optimal loss value'.

the only way i could explain it is with the gradient computation : gradient is computed using :

$Grad = loss*lr$

So to me, changing the loss range is the same as changing the learning rate of your optimizer, it could be a good experiment to increase your lr instead of increasing the loss function range and see what happens to the training. I'm not aware of the subtilties of Adam optimizer so my explanation may be completly wrong.

Answer 2

I just made a similar observation, experimenting with a custom loss function that contains a scaling factor. Running a small grid search for this scaling parameter, with an adam optimiser, I found better results for bigger factors, equaling bigger loss values. Right now this is n=1, so more anecdotal evidence, but I am interested in the intuition behind the interaction between loss function and optimiser.

From my understanding upscaling the loss values should translate to increasing the "total height of solution space". Therefore the valleys should become lower and the peaks should become higher, leaving steeper slops in between them. From this intuition, the effect of upscaling the loss should be roughly the same as increasing the momentum of the optimiser (?). Further, staying with this picture of peaks and valleys in solution space, one benefit of upscaling the loss values could be that the difference between almost equally shallow minima is increased. On a hard classification problem, where there are two or more solutions that are almost equally good, upscaling the loss values could therefore increase the likelihood of the optimiser finding the slightly better solution (= deeper valley).

Does this make any sense?